Failover Network Connections?

T

TechieSooner

Supreme [H]ardness
Joined
Nov 7, 2007
Messages
7,601
Suggestions on this?

Basically, if one goes down, the other needs to come up. I don't care about load balancing or anything like that (though if I can do it, wouldn't be that bad of deal), mainly just wanting redundancy.

So, I'm planning on two different technologies (Fiber, Cable, T1, etc) with two different ISPs.

The biggest issue I see is incoming IP addresses.

IE, I have a web server on 210.210.210.210, another on 210.210.210.211, and Exchange on 210.210.210.212 (Bogus IPs). Going to web1.domain.com resolves to 210, web2.domain.com resolves to web2.domain.com, exchange.domain.com resolves to 212.

You get the idea. Let's say all those IPs are on a Cable connection. The cable connection goes down, so I want to use the Fiber connection. For web browsing, tweaking the router config with the new IPs, it works instantly. But those IPs on that Cable connection are now dead. They aren't getting routed to the Fiber IPs.

What's the solution to that, other than waiting hours for DNS changes to propagate?


I think it would require cooperation from both ISPs to get some sort of failover system setup. Which getting competitors to compete, probably is going to be a bigger pain than it is worth!
 
PFSense comes to mind.

DNS issue i don't know about. I would think a program that would send the new connection details to DNS account.
 
how about sticking with one isp, but get them to provide you with two connections to different internet pop's? they could then use bgp to provide you an active/passive (or maybe even active/active if you want to pay for it) internet connection utilising the same block of public ip addresses. combine that with a pair of decent firewalls (i would recommend juniper netscreen) in a ha configuration, with a pair of switches in a ha configuration and your good to go. none of this will be cheap, but it depends just how reliant upon their internet connection this customer is, i guess?
 
Shockey said:
DNS issue i don't know about. I would think a program that would send the new connection details to DNS account.
Click to expand...
But what about reverse DNS? That's the issue I see with that, when my Exchange server gets onto a different IP address.

marley1 said:
You would want 2 mx records with diff priority, no?
Click to expand...
That would work for Exchange, sure. What about all my IPSec tunnels and such, which all run off IP addresses?

atomiser said:
how about sticking with one isp, but get them to provide you with two connections to different internet pop's?
Click to expand...
The problem is, if I do that, it all connects to their Central Office in the same box. That box fails (which has happened to me before), both lines are screwed. That's why I've got to make sure they go through different offices.
 
If you're wanting that kind of redundancy you will either need to work with ISPs that will do BGP advertising for the used address space (not exactly cheap) or colocate the critical infrastructure elsewhere. Honestly your best option is probably colo.
 
You could try something like this. I've never used anything like that before, but it seems like it should work ok. I doubt you'd have to worry too much about the time to propagate, it shouldn't be bad, I think the time it takes for caches to clear could be an issue though.
 
reverse dns is seasy if you use a smarthost just have exchange configured to use that, doesn't matter what ip you are on.
 
Vito_Corleone said:
You could try something like this. I've never used anything like that before, but it seems like it should work ok. I doubt you'd have to worry too much about the time to propagate, it shouldn't be bad, I think the time it takes for caches to clear could be an issue though.
Click to expand...
Could work. I could get a dual-WAN router, do NAT from each connection, and basically it takes care of itself from there if either connection goes down.

My question on that I wonder, is how your DNS is setup? For my Reverse DNS to work, the ISP (ATT) must do my DNS resolution. Do I setup A record to go to dnsservice.dnsmadeasy.com (another DNS entry)???

Xipher said:
If you're wanting that kind of redundancy you will either need to work with ISPs that will do BGP advertising for the used address space (not exactly cheap) or colocate the critical infrastructure elsewhere. Honestly your best option is probably colo.
Click to expand...
Bah, I don't want to do a colo... I really don't need 100% HA, I just want something that's fairly simple to switch to and from, in a couple hours time.
 
Vito_Corleone said:
You could try something like this. I've never used anything like that before, but it seems like it should work ok. I doubt you'd have to worry too much about the time to propagate, it shouldn't be bad, I think the time it takes for caches to clear could be an issue though.
Click to expand...

With DNS failover they usually set the TTL so low caching is not a serious issue.

The issue with DNS failover though is that it doesn't handle existing connections. If the one connection is down packets won't get re-routed over to the new connection until they time out and resolve the name and try connecting again. When you let the routing protocol handle the work existing connections will probably see a problem for a while but can continue once the routing advertisement propagates.


TechieSooner said:
Bah, I don't want to do a colo... I really don't need 100% HA, I just want something that's fairly simple to switch to and from, in a couple hours time.
Click to expand...

Sounds like the DNS failover would be a good option.
 
Know how it works though? To keep my Reverse DNS intact I need to keep the A records with my ISP though...
 
What exact type of connections are these?

If they are business connections and are decent (the T1 provider might do it), try to inquire about BGP routing and setup your own ASN.
 
Ockie said:
What exact type of connections are these?

If they are business connections and are decent (the T1 provider might do it), try to inquire about BGP routing and setup your own ASN.
Click to expand...

Well, I'm still debating about that. I'm leaning toward doing a T1 and then a business Cable line.

The problem is I'm not sure if I can get the two ISPs to actually cooperate.
 
I'll give you a quick (and basic since I don't know understand it in great detail myself) description. Basically you would be running BGP multi-homed to two (or more) ISPs. So you would be advertising your address space to both. When the path through ISPa was unavailable the path through ISPb would be taken. This would allow you to use the same address space through two ISP, so you wouldn't have to worry about DNS or any of that.
 
So, that Address Space that I buy on my own, how is it aware that ISPa is down? It still sounds to me like there would have to be some sort of external entity that would have to test each route to see which to take.
 
I know I'm a little late to this party, but figured I'd drop my $0.02.

I'm currently in the process of stetting this up for a customer.

First of all, to even get your own BGP AS and address space you have to submit a request to arin. For a multi-homed solution, the smallest subnet of public address they will hand out is a /22. You also have to be able to use 25% of the address at implementation, and 75% in 6months. (IIRC) This is all documented and reviewed by arin for approval.

This is no easy feat and is required.

I highly suggest NOT attempting to do this unless your company is ready to shell out big bucks and your company really has a use for it. Internet access for most companies is important, but not worth the huge investment of 24/7/365 service. Most times, they can live with short outages. Unless you are dealing with a companies business that is internet based.

Just my $0.02, FWIW.
 
WesM63 said:
Unless you are dealing with a companies business that is internet based.
Click to expand...

If you are this business take into serious consideration co-locating. Lots of places have Service Level Agreements for up time. and already have in place what your trying to do now. IMO I would just let them worry about keeping the internet alive in their facility and my business going. That is my two cents.
 
Wes hit the nail on the head. BGP will most likely be way out of your league, unless you're willing to shell out some cash. Also expect to be pushed heavily to become a transit since you're peering with multiple ISPs. If any of the verbage goes over your head, please don't even consider BGP as nothing in this thread even brushed the surface.

I have recommended this solution to MANY people on these forums, and ill recommend it again. Say it with me, IP SLA(or some sort of variant of upstream tracking). If you have Cisco on the mix, great... IP SLA is what you're going to want to use.. You're going to want to do advanced object tracking and tie that object to one of your(primary probably) static default routes. That object will ping something OTHER than your gateway, perhaps the good ole' 4.2.2.2.. if that fails, your primary route will be removed from the routing table and the secondary(ISP) will take over.

If you want a linux solution, have a script that runs a cron to ping a public address ever few minutes.. if its not available switch your iptables/ipchains/pf whatever to the standby ISPs next hop gateway.
 
calvinj said:
If you are this business take into serious consideration co-locating. Lots of places have Service Level Agreements for up time. and already have in place what your trying to do now. IMO I would just let them worry about keeping the internet alive in their facility and my business going. That is my two cents.
Click to expand...

Just to clarify, I wouldn't suggest doing multi-homed BGP, unless your a co-loc facility or similar.

I completely agree though, if OP is looking for something similar, co-loc is your best bet.
 
WesM63 said:
BGP AS and address space you have to submit a request to arin. For a multi-homed solution, the smallest subnet of public address they will hand out is a /22. You also have to be able to use 25% of the address at implementation, and 75% in 6months. (IIRC) This is all documented and reviewed by arin for approval.:eek:
Click to expand...
Does mapping multiple IPs to the same host count? :D
Otherwise there'd be no way.

calvinj said:
If you are this business take into serious consideration co-locating. Lots of places have Service Level Agreements for up time. and already have in place what your trying to do now. IMO I would just let them worry about keeping the internet alive in their facility and my business going. That is my two cents.
Click to expand...
Yea... I've had SLAs and even then, ISPs still go down.
 
TechieSooner said:
Does mapping multiple IPs to the same host count? :D
Otherwise there'd be no way.


Yea... I've had SLAs and even then, ISPs still go down.
Click to expand...
Did you miss my post? Perfect solution for you, all wrapped up with a bow :p I completely agree with your above statements as well, I wouldn't like to leave my failover solution to an SLA. :rolleyes:
 
xphil3 said:
Did you miss my post? Perfect solution for you, all wrapped up with a bow :p
Click to expand...

I don't disagree. I just don't think the two only ISPs in this area would do what I'm asking to get BGP functional.

One connection, they might (A T1) however the other one is just business class cable, I really don't think they'd help me out unless I had a dedicated symmetric cable connection of some sort.
 
TechieSooner said:
Yea... I've had SLAs and even then, ISPs still go down.
Click to expand...

Ok, let me clear up this confusion.

SLA = An agreement with your ISP that you will have certain performance, reliability, etc.. it's basically a contract telling you that they won't screw it up (which is usually a lie).

IPSLA = A (cool?) Cisco IOS feature that sends probes out and reports statistics back to the router. These can be a bunch of different types, but some of the more advanced ones require an IOS responder on the other end. Although, a generic ICMP echo is supported, which obviously works for any host (like the suggested root dns 4.2.2.2)

xphil3's suggesting that you setup something called tracked object routes. Essentially, IOS will setup an IPSLA for you. Then, if that tracked object fails to respond past your configured threshold, the route will be removed from your routing table until the probe works again. Config is pretty simple, but it'll be a good exercise to figure it out.

IPSLAs are very reliable and used in just about any serious network.
 
just2cool said:
Ok, let me clear up this confusion.

SLA = An agreement with your ISP that you will have certain performance, reliability, etc.. it's basically a contract telling you that they won't screw it up (which is usually a lie).

IPSLA = A (cool?) Cisco IOS feature that sends probes out and reports statistics back to the router. These can be a bunch of different types, but some of the more advanced ones require an IOS responder on the other end. Although, a generic ICMP echo is supported, which obviously works for any host (like the suggested root dns 4.2.2.2)

xphil3's suggesting that you setup something called tracked object routes. Essentially, IOS will setup an IPSLA for you. Then, if that tracked object fails to respond past your configured threshold, the route will be removed from your routing table until the probe works again. Config is pretty simple, but it'll be a good exercise to figure it out.

IPSLAs are very reliable and used in just about any serious network.
Click to expand...
That sounds like it'll work for outgoing stuff, but how's it going to work when a packet on ISP #1 destined for 123.123.123.123, and ISP #1 fails?

Ockie said:
We don't go down :)
Click to expand...
I'm assuming that's a kid.......

In a bigger city where fiber is availible and cheap, I'm sure it's much cheaper ;)
 
TechieSooner said:
That sounds like it'll work for outgoing stuff, but how's it going to work when a packet on ISP #1 destined for 123.123.123.123, and ISP #1 fails?
Click to expand...
With any solution and your requirement you're going to need to do multi-homing if that server is expected to have a public IP address supplied by your ISP(s). DNS cacheing will also need to be configure accordingly.
 
xphil3 said:
With any solution and your requirement you're going to need to do multi-homing if that server is expected to have a public IP address supplied by your ISP(s). DNS cacheing will also need to be configure accordingly.
Click to expand...

So I have my WWW record for web1.domain.com going to 123.123.123.123, and ISP #1 goes down.

There's no way to have it filter over to ISP #2 and going to 234.234.234.234 without changing the DNS....
 
what is the reasoning behind not colocating this stuff?

what is the web server doing?

talk to ockie bout colocating with him, i would imaging price of 2 ISP with a SLA (1 being T1 like you mentioned) and all the hardware, labor to configure, troubleshooting would be way higher then colocating
 
marley1 said:
what is the reasoning behind not colocating this stuff?

what is the web server doing?

talk to ockie bout colocating with him, i would imaging price of 2 ISP with a SLA (1 being T1 like you mentioned) and all the hardware, labor to configure, troubleshooting would be way higher then colocating
Click to expand...

It would be significantly higher for sure. It's cheaper to co locate than to just buy even a residential cable service.
 
Ockie said:
It would be significantly higher for sure. It's cheaper to co locate than to just buy even a residential cable service.
Click to expand...

Co-Loc would be fine for a web server, but I was just using that as an example. I've got to keep that stuff in house.
 
You must log in or register to reply here.
Back
Top