![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
archivalbackup
Gawd
- Joined
- Oct 12, 2007
- Messages
- 643
First - Google had major outages, this is still very new so I do not know how widespread it was, but I know it affected a number of states and seemed to primary affect AT&T's network and Qwest's network (unfortunately our two ISPs at work) but not McCloud.
I was fortunate to have a bunch of debugs while trying to track down the problem that I wanted to share for everyone to discuss. I believe the problem was an ISP (ntt.net) did not filter a customer BGP advertisement, who decided to advertise a best route for google which several major ISPs picked up. Below are my logs and brief analysis.
0. My gmail stops working. Switch machines, not working. Check the router:
My home router is fine
Call an ISP down the street I know and they are not having any problems out via McCloud or their other ISP. Check internet health report - problems with AT&T to Internap and internap to Cogent (remember Cogent) packet losses close to 4%.
Ok, Checking looking glass ... most of which I can not reach, but I get into oregon-ix.net (an .edu) and I later I parse down to this important tidbit:
So my google.com traffic is getting routed through AS 2914 which is some Japanese ISP that happens to have a presence in Centennial Colorado, and I guess major peering with Qwest and AT&T.
1. Call into AT&T and after a long time on hold get a tech that 'yeah our network is [fucked] having problems' and we have a master ticket number you can have, but other ISPs are down too.
2. Chat with Xphile3 and learn this is a major problem, different state.
3. Things get fixed, now what changed:
Ok we go from AS 7018 (AT&T) to AS 3356 (Level 3). When it was broken we went:
When it was fixed we now go:
Let's take a peak at Level 3's network status page and see if there is anything interesting going on ... Holy hell look at Cognet and and global crossing! http://www.esupport24.com/netstatus/
-------------------------
edit: I am going to keep editing this to add information / analysis. I just noticed that the advertised route from ntt.net was:
Google's network is actually a /18 not /23 so another issue at play was longest match in BGP:
I was fortunate to have a bunch of debugs while trying to track down the problem that I wanted to share for everyone to discuss. I believe the problem was an ISP (ntt.net) did not filter a customer BGP advertisement, who decided to advertise a best route for google which several major ISPs picked up. Below are my logs and brief analysis.
0. My gmail stops working. Switch machines, not working. Check the router:
Ah crap! See which ISP it is coming from, and trace it:
Ok WTF! Tokyo?!?!?! that's fucked. Kill our AT&T connection and fail over to Qwest, still having problems. Try from home - google is fine. Try from my iPhone, google is down (Ok calmed down a bit that all of AT&T's network is jacked getting to google).
My home router is fine
Call an ISP down the street I know and they are not having any problems out via McCloud or their other ISP. Check internet health report - problems with AT&T to Internap and internap to Cogent (remember Cogent) packet losses close to 4%.
Remember AS 2914. AS 15169 is google. AS 7018 is AT&T where I get my BGP route from. So why the F is google getting advertised by NTT.net as a 1 hop route via Tokyo to a vast majority of the internet? I have to guess that it is the same deal as the youtube black hole problem that happened a few months back in which some ass decided to advertise youtube as a null route to block it, but their upstream ISP did not have in place basic blocking or filtering of customer advertised routes.
So my google.com traffic is getting routed through AS 2914 which is some Japanese ISP that happens to have a presence in Centennial Colorado, and I guess major peering with Qwest and AT&T.
1. Call into AT&T and after a long time on hold get a tech that 'yeah our network is [fucked] having problems' and we have a master ticket number you can have, but other ISPs are down too.
2. Chat with Xphile3 and learn this is a major problem, different state.
3. Things get fixed, now what changed:
Ok we go from AS 7018 (AT&T) to AS 3356 (Level 3). When it was broken we went:
from 7018 (AT&T) to 2914 (NTT.NET)
When it was fixed we now go:
fixed said:
Let's take a peak at Level 3's network status page and see if there is anything interesting going on ... Holy hell look at Cognet and and global crossing! http://www.esupport24.com/netstatus/
-------------------------
edit: I am going to keep editing this to add information / analysis. I just noticed that the advertised route from ntt.net was:
my view during outage said:
Google's network is actually a /18 not /23 so another issue at play was longest match in BGP:
arin said:
Last edited:
