A "Server Misconfiguration" Was Behind the Facebook Outage

AlphaAtlas

[H]ard|Gawd
Staff member
Joined
Mar 3, 2018
Messages
1,713
Facebook and Instagram were down for about 14 hours earlier this week, and earlier reports suggested that the underlying cause was a BGP routing issue. As time went on without an official explanation, many started to suspect that the outage was related to an attack on the platform. But yesterday, Facebook tweeted that the trouble was a "result of a server configuration change." There's still no mention of the incident in Facebook's official newsroom, and some news outlets are starting to criticize the timeliness of Facebook's response and explanation. One security analyst told the BBC that "Facebook's motto always used to be 'move fast and break things'. That's fine when you're an innovative start-up, but when billions of people are using your site every month it's not a good way to run the business."

Yesterday, as a result of a server configuration change, many people had trouble accessing our apps and services. We've now resolved the issues and our systems are recovering. We're very sorry for the inconvenience and appreciate everyone's patience.
 

TonyZ

Limp Gawd
Joined
Jul 29, 2004
Messages
168
Facebook, the platform of idiots.... You couldn't pay me to sign up for that...
 

Darunion

2[H]4U
Joined
Oct 6, 2010
Messages
4,029
I use facebook, but seriously didnt know anything about it until i saw it stated here. No mention from any of my friends either. Now if my WoW connection goes down imma flip some damn tables #nerdrage....
 

jardows

[H]ard|Gawd
Joined
Jun 10, 2015
Messages
1,888
I can't correctly remember, but wasn't there another, very significant outage with a major organization that was caused by "server misconfiguration" in the last couple of years?
 

the-one1

2[H]4U
Joined
Jan 16, 2003
Messages
2,982
FB goes down for 14 hrs, people freak out. FB gets into trouble for selling data, which is how they make money, people freak out.
 

CaptNumbNutz

Bulls[H]it Master
Joined
Apr 11, 2007
Messages
21,224
Although a BGP configuration issue sounds a like a plausible cause, I don't buy it because they keep saying "server misconfiguration".

1. Why is Facebook doing anything BGP related? Border Gateway Protocol is used by ISP's and internet backbone providers to route between network providers. Facebook is neither.
2. Why is anyone doing anything BGP related on a "server"? BGP is usually handled by routers and layer 3 managed switches.
 

Exavior

[H]F Junkie
Joined
Dec 13, 2005
Messages
9,694
Although a BGP configuration issue sounds a like a plausible cause, I don't buy it because they keep saying "server misconfiguration".

1. Why is Facebook doing anything BGP related? Border Gateway Protocol is used by ISP's and internet backbone providers to route between network providers. Facebook is neither.
2. Why is anyone doing anything BGP related on a "server"? BGP is usually handled by routers and layer 3 managed switches.
you are mixing and matching different issues. The BGP issue was reported as an ISP having misconfigured their router and incorrectly told the world they managed that block of IPs. Resulting in what was thought to then be both routers fighting and taking control back and forth.

they found that wasn't the issue and that they fucked up a server. Two different possible issues caused by two groups.
 

CaptNumbNutz

Bulls[H]it Master
Joined
Apr 11, 2007
Messages
21,224
you are mixing and matching different issues. The BGP issue was reported as an ISP having misconfigured their router and incorrectly told the world they managed that block of IPs. Resulting in what was thought to then be both routers fighting and taking control back and forth.

they found that wasn't the issue and that they fucked up a server. Two different possible issues caused by two groups.
Gotcha... I think I misread that. It read to me like FB was originally claiming a BGP issue, when it was others that were claiming it.

Thanks.
 

STEM

Gawd
Joined
Jun 7, 2007
Messages
610
Gotcha... I think I misread that. It read to me like FB was originally claiming a BGP issue, when it was others that were claiming it.

Thanks.
I find it beyond hilarious that they claim self-sabotage. It looked more like... they were making some internal changes on a very large scale. Considering how distributed Facebook is, for so many errors to happen continuously, it must be a massive outage. They will never tell us the truth, and I honestly don't care. One can only hope that this network of idiots will be gone from this earth one day.
 

toast0

Gawd
Joined
Jan 26, 2010
Messages
993
Although a BGP configuration issue sounds a like a plausible cause, I don't buy it because they keep saying "server misconfiguration".

1. Why is Facebook doing anything BGP related? Border Gateway Protocol is used by ISP's and internet backbone providers to route between network providers. Facebook is neither.
I work at Facebook, and I'm not authorized or going to discuss the issue at hand; but FB runs BGP because of its large network of Points of Presence throughout the world, and the peering and transit relationships that involves. Really, anyone operating a service on hardware they own in multiple locations is going to want to own their IPs and run an AS, so they can use multiple transit providers, and that means running BGP. You can see public information about this at Hurricane Electric's BGP info, here or on PeeringDB here.
 

CaptNumbNutz

Bulls[H]it Master
Joined
Apr 11, 2007
Messages
21,224
I work at Facebook, and I'm not authorized or going to discuss the issue at hand; but FB runs BGP because of its large network of Points of Presence throughout the world, and the peering and transit relationships that involves. Really, anyone operating a service on hardware they own in multiple locations is going to want to own their IPs and run an AS, so they can use multiple transit providers, and that means running BGP. You can see public information about this at Hurricane Electric's BGP info, here or on PeeringDB here.
Fine. That makes more sense. The way it's explained means that many large spread out companies may do this type of setup... e.g. Netflix, Google, etc.

It doesn't explain my point #2 though. If this was a BGP issue, I wouldn't expect it to be a "server" misconfiguration. More like a router misconfiguration. However, it's now sounding like BGP was not the issue to begin with.
 

STEM

Gawd
Joined
Jun 7, 2007
Messages
610
I work at Facebook, and I'm not authorized or going to discuss the issue at hand;
I don't think that anyone asked or wanted to know, however, thank you for sharing that info. Personally, I don't think that Facebook does anything wrong, it's a free-to-join service after all where the users are the product. In my oh-so-humble opinion, the issue lies with people who are stupid enough, join, share their most intimate personal, and allows Facebook to spy on them, and then they bitch about it when they find out that Facebook did something with their personal info that if shouldn't have. Or my favorite is when people complain that Facebook manipulated their decision-making process like I don't know, who to vote with for example.

Since you're here, I want to ask you something, because I'm terrified of installing the Facebook app on my phone: Is it true that the Facebook app listens, and possibly records everything that it can hear over your phone's microphone? If so, I think that this is something that should be disclosed. Yes, I have friends I need to keep in touch with over FB because they're too lazy to use anything else like Signal for example.

Thank you.
 

Aireoth

Supreme [H]ardness
Joined
Oct 12, 2005
Messages
4,122
excuse us while we move all our collected data off server prior to announcing our new 'privacy focused' offerings.
 

STEM

Gawd
Joined
Jun 7, 2007
Messages
610
excuse us while we move all our collected data off server prior to announcing our new 'privacy focused' offerings.
Shoosh, you'll scare him away before I get a chance to find out if Facebook spies on us via our smartphone's microphone.
 

Sonicks

[H]ard|Gawd
Joined
Jul 24, 2005
Messages
1,401
Since you're here, I want to ask you something, because I'm terrified of installing the Facebook app on my phone: Is it true that the Facebook app listens, and possibly records everything that it can hear over your phone's microphone? If so, I think that this is something that should be disclosed. Yes, I have friends I need to keep in touch with over FB because they're too lazy to use anything else like Signal for example.
The myth that the FB app hot mics and listens to everything has been disproven and Facebook themselves deny this. Though it definitely seems suspicious at times when you converse about something and suddenly see advertisements for it, there is apparently some algorithmic magic going on between shopping history of you and your friends circle as well as geo-tagged info, posting topics, likes, ad engagements, etc... that it uses to dish out targeted ads that appear so specific so as to make one think it's simply spying.

I still think it listens though...
 

toast0

Gawd
Joined
Jan 26, 2010
Messages
993
Since you're here, I want to ask you something, because I'm terrified of installing the Facebook app on my phone: Is it true that the Facebook app listens, and possibly records everything that it can hear over your phone's microphone? If so, I think that this is something that should be disclosed. Yes, I have friends I need to keep in touch with over FB because they're too lazy to use anything else like Signal for example.
There's an official response to this, here. To the best of my knowledge, that's a truthful statement, but I don't really work in that part of the company. It's less convenient, but the mobile website version of FB is pretty reasonable, and being in a webbrowser gives you more control over what it can do, I think you can do messaging, but I don't think you can get push messages.

Sorry, CaptNumbNutz, I understand your question, but I can't answer it; a clear statement from Facebook addressing that would be nice; I could really only find a third party report from NetScout that was mentioned in the earlier [H] coverage and another from ThousandEyes. Between that and 'server configuration error' being enormously vague, you'll have to make up your own mind. Edit, according to another 3rd party tweet, NetScout claims their initial claim was a miscommunication.
 

STEM

Gawd
Joined
Jun 7, 2007
Messages
610
Facebook, in and of itself, is a server misconfiguration.

The real news here is Google gmail went down with nary a peep about the cause.
https://www.cnet.com/news/gmail-is-down-outage-around-the-world-for-some-users/
Yeah, but Google is cool :D /s

There's an official response to this, here. To the best of my knowledge, that's a truthful statement, but I don't really work in that part of the company. It's less convenient, but the mobile website version of FB is pretty reasonable, and being in a webbrowser gives you more control over what it can do, I think you can do messaging, but I don't think you can get push messages.

Sorry, CaptNumbNutz, I understand your question, but I can't answer it; a clear statement from Facebook addressing that would be nice; I could really only find a third party report from NetScout that was mentioned in the earlier [H] coverage and another from ThousandEyes. Between that and 'server configuration error' being enormously vague, you'll have to make up your own mind. Edit, according to another 3rd party tweet, NetScout claims their initial claim was a miscommunication.
Thank you, that was a good answer :)
 

PhaseNoise

2[H]4U
Joined
May 11, 2005
Messages
3,085
I misread the title and saw "outrage" instead of "outage".

I realized that's because MainMedia is just a constant torrent of someone being outraged by something, so mental-autocorrect went there.
 
Top