A "Server Misconfiguration" Was Behind the Facebook Outage

AlphaAtlas

AlphaAtlas

[H]ard|Gawd
Staff member
Joined
Mar 3, 2018
Messages
1,713
Facebook and Instagram were down for about 14 hours earlier this week, and earlier reports suggested that the underlying cause was a BGP routing issue. As time went on without an official explanation, many started to suspect that the outage was related to an attack on the platform. But yesterday, Facebook tweeted that the trouble was a "result of a server configuration change." There's still no mention of the incident in Facebook's official newsroom, and some news outlets are starting to criticize the timeliness of Facebook's response and explanation. One security analyst told the BBC that "Facebook's motto always used to be 'move fast and break things'. That's fine when you're an innovative start-up, but when billions of people are using your site every month it's not a good way to run the business."

Yesterday, as a result of a server configuration change, many people had trouble accessing our apps and services. We've now resolved the issues and our systems are recovering. We're very sorry for the inconvenience and appreciate everyone's patience.
 
Facebook, the platform of idiots.... You couldn't pay me to sign up for that...
 
I use facebook, but seriously didnt know anything about it until i saw it stated here. No mention from any of my friends either. Now if my WoW connection goes down imma flip some damn tables #nerdrage....
 
I can't correctly remember, but wasn't there another, very significant outage with a major organization that was caused by "server misconfiguration" in the last couple of years?
 
FB goes down for 14 hrs, people freak out. FB gets into trouble for selling data, which is how they make money, people freak out.
 
Although a BGP configuration issue sounds a like a plausible cause, I don't buy it because they keep saying "server misconfiguration".

1. Why is Facebook doing anything BGP related? Border Gateway Protocol is used by ISP's and internet backbone providers to route between network providers. Facebook is neither.
2. Why is anyone doing anything BGP related on a "server"? BGP is usually handled by routers and layer 3 managed switches.
 
CaptNumbNutz said:
Although a BGP configuration issue sounds a like a plausible cause, I don't buy it because they keep saying "server misconfiguration".

1. Why is Facebook doing anything BGP related? Border Gateway Protocol is used by ISP's and internet backbone providers to route between network providers. Facebook is neither.
2. Why is anyone doing anything BGP related on a "server"? BGP is usually handled by routers and layer 3 managed switches.
Click to expand...

you are mixing and matching different issues. The BGP issue was reported as an ISP having misconfigured their router and incorrectly told the world they managed that block of IPs. Resulting in what was thought to then be both routers fighting and taking control back and forth.

they found that wasn't the issue and that they fucked up a server. Two different possible issues caused by two groups.
 
Exavior said:
you are mixing and matching different issues. The BGP issue was reported as an ISP having misconfigured their router and incorrectly told the world they managed that block of IPs. Resulting in what was thought to then be both routers fighting and taking control back and forth.

they found that wasn't the issue and that they fucked up a server. Two different possible issues caused by two groups.
Click to expand...
Gotcha... I think I misread that. It read to me like FB was originally claiming a BGP issue, when it was others that were claiming it.

Thanks.
 
CaptNumbNutz said:
Gotcha... I think I misread that. It read to me like FB was originally claiming a BGP issue, when it was others that were claiming it.

Thanks.
Click to expand...

I find it beyond hilarious that they claim self-sabotage. It looked more like... they were making some internal changes on a very large scale. Considering how distributed Facebook is, for so many errors to happen continuously, it must be a massive outage. They will never tell us the truth, and I honestly don't care. One can only hope that this network of idiots will be gone from this earth one day.
 
CaptNumbNutz said:
Although a BGP configuration issue sounds a like a plausible cause, I don't buy it because they keep saying "server misconfiguration".

1. Why is Facebook doing anything BGP related? Border Gateway Protocol is used by ISP's and internet backbone providers to route between network providers. Facebook is neither.
Click to expand...

I work at Facebook, and I'm not authorized or going to discuss the issue at hand; but FB runs BGP because of its large network of Points of Presence throughout the world, and the peering and transit relationships that involves. Really, anyone operating a service on hardware they own in multiple locations is going to want to own their IPs and run an AS, so they can use multiple transit providers, and that means running BGP. You can see public information about this at Hurricane Electric's BGP info, here or on PeeringDB here.
 
toast0 said:
I work at Facebook, and I'm not authorized or going to discuss the issue at hand; but FB runs BGP because of its large network of Points of Presence throughout the world, and the peering and transit relationships that involves. Really, anyone operating a service on hardware they own in multiple locations is going to want to own their IPs and run an AS, so they can use multiple transit providers, and that means running BGP. You can see public information about this at Hurricane Electric's BGP info, here or on PeeringDB here.
Click to expand...
Fine. That makes more sense. The way it's explained means that many large spread out companies may do this type of setup... e.g. Netflix, Google, etc.

It doesn't explain my point #2 though. If this was a BGP issue, I wouldn't expect it to be a "server" misconfiguration. More like a router misconfiguration. However, it's now sounding like BGP was not the issue to begin with.
 
toast0 said:
I work at Facebook, and I'm not authorized or going to discuss the issue at hand;
Click to expand...

I don't think that anyone asked or wanted to know, however, thank you for sharing that info. Personally, I don't think that Facebook does anything wrong, it's a free-to-join service after all where the users are the product. In my oh-so-humble opinion, the issue lies with people who are stupid enough, join, share their most intimate personal, and allows Facebook to spy on them, and then they bitch about it when they find out that Facebook did something with their personal info that if shouldn't have. Or my favorite is when people complain that Facebook manipulated their decision-making process like I don't know, who to vote with for example.

Since you're here, I want to ask you something, because I'm terrified of installing the Facebook app on my phone: Is it true that the Facebook app listens, and possibly records everything that it can hear over your phone's microphone? If so, I think that this is something that should be disclosed. Yes, I have friends I need to keep in touch with over FB because they're too lazy to use anything else like Signal for example.

Thank you.
 
excuse us while we move all our collected data off server prior to announcing our new 'privacy focused' offerings.
 
Aireoth said:
excuse us while we move all our collected data off server prior to announcing our new 'privacy focused' offerings.
Click to expand...

Shoosh, you'll scare him away before I get a chance to find out if Facebook spies on us via our smartphone's microphone.
 
STEM said:
Since you're here, I want to ask you something, because I'm terrified of installing the Facebook app on my phone: Is it true that the Facebook app listens, and possibly records everything that it can hear over your phone's microphone? If so, I think that this is something that should be disclosed. Yes, I have friends I need to keep in touch with over FB because they're too lazy to use anything else like Signal for example.
Click to expand...

The myth that the FB app hot mics and listens to everything has been disproven and Facebook themselves deny this. Though it definitely seems suspicious at times when you converse about something and suddenly see advertisements for it, there is apparently some algorithmic magic going on between shopping history of you and your friends circle as well as geo-tagged info, posting topics, likes, ad engagements, etc... that it uses to dish out targeted ads that appear so specific so as to make one think it's simply spying.

I still think it listens though...
 
STEM said:
Since you're here, I want to ask you something, because I'm terrified of installing the Facebook app on my phone: Is it true that the Facebook app listens, and possibly records everything that it can hear over your phone's microphone? If so, I think that this is something that should be disclosed. Yes, I have friends I need to keep in touch with over FB because they're too lazy to use anything else like Signal for example.
Click to expand...

There's an official response to this, here. To the best of my knowledge, that's a truthful statement, but I don't really work in that part of the company. It's less convenient, but the mobile website version of FB is pretty reasonable, and being in a webbrowser gives you more control over what it can do, I think you can do messaging, but I don't think you can get push messages.

Sorry, CaptNumbNutz, I understand your question, but I can't answer it; a clear statement from Facebook addressing that would be nice; I could really only find a third party report from NetScout that was mentioned in the earlier [H] coverage and another from ThousandEyes. Between that and 'server configuration error' being enormously vague, you'll have to make up your own mind. Edit, according to another 3rd party tweet, NetScout claims their initial claim was a miscommunication.
 
Jim Kim said:
Facebook, in and of itself, is a server misconfiguration.

The real news here is Google gmail went down with nary a peep about the cause.
https://www.cnet.com/news/gmail-is-down-outage-around-the-world-for-some-users/
Click to expand...

Yeah, but Google is cool :D /s

toast0 said:
There's an official response to this, here. To the best of my knowledge, that's a truthful statement, but I don't really work in that part of the company. It's less convenient, but the mobile website version of FB is pretty reasonable, and being in a webbrowser gives you more control over what it can do, I think you can do messaging, but I don't think you can get push messages.

Sorry, CaptNumbNutz, I understand your question, but I can't answer it; a clear statement from Facebook addressing that would be nice; I could really only find a third party report from NetScout that was mentioned in the earlier [H] coverage and another from ThousandEyes. Between that and 'server configuration error' being enormously vague, you'll have to make up your own mind. Edit, according to another 3rd party tweet, NetScout claims their initial claim was a miscommunication.
Click to expand...

Thank you, that was a good answer :)
 
I misread the title and saw "outrage" instead of "outage".

I realized that's because MainMedia is just a constant torrent of someone being outraged by something, so mental-autocorrect went there.
 
You must log in or register to reply here.
Back
Top