A "Server Misconfiguration" Was Behind the Facebook Outage

Discussion in 'HardForum Tech News' started by AlphaAtlas, Mar 15, 2019.

  1. AlphaAtlas

    AlphaAtlas [H]ard|Gawd Staff Member

    Messages:
    1,716
    Joined:
    Mar 3, 2018
    Facebook and Instagram were down for about 14 hours earlier this week, and earlier reports suggested that the underlying cause was a BGP routing issue. As time went on without an official explanation, many started to suspect that the outage was related to an attack on the platform. But yesterday, Facebook tweeted that the trouble was a "result of a server configuration change." There's still no mention of the incident in Facebook's official newsroom, and some news outlets are starting to criticize the timeliness of Facebook's response and explanation. One security analyst told the BBC that "Facebook's motto always used to be 'move fast and break things'. That's fine when you're an innovative start-up, but when billions of people are using your site every month it's not a good way to run the business."

    Yesterday, as a result of a server configuration change, many people had trouble accessing our apps and services. We've now resolved the issues and our systems are recovering. We're very sorry for the inconvenience and appreciate everyone's patience.
     
  2. TonyZ

    TonyZ Limp Gawd

    Messages:
    173
    Joined:
    Jul 29, 2004
    Facebook, the platform of idiots.... You couldn't pay me to sign up for that...
     
  3. The Mad Atheist

    The Mad Atheist Gawd

    Messages:
    904
    Joined:
    Mar 9, 2018
    Shame it wasn't an attack.
     
  4. Darunion

    Darunion 2[H]4U

    Messages:
    3,679
    Joined:
    Oct 6, 2010
    I use facebook, but seriously didnt know anything about it until i saw it stated here. No mention from any of my friends either. Now if my WoW connection goes down imma flip some damn tables #nerdrage....
     
    Sulphademus likes this.
  5. jardows

    jardows [H]ard|Gawd

    Messages:
    1,425
    Joined:
    Jun 10, 2015
    I can't correctly remember, but wasn't there another, very significant outage with a major organization that was caused by "server misconfiguration" in the last couple of years?
     
  6. the-one1

    the-one1 2[H]4U

    Messages:
    2,983
    Joined:
    Jan 16, 2003
    FB goes down for 14 hrs, people freak out. FB gets into trouble for selling data, which is how they make money, people freak out.
     
  7. RealBeast

    RealBeast Gawd

    Messages:
    651
    Joined:
    Aug 4, 2010
    Those who still find Facebook relevant (octogenarians) would not know how. ;)
     
  8. Sonicks

    Sonicks [H]ard|Gawd

    Messages:
    1,416
    Joined:
    Jul 24, 2005
    I thought we had crappy Change Control at work.

    14 hours is a long time to pin-point and roll back configuration changes....
     
    steakman1971 and SamuelL421 like this.
  9. CaptNumbNutz

    CaptNumbNutz [H]ard as it Gets

    Messages:
    19,416
    Joined:
    Apr 11, 2007
    Although a BGP configuration issue sounds a like a plausible cause, I don't buy it because they keep saying "server misconfiguration".

    1. Why is Facebook doing anything BGP related? Border Gateway Protocol is used by ISP's and internet backbone providers to route between network providers. Facebook is neither.
    2. Why is anyone doing anything BGP related on a "server"? BGP is usually handled by routers and layer 3 managed switches.
     
  10. Exavior

    Exavior [H]ardForum Junkie

    Messages:
    9,678
    Joined:
    Dec 13, 2005
    you are mixing and matching different issues. The BGP issue was reported as an ISP having misconfigured their router and incorrectly told the world they managed that block of IPs. Resulting in what was thought to then be both routers fighting and taking control back and forth.

    they found that wasn't the issue and that they fucked up a server. Two different possible issues caused by two groups.
     
    CaptNumbNutz likes this.
  11. CaptNumbNutz

    CaptNumbNutz [H]ard as it Gets

    Messages:
    19,416
    Joined:
    Apr 11, 2007
    Gotcha... I think I misread that. It read to me like FB was originally claiming a BGP issue, when it was others that were claiming it.

    Thanks.
     
  12. STEM

    STEM Gawd

    Messages:
    583
    Joined:
    Jun 7, 2007
    I find it beyond hilarious that they claim self-sabotage. It looked more like... they were making some internal changes on a very large scale. Considering how distributed Facebook is, for so many errors to happen continuously, it must be a massive outage. They will never tell us the truth, and I honestly don't care. One can only hope that this network of idiots will be gone from this earth one day.
     
    CaptNumbNutz likes this.
  13. toast0

    toast0 Gawd

    Messages:
    895
    Joined:
    Jan 26, 2010
    I work at Facebook, and I'm not authorized or going to discuss the issue at hand; but FB runs BGP because of its large network of Points of Presence throughout the world, and the peering and transit relationships that involves. Really, anyone operating a service on hardware they own in multiple locations is going to want to own their IPs and run an AS, so they can use multiple transit providers, and that means running BGP. You can see public information about this at Hurricane Electric's BGP info, here or on PeeringDB here.
     
    viscountalpha and CaptNumbNutz like this.
  14. CaptNumbNutz

    CaptNumbNutz [H]ard as it Gets

    Messages:
    19,416
    Joined:
    Apr 11, 2007
    Fine. That makes more sense. The way it's explained means that many large spread out companies may do this type of setup... e.g. Netflix, Google, etc.

    It doesn't explain my point #2 though. If this was a BGP issue, I wouldn't expect it to be a "server" misconfiguration. More like a router misconfiguration. However, it's now sounding like BGP was not the issue to begin with.
     
  15. STEM

    STEM Gawd

    Messages:
    583
    Joined:
    Jun 7, 2007
    I don't think that anyone asked or wanted to know, however, thank you for sharing that info. Personally, I don't think that Facebook does anything wrong, it's a free-to-join service after all where the users are the product. In my oh-so-humble opinion, the issue lies with people who are stupid enough, join, share their most intimate personal, and allows Facebook to spy on them, and then they bitch about it when they find out that Facebook did something with their personal info that if shouldn't have. Or my favorite is when people complain that Facebook manipulated their decision-making process like I don't know, who to vote with for example.

    Since you're here, I want to ask you something, because I'm terrified of installing the Facebook app on my phone: Is it true that the Facebook app listens, and possibly records everything that it can hear over your phone's microphone? If so, I think that this is something that should be disclosed. Yes, I have friends I need to keep in touch with over FB because they're too lazy to use anything else like Signal for example.

    Thank you.
     
  16. Aireoth

    Aireoth 2[H]4U

    Messages:
    2,288
    Joined:
    Oct 12, 2005
    excuse us while we move all our collected data off server prior to announcing our new 'privacy focused' offerings.
     
  17. STEM

    STEM Gawd

    Messages:
    583
    Joined:
    Jun 7, 2007
    Shoosh, you'll scare him away before I get a chance to find out if Facebook spies on us via our smartphone's microphone.
     
  18. Sonicks

    Sonicks [H]ard|Gawd

    Messages:
    1,416
    Joined:
    Jul 24, 2005
    The myth that the FB app hot mics and listens to everything has been disproven and Facebook themselves deny this. Though it definitely seems suspicious at times when you converse about something and suddenly see advertisements for it, there is apparently some algorithmic magic going on between shopping history of you and your friends circle as well as geo-tagged info, posting topics, likes, ad engagements, etc... that it uses to dish out targeted ads that appear so specific so as to make one think it's simply spying.

    I still think it listens though...
     
  19. toast0

    toast0 Gawd

    Messages:
    895
    Joined:
    Jan 26, 2010
    There's an official response to this, here. To the best of my knowledge, that's a truthful statement, but I don't really work in that part of the company. It's less convenient, but the mobile website version of FB is pretty reasonable, and being in a webbrowser gives you more control over what it can do, I think you can do messaging, but I don't think you can get push messages.

    Sorry, CaptNumbNutz, I understand your question, but I can't answer it; a clear statement from Facebook addressing that would be nice; I could really only find a third party report from NetScout that was mentioned in the earlier [H] coverage and another from ThousandEyes. Between that and 'server configuration error' being enormously vague, you'll have to make up your own mind. Edit, according to another 3rd party tweet, NetScout claims their initial claim was a miscommunication.
     
  20. Jim Kim

    Jim Kim 2[H]4U

    Messages:
    3,181
    Joined:
    May 24, 2012
  21. STEM

    STEM Gawd

    Messages:
    583
    Joined:
    Jun 7, 2007
    Yeah, but Google is cool :D /s

    Thank you, that was a good answer :)
     
  22. PhaseNoise

    PhaseNoise [H]ard|Gawd

    Messages:
    1,260
    Joined:
    May 11, 2005
    I misread the title and saw "outrage" instead of "outage".

    I realized that's because MainMedia is just a constant torrent of someone being outraged by something, so mental-autocorrect went there.
     
    Sonicks likes this.
  23. HeadRusch

    HeadRusch [H]ard|Gawd

    Messages:
    1,097
    Joined:
    Jun 8, 2007
    I'm calling this a feature, not a bug....