Amazon Outage Breaks Large Parts of the Internet

Discussion in 'HardForum Tech News' started by Megalith, Feb 28, 2017.

  1. Megalith

    Megalith 24-bit/48kHz Staff Member

    Messages:
    13,004
    Joined:
    Aug 20, 2006
    Something hit the fan with Amazon’s S3 web-based storage service earlier, and as a result, a ton of websites, apps, and devices were/are broken and inaccessible. I read that this was “sweeping the East Coast,” but I am in the Tri-state area and haven’t run into anything so far. How about you?

    If your favorite website or internet service isn't accessible today, it's probably not just you. Amazon Web Services' S3 cloud storage is experiencing "high error rates" that have caused chaos among many of the sites that depend on AWS to work. [Engadget] is one of them, but the failure has also affected some or all of Giphy, Medium, Slack, Quora and a slew of other websites and apps you likely use. Some connected home devices aren't working correctly, either. Even Amazon's ability to report problems was broken for a while -- the AWS dashboard wasn't changing color because its issue was "related" to S3's problems.
     
  2. mdaniel

    mdaniel Limp Gawd

    Messages:
    344
    Joined:
    Jan 8, 2004
    I noticed a few minutes ago when I turned on my computer that the Razer Synapse app unusually asked me to sign on and then wouldn't take my password. A couple financial sites aren't updating in Quicken either...more than the normal one or two. I'm in Florida.
     
  3. 11EBSCREW

    11EBSCREW n00b

    Messages:
    16
    Joined:
    Nov 14, 2014
    Trello is down :(
     
    xorbe likes this.
  4. BreezeDM

    BreezeDM Gawd

    Messages:
    608
    Joined:
    Jun 25, 2008
    A lot more then S3 was down, practically all of US-East. Only like 1 out of 15 EC2 Instances(virtual servers) were up. Right now somethings are up, but S3 puts are down. It is causing my SQS (queue) to fill up. When it comes back it will eat all my burst credits.
     
  5. thesmokingman

    thesmokingman [H]ardness Supreme

    Messages:
    4,772
    Joined:
    Nov 22, 2008
    Part of Amazon's website services are down, can't access my orders or print return ship labels, etc.
     
  6. raz-0

    raz-0 [H]ardness Supreme

    Messages:
    4,495
    Joined:
    Mar 9, 2003
    A bunch of our stuff is down. Of course this happens just as I'm done handing in paperwork to move some of our services into our AWS based virtual machine room.

    Gotta love the cloud. Management tells us it is more reliable. So far, not so much. But hey they sign the checks.
     
  7. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    27,729
    Joined:
    Oct 29, 2000
    It was taking me 4 minutes to load the H main page, and I had to reload the forums multiple times in order to post anything earlier today.

    Every time my browser would be "waiting for s3.amazonaws.com"

    It seems to be working better now though.
     
  8. BreezeDM

    BreezeDM Gawd

    Messages:
    608
    Joined:
    Jun 25, 2008
    The president of our company is doing a demo at a conference today.
     
    klank likes this.
  9. zomby

    zomby Limp Gawd

    Messages:
    334
    Joined:
    Oct 11, 2006
    I shop at amazon a lot. If I want to find quality products with good prices, anyone recommend other online stores other than Amazon? not ali express but other popular online stores?
     
  10. ruffbytes

    ruffbytes Limp Gawd

    Messages:
    447
    Joined:
    Oct 22, 2015
    It broke all our stuff here at work. :(

    Fortunately, not much to be done about it.
     
  11. Wolfie

    Wolfie Limp Gawd

    Messages:
    501
    Joined:
    Jun 2, 2006
    My GarminConnect App will not sync.
    a few other cloud based storage systems are not responding.

    too soon for an "Alexa Break the internet" joke?
     
  12. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    27,729
    Joined:
    Oct 29, 2000
    And the downsides of "moving your business to the cloud" start becoming evident...
     
    ruffbytes and Ranulfo like this.
  13. BreezeDM

    BreezeDM Gawd

    Messages:
    608
    Joined:
    Jun 25, 2008
    I'm no fan of AWS, but I don't think I could have the same yearly uptime percentage if everything was on site. Our ISP uptime alone would prevent that from happening.
     
  14. caddys83

    caddys83 2[H]4U

    Messages:
    3,315
    Joined:
    Feb 14, 2009
    I can't access assemblergames :(
     
  15. Methadras

    Methadras [H]ardness Supreme

    Messages:
    6,134
    Joined:
    Dec 19, 2000
    And this is exactly the illustration of the weakness that server/client/cloudbase apps exhibit. Client side apps are still the way to go. I can work offline and not worry about a thing.
     
  16. polonyc2

    polonyc2 [H]ardForum Junkie

    Messages:
    16,331
    Joined:
    Oct 25, 2004
    I'm in NYC...a lot of websites I visit were slow to load and had errors earlier this morning
     
  17. Tiberian

    Tiberian DILLIGAFuck

    Messages:
    5,725
    Joined:
    Feb 12, 2012
    All the errors seemed to be in the Northern Virginia region for AWS and yet it affected so much more than just that particular region, it's kinda messed up how that can happen but as usual it's still a "weakest link" kind of situation.
     
  18. AthlonXP

    AthlonXP [H]ard as it Gets

    Messages:
    20,237
    Joined:
    Oct 14, 2001
    This is what I keep saying at work..
     
  19. Probleminfected

    Probleminfected [H]ard|Gawd

    Messages:
    1,360
    Joined:
    Dec 20, 2013
    Yep.

    H was super slow
    Amazon wasn't displaying pictures
    Bunch of websites came back as 404

    Couple other weird issues.
     
  20. Danny Dawg

    Danny Dawg [H]ard|Gawd

    Messages:
    1,611
    Joined:
    Nov 5, 2008
    Noticed slow sites this afternoon and even now.
     
  21. Red Squirrel

    Red Squirrel [H]ardForum Junkie

    Messages:
    9,211
    Joined:
    Nov 29, 2009
    Management: "We need to move everything to the cloud and get rid of our servers and IT staff, that way it's never going to go down and we can save lot of money!"

    Checkmate, people who say stuff like that.

    :p
     
  22. Orddie

    Orddie 2[H]4U

    Messages:
    2,447
    Joined:
    Dec 20, 2010
    thankfully pornhub was safe. But since other sites were down, i can hardly hold a glass.

    but the small downtime falls within the normal window of downtime typically associated with patches, reboots, & hardware related maintenance. Cost savings of not having staff stay after hours, power, and cooling allows SOME orgs to act like it never happened.
     
  23. S-F

    S-F Gawd

    Messages:
    652
    Joined:
    Aug 5, 2010
    Fuck "The Cloud". People need to be grownups and get some redundant and high availability infrastructure at home and at a third party location. Shit! I have seen job offers for doing nothing but being prepared for the possibility of this kind of disaster management IN HOUSE.
     
  24. iamjanco

    iamjanco Limp Gawd

    Messages:
    442
    Joined:
    Jul 8, 2016
    Got the notice it took down one of our client's sites today just after 1PM EST, one he counts on for about 300k in monthly sales. Wasn't fully back up until about an hour ago. Of course, I wasn't made aware of the outage until about halfway through that, which just made things worse. Add in another hour or so after that to figure out his port 443 settings were blown away by whatever happened, and it wasn't that much fun (though the hours were billable).

    Disclaimer: wasn't my idea to go with AWS. Been there in the past, done that.
     
  25. Dead Parrot

    Dead Parrot 2[H]4U

    Messages:
    2,401
    Joined:
    Mar 4, 2013
    The following article should be read at the start of any meeting that involves putting full trust in "The Cloud".

    http://www.computerworld.com/article/2552961/it-management/what--it-worry-.html
     
  26. GuruX

    GuruX Gawd

    Messages:
    639
    Joined:
    Jul 8, 2001
    RedFlagDeals (Canadian Deals Site) was borked. Especially the forums.
     
  27. Krazy925

    Krazy925 2[H]4U

    Messages:
    2,484
    Joined:
    Sep 29, 2012
    Not sure if related but full episodes of the walking dead are unviewable on AMC. Maybe just a kwinky dink.
     
  28. RealBeast

    RealBeast Gawd

    Messages:
    648
    Joined:
    Aug 4, 2010
    Azure teams smiling this morning.
     
  29. almalino

    almalino [H]ard|Gawd

    Messages:
    1,553
    Joined:
    Sep 15, 2009
  30. Rahh

    Rahh [H]ard|Gawd

    Messages:
    1,608
    Joined:
    Jan 14, 2005
    I envision businesses are going to take a step backwards in the years to come and go back to realizing self-hosting is way better if you have the funds to do it.
     
  31. raz-0

    raz-0 [H]ardness Supreme

    Messages:
    4,495
    Joined:
    Mar 9, 2003
    Probably not IMO. Some info-week reading idiots aside, a lot of existing businesses have to make a financial case to move services to the cloud. Initial roll out of a service may be different, but still likely has to meet a price point. The reality is that the cloud is more expensive unless you do one or more of the following.

    1) Avoid building machine rooms.
    2) Avoid buying durable hardware for a transient project.
    3) Eliminate staff.
    4) Avoid adding staff.

    Lots of start ups love the cloud because they get 1,2, and 4 out of it. This is a massive reduction in need for capital to get going.
     
  32. oldmanbal

    oldmanbal [H]ard|Gawd

    Messages:
    2,038
    Joined:
    Aug 27, 2010
    EPIC's game servers and logins were all down for a few hours yesterday essentially breaking the ability to play their new games online.