Amazon Outage Breaks Large Parts of the Internet

Megalith

24-bit/48kHz
Staff member
Joined
Aug 20, 2006
Messages
13,003
Something hit the fan with Amazon’s S3 web-based storage service earlier, and as a result, a ton of websites, apps, and devices were/are broken and inaccessible. I read that this was “sweeping the East Coast,” but I am in the Tri-state area and haven’t run into anything so far. How about you?

If your favorite website or internet service isn't accessible today, it's probably not just you. Amazon Web Services' S3 cloud storage is experiencing "high error rates" that have caused chaos among many of the sites that depend on AWS to work. [Engadget] is one of them, but the failure has also affected some or all of Giphy, Medium, Slack, Quora and a slew of other websites and apps you likely use. Some connected home devices aren't working correctly, either. Even Amazon's ability to report problems was broken for a while -- the AWS dashboard wasn't changing color because its issue was "related" to S3's problems.
 

mdaniel

Limp Gawd
Joined
Jan 8, 2004
Messages
344
I noticed a few minutes ago when I turned on my computer that the Razer Synapse app unusually asked me to sign on and then wouldn't take my password. A couple financial sites aren't updating in Quicken either...more than the normal one or two. I'm in Florida.
 

BreezeDM

Gawd
Joined
Jun 25, 2008
Messages
608
A lot more then S3 was down, practically all of US-East. Only like 1 out of 15 EC2 Instances(virtual servers) were up. Right now somethings are up, but S3 puts are down. It is causing my SQS (queue) to fill up. When it comes back it will eat all my burst credits.
 

thesmokingman

Supreme [H]ardness
Joined
Nov 22, 2008
Messages
6,104
Part of Amazon's website services are down, can't access my orders or print return ship labels, etc.
 

raz-0

Supreme [H]ardness
Joined
Mar 9, 2003
Messages
4,705
A bunch of our stuff is down. Of course this happens just as I'm done handing in paperwork to move some of our services into our AWS based virtual machine room.

Gotta love the cloud. Management tells us it is more reliable. So far, not so much. But hey they sign the checks.
 

Zarathustra[H]

Fully [H]
Joined
Oct 29, 2000
Messages
30,704
It was taking me 4 minutes to load the H main page, and I had to reload the forums multiple times in order to post anything earlier today.

Every time my browser would be "waiting for s3.amazonaws.com"

It seems to be working better now though.
 

BreezeDM

Gawd
Joined
Jun 25, 2008
Messages
608
A bunch of our stuff is down. Of course this happens just as I'm done handing in paperwork to move some of our services into our AWS based virtual machine room.

Gotta love the cloud. Management tells us it is more reliable. So far, not so much. But hey they sign the checks.
The president of our company is doing a demo at a conference today.
 
  • Like
Reactions: klank
like this

zomby

Limp Gawd
Joined
Oct 11, 2006
Messages
334
I shop at amazon a lot. If I want to find quality products with good prices, anyone recommend other online stores other than Amazon? not ali express but other popular online stores?
 

Wolfie

Limp Gawd
Joined
Jun 2, 2006
Messages
501
My GarminConnect App will not sync.
a few other cloud based storage systems are not responding.

too soon for an "Alexa Break the internet" joke?
 

BreezeDM

Gawd
Joined
Jun 25, 2008
Messages
608
And the downsides of "moving your business to the cloud" start becoming evident...
I'm no fan of AWS, but I don't think I could have the same yearly uptime percentage if everything was on site. Our ISP uptime alone would prevent that from happening.
 

Methadras

Supreme [H]ardness
Joined
Dec 19, 2000
Messages
6,132
And this is exactly the illustration of the weakness that server/client/cloudbase apps exhibit. Client side apps are still the way to go. I can work offline and not worry about a thing.
 

polonyc2

Fully [H]
Joined
Oct 25, 2004
Messages
18,562
I'm in NYC...a lot of websites I visit were slow to load and had errors earlier this morning
 
D

Deleted member 245375

Guest
All the errors seemed to be in the Northern Virginia region for AWS and yet it affected so much more than just that particular region, it's kinda messed up how that can happen but as usual it's still a "weakest link" kind of situation.
 

Probleminfected

[H]ard|Gawd
Joined
Dec 20, 2013
Messages
1,373
Yep.

H was super slow
Amazon wasn't displaying pictures
Bunch of websites came back as 404

Couple other weird issues.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Management: "We need to move everything to the cloud and get rid of our servers and IT staff, that way it's never going to go down and we can save lot of money!"

Checkmate, people who say stuff like that.

:p
 

Orddie

2[H]4U
Joined
Dec 20, 2010
Messages
2,844
thankfully pornhub was safe. But since other sites were down, i can hardly hold a glass.

And the downsides of "moving your business to the cloud" start becoming evident...
but the small downtime falls within the normal window of downtime typically associated with patches, reboots, & hardware related maintenance. Cost savings of not having staff stay after hours, power, and cooling allows SOME orgs to act like it never happened.
 

S-F

Gawd
Joined
Aug 5, 2010
Messages
671
Fuck "The Cloud". People need to be grownups and get some redundant and high availability infrastructure at home and at a third party location. Shit! I have seen job offers for doing nothing but being prepared for the possibility of this kind of disaster management IN HOUSE.
 

iamjanco

Limp Gawd
Joined
Jul 8, 2016
Messages
458
Got the notice it took down one of our client's sites today just after 1PM EST, one he counts on for about 300k in monthly sales. Wasn't fully back up until about an hour ago. Of course, I wasn't made aware of the outage until about halfway through that, which just made things worse. Add in another hour or so after that to figure out his port 443 settings were blown away by whatever happened, and it wasn't that much fun (though the hours were billable).

Disclaimer: wasn't my idea to go with AWS. Been there in the past, done that.
 

Dead Parrot

2[H]4U
Joined
Mar 4, 2013
Messages
2,831

Krazy925

Supreme [H]ardness
Joined
Sep 29, 2012
Messages
4,646
Not sure if related but full episodes of the walking dead are unviewable on AMC. Maybe just a kwinky dink.
 

Rahh

[H]ard|Gawd
Joined
Jan 14, 2005
Messages
1,607
I envision businesses are going to take a step backwards in the years to come and go back to realizing self-hosting is way better if you have the funds to do it.
 

raz-0

Supreme [H]ardness
Joined
Mar 9, 2003
Messages
4,705
I envision businesses are going to take a step backwards in the years to come and go back to realizing self-hosting is way better if you have the funds to do it.
Probably not IMO. Some info-week reading idiots aside, a lot of existing businesses have to make a financial case to move services to the cloud. Initial roll out of a service may be different, but still likely has to meet a price point. The reality is that the cloud is more expensive unless you do one or more of the following.

1) Avoid building machine rooms.
2) Avoid buying durable hardware for a transient project.
3) Eliminate staff.
4) Avoid adding staff.

Lots of start ups love the cloud because they get 1,2, and 4 out of it. This is a massive reduction in need for capital to get going.
 

oldmanbal

2[H]4U
Joined
Aug 27, 2010
Messages
2,260
EPIC's game servers and logins were all down for a few hours yesterday essentially breaking the ability to play their new games online.
 
Top