Office 365 and Teams Down

L

Lakados

[H]F Junkie
2FA
Joined
Feb 3, 2014
Messages
13,991
Someone at Microsoft must have forgotten to pay their Electrical bill because their Datacenter’s are dark.

Outlook and Teams down for the continent.


Quietest day I’ve had in a decade.

Here’s hoping it’s broken tomorrow too, might get caught up with the guys in Forever.
 
This is when I'm thankful we use Google Workspace at the office... it wouldn't be a day off if we used Microsoft, as we'd have to make up for the lost time!
 
Teams is partially working for me -- chats, calls and meetings mostly work but with messed up audio quality and screen share quality, calendar and sharepoint features are all broken, can't see people's pictures/icons/organization info. Can't send animated gifs because that feature is broke lol.
Outlook is completey down for me.
Sharepoint is completely down for me.
I can get into M365 Admin Center, but everything is blank where all my users/licenses/etc should be lol.
Copilot completely down.
It has been this way since about noon Eastern, so 4.5hrs now.
 
Struggling here today. All of our transport customers communicate via email, and the fleet management companies all communicate via email.

I can't send or receive trucking requests or Bills of Lading, I can't request or receive repair authorizations, etc.

Most of these fleet management companies don't even have a way that we can call in to THEM. They can call in to us if they need us, but they have no way of knowing if we need to talk to THEM.

Nightmare...
 
Lakados said:
We’ve got no teams, Dynamics and SharePoint are sketchy as hell.
Click to expand...
Teams still working, just tested a call as well, but Sharepoint is shot (Onedrive seems fine) , haven't tried any of the O365 applications. Do have some people that seem to have problems with Teams so maybe its hit or miss (I only know because they hopped onto webex teams since we have it as well). Oh well, gives me time to test out OpenShift 5 that was just released on some of my spare Sierra Forest servers, should be interesting.
 
lol same.. we've switched to cell/sat comm because now Teams has timed out on us a few times in the last 45min or so.
 
Lakados said:
Someone at Microsoft must have forgotten to pay their Electrical bill because their Datacenter’s are dark.

Outlook and Teams down for the continent.


Quietest day I’ve had in a decade.

Here’s hoping it’s broken tomorrow too, might get caught up with the guys in Forever.
Click to expand...
So nice for me, as I was going to look at my email, what does my boss, or my students have to say.. what's that? Nothing? Oh happy days!
 
Ok after 3 pm, and then school IT just announced there's a problem... guess I wasnt the only one laying low and wanting a quiet day instead if mentioning to IT there is a problem
 
I guess Microsoft didn't pay IBM for the RHEL extended warranty?
In all seriousness sounds like MS needs to rethink their infrastructure a little.

IF it instead turns out to be fallout from CVE-2026-124194. Then I think they might want to explain how something they patched yesterday effected them today... that and why they flagged it as a moderate threat. I don't know might be completely unrelated. Clearly it can't effect their Linux servers, but did it effect some clients so hard that they pulled the plug? I don't know hearing that its related but I don't see how it cold be really. I mean even if their servers were flagging infected messages, it would just stop them you would think. Not sure how that would result in servers going offline. Be interesting to see find out what the issue really was.
 
Still down for me.. about done with work for the day. The list has grown longer of impacted services.

1794533674820.png
 
jobert said:
Crap it is still down for me.
Click to expand...
They put out a very non-informational statement saying that outages may persist for some time as fixes are not "taking as expected." I feel like this has to be a hack or something like that. If it was just a misconfiguration, or hardware loss, that really isn't something that takes all that long to revert or to route to new hardware. The fact that they are having trouble getting it fixed and keep using very cagey language that doesn't say much tells me something core got fucked hard.
 
My office's email has been completely down. If it's still down tomorrow, I'll be using my personal email for a few things. Fortunately, everyone has various local versions of office rather than 365, so that was completely unaffected.

"Update - Microsoft's investigation has found that a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests. Microsoft is preparing a configuration change to address this behavior that can be applied across all affected servers, and they're validating it on a subset of servers before applying it more broadly alongside restarts of the affected infrastructure. In parallel, Microsoft begun reverting a recent network configuration change that correlates with the start of impact and are monitoring to determine whether this contributes to recovery. The temporary traffic limitations Microsoft previously described at https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect are now in effect in specific locations. They are monitoring the affected servers in those locations to confirm whether they remain stable while handling a narrower set of connections, after which they'll gradually reintroduce additional traffic."
 
It's strange because it isn't everything, or everywhere. Like at our work Teams is fine, at least established chats are (not sure about new ones) but Outlook is down.
 
Didn't notice any Teams issues but the Outlook outage is worldwide which is wild. It went down before I went to sleep and it's still happening this morning like wtf.
 
Sycraft said:
They put out a very non-informational statement saying that outages may persist for some time as fixes are not "taking as expected." I feel like this has to be a hack or something like that. If it was just a misconfiguration, or hardware loss, that really isn't something that takes all that long to revert or to route to new hardware. The fact that they are having trouble getting it fixed and keep using very cagey language that doesn't say much tells me something core got fucked hard.
Click to expand...

This is far from the first time that a cloud platform has blown itself up in a way that ends up being a disaster to recover from. Compared to simple 1 server for an application systems large distributed systems are much more resilient against failure; but when they do go down the potential failure modes get far nastier. Just for one example when parts of the system stop talking to each other, or at the boundary between working parts of the system (and for email that includes the rest of the world trying to email Outlook users), you can end up with huge backlogs of messages including retry duplicates (because the first got stuck mid-way through the pipeline before encountering the failure) that need processed that overload the failed systems as soon as they attempt to bring them back online: Best case here is that everything is just lagged to hell for an extended period while draining the backlogged message cues and handling many orders of magnitude more duplicate messages than before. Worst case is when the flood ends up crashing the affected systems and isn't something that can be solved by a simple delete query (ie reply all floods in large companies) because the data is valuable and duplicates aren't trivially detectable.
 
Obviously services have gone down - this thing happens - but it's been a while since we last saw such a high profile and extended outage.
 
I wonder what the amount of productivity ($$) is lost when a large, extended outage such as this happens.. So much happens not only on the surface..but behind the scenes with email.
 
Since we are a Google school, nothing has happened to us. BUT, we do you O365 desktop apps for our admin staff and church staff. Other than them using teams (even though I push them to use Meet) nothing has been affected.
 
Has anyone been following the thread on X?

Seems to me like they still don't know what the underlying issue is, and that they've simply rerouted traffic around the issue in order to start restoring service.
 
Some of my stuff is back today. Status:
- Teams is mostly working with the exception of attachments and search. Status available/away icons not working though.
- CoPilot is back but very slow
- SharePoint is unreachable.
- Outlook and calendar is back.
- Admin Center is reachable but still blank.
 
Hash_Immutable_1337x said:
I wonder what the amount of productivity ($$) is lost when a large, extended outage such as this happens.. So much happens not only on the surface..but behind the scenes with email.
Click to expand...
For every company/individual impacted? Easily tens of billions of dollars just in the US. Just at my workplace trying to figure out workarounds has delayed our schedule which leads to a loss of ~$2M/day.
 
This is the kind of outage that will drive businesses directly to Google. You don't lose major services for over a full day with no clear end in sight.
 
I'd wager it was because of AI and some dipshit executive.

Executive: I just vibecoded Copilot to manage our datacenters. We can now let go thousands of employees.
CEO: Okay, let's see.
Executive: Done.
*** A few moments later ***
Executive: Oh no...
CEO: Nice! It's just a hiccup, I'm sure. I'm giving you a bonus and raise!
 
You must log in or register to reply here.
Back
Top