Global Microsoft outage hits due to CrowdStrike Update Definitions

SPARTAN VI said:
That's gotta be a joke.
Click to expand...

Has to be. Onboarding takes at least a week everywhere these days, if not more.

Otoh, when I was at Facebook, it was pretty common for an intern to take the whole site down every summer, with a little help from broken process.

Although, the latest rounds of whole FB downtime seem to be related to full time devs, with broken process.

This incident seems like a clear case study in why nearly all updates need to be staggered and stopped when a significant number of updated clients don't come back after update. It's one thing if you take down 10% of online customer machines, it's another when you take them all down.
 
erek said:
View attachment 666652
Click to expand...
At this point even the Salvation Army could conquer us. :p

Imagine if this shit happened to iphones, an entire generation would devolve to infancy.....meanwhile the rest of us comfortably over 40 would proceed un-phased and unaware.......

To all of you rocking the IT ADMIN SySOP Roles, We Thank You For Your Service. \m/
 
erek said:
I don’t know if it’s true or not yet
Click to expand...
Quick google fu, some random no name website (afaik) says hoax/satire.

https://www.outlookindia.com/intern...bal-microsoft-outage-in-viral-satirical-video

(Edit) Yeah 100% fake. Here's the photo he used to chop himself over it.

1721410046354.png
 
erek said:
I don’t know if it’s true or not yet
Click to expand...
Not sure if serious.... the green screen effect is a little bit visible yes, but even if it was perfect, this is so obviously a joke.... (the paid twitter account, the content, the idea that a new of this day hire could push in production at a job like that, the taken the afternoon off joke in it etc....)
 
LukeTbk said:
I naively vastly underestimated how much windows system there still was when you are not forced in any way to use it (maybe where the naivety come from), one would have thought in 2024 that big airlines chain terminal, hostel and so on where not Windows anymore..., maybe I underestimate how good Windows is for that kind of stuff...
Click to expand...
You can make Win10 and 11 run on a toaster if you need to.
And the fact a single software update could do this tells you one important thing.
These companies are finally doing their god damned updates!!!
Microsoft has made remotely managing, updating, validating, securing, etc… easy to handle with a small team.
Domained machine with GPO, Intune, and Intel vPro with an EMA server makes this somewhat trivial
Deploy the fix to GPO or Intune, and EMA can remotely issue the boot to safe mode via a console script to any machine with wake on LAN enabled.
So assuming they have the correct tools in place once the fix is determined it could be deployed by one person to 10,000+ machines with a few button clicks.
 

Sanctioned Russia Emerges Unscathed in Global IT Outage​

msmash 20 minutes ago		9
Russian officials boasted on Friday that Moscow was spared the impact of the global IT systems outagebecause of its increased self-sufficiency after years of Western sanctions, though some experts said Russian systems could still be vulnerable. From a report: Microsoft and other IT firms have suspended sales of new products in Russia and have been scaling down their operations in line with sanctions imposed over Russia's war in Ukraine, which Moscow describes as a special military operation. The Kremlin, along with companies from state nuclear giant Rosatom, which operates all of Russia's nuclear plants, to major lenders and airlines, reported no glitches amid the outage that affected international companies across the globe. "The situation once again highlights the significance of foreign software substitution," Russia's digital development ministry said. Russian financial and currency markets also ran smoothly.
 
LukeTbk said:
I naively vastly underestimated how much windows system there still was when you are not forced in any way to use it (maybe where the naivety come from), one would have thought in 2024 that big airlines chain terminal, hostel and so on where not Windows anymore..., maybe I underestimate how good Windows is for that kind of stuff...
Click to expand...

I think Microsoft still has a chokehold on everything client-side. Businesses want to use Microsoft Office / Outlook, and they want to be able to run readily available commercial software that is windows based, and they often prefer to manage a minimal number of different configurations, and thus just roll out windows organization wide for everything. It doesn't hurt that the default configuration for every OEM they have contracts with is usually windows based.

Now backend/server side there is a lot of unix/linux stuff, but in my career spanning some 10 different manufacturing / engineering development companies, outside of the random spoiled executive who gets what he wants, and IT makes an exception for, I have never seen anything but Windows client side in the office.
 
erek said:
The official fix statement,

"To Fix CrowdStrike Blue Screen of Death Simply Reboot 15 Straight Times, Microsoft Says"
Click to expand...
Seriously glad that this kind of fix works

Otherwise the poor support guys will be stressed too much
 
Marees said:
Seriously glad that this kind of fix works

Otherwise the poor support guys will be stressed too much
Click to expand...
Ya, or booting into safe mode to see if the falcon will pull the updated file before the old one BSOD's the system.
 
erek said:
"To Fix CrowdStrike Blue Screen of Death Simply Reboot 15 Straight Times, Microsoft Says"
Click to expand...

Shit, that actually works? I saw that a few hours ago, butt I thought that was just some sort of joke or meme.

How does that even work? Is there some sort of built in automatic roll-back of kernel drivers if you are forced to reboot lots of times?
 
Lakados said:
You can make Win10 and 11 run on a toaster if you need to.
And the fact a single software update could do this tells you one important thing.
These companies are finally doing their god damned updates!!!
Microsoft has made remotely managing, updating, validating, securing, etc… easy to handle with a small team.
Domained machine with GPO, Intune, and Intel vPro with an EMA server makes this somewhat trivial
Click to expand...
What if your server has AMD?
 
Zarathustra[H] said:
Shit, that actually works? I saw that a few hours ago, butt I thought that was just some sort of joke or meme.

How does that even work? Is there some sort of built in automatic roll-back of kernel drivers if you are forced to reboot lots of times?
Click to expand...
Apparently it works if you have a fast network & new version of the file is downloaded
 
pendragon1 said:
if its just replacing a file, any way you can access "c:" should work
Click to expand...
How do you access c: in a system that is not booting 🤔

But apparently there is a work-around
👇
jlbenedict said:
Got called this morning around 7 to hop on a conference bridge to do just that .. fun times..

Apparently there is some sort of script/workaround.. involving putting a group policy in place to set safemode via bcdedit.. and remove the file..
then reverse the bcdedit settings..
Click to expand...
 
Marees said:
How do you access c: in a system that is not booting
Click to expand...
the same way we always have, a live boot of some sort. windows installer, hirens, a linux live disk etc etc. of course this will vary by situation and may not be possible remotely...
also, see post below.
Marees said:
But apparently there is a work-around
Click to expand...
yes, there are a couple now.
 
Zarathustra[H] said:
n the office.
Click to expand...
Inside the office I get, the big airlines/hostels chain computers that end up used to do the actual stuff with reservation to tickets terminal (that will not run office or any application other than the custom reservation system I thought), I thought those were less windows heavy by now.

But like Lakados pointed out, maybe Microsoft got really good at terminals type device management.
 
SPARTAN VI said:
Production alert went out for my company, one of the largest banks in the US. Unfortunately it's having zero direct impact to my operations, but I assume it's going to be a slower day because my customers are impacted outside of my product/application and wont be bothering me today.
Click to expand...

Same here, not really affecting any of my systems outside of one of our journaling vendors being down. Everyone wants to be in the cloud until the cloud goes down because its just someone else's computer
 
Zarathustra[H] said:
Shit, that actually works? I saw that a few hours ago, butt I thought that was just some sort of joke or meme.

How does that even work? Is there some sort of built in automatic roll-back of kernel drivers if you are forced to reboot lots of times?
Click to expand...
I think it just comes more down to luck of the Falcon app what ever reaching out and grabbing the new files before the bad file kills windows.
 
we are still working through it....some major things just came back up for us. Desktop still on the phone with tons of employees to get their laptops working again.
 
Kardonxt said:
This is one of the few days a year that I'm glad my employer's product stack is mediocre. CrowdStrike was on our list of products to evaluate if our per seat budget is ever increased.
Click to expand...
I like SentinelOne, check it out.
It’s “working” for us in that we have it and it haven’t had an attack or incident where it’s services were required.
 
You must log in or register to reply here.
Back
Top