"Other companies may hypothetically do bad things, so I'm going to stick with a company with a proven track record of doing bad things."
Got it.
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Global Microsoft outage hits due to CrowdStrike Update Definitions
- Thread starter erek
- Start date
Got it.
Legendary Gamer
2[H]4U
- Joined
- Jan 14, 2012
- Messages
- 2,166
367 Machines Impacted by this, still getting some trickle in here and there. I worked Friday, Saturday and Monday to get most of em resolved. It's not the worst issue I have ever seen. Just a prior Dell had pushed a BIOS update that bricked over 20 of our Latitude 5420's...
They didn't brick mine, but FUUUUUCK they took a long time to boot after that one, they just hung at a black screen.
I just left them on and was like well I can't deal with this today so I am going to leave them here and deal with it in the morning, and in the morning they were all sitting at a login screen.
I get I was part of the lucky minority on that batch.
The Mad Atheist
2[H]4U
- Joined
- Mar 9, 2018
- Messages
- 3,860
I never noticed an issue, I think I was at work and didn't find out till a day or 2 after it happened.
MrGuvernment
Fully [H]
- Joined
- Aug 3, 2004
- Messages
- 22,366
If you keep moving companies you will eventually run out because every company has done bad things at some level. At this point it is pick the one least worst one.
CS did really screw the pooch on this one as it seems they pushed the update out by passing configs to slow release patches customer had in place, which if true, lawsuits are going to be insane for this one.
No company has done what they've done. I'm not sure what you're defending. The least worst one would be anyone but CrowdStrike.
pendragon1
Extremely [H]
- Joined
- Oct 7, 2000
- Messages
- 62,267
thas a very odd way of defending them....
CrowdStrike reportedly sends $10 Uber Eats gift cards to clients as apology for global IT meltdown
In an email addressed to “CrowdStrike partners” obtained by TechCrunch, the embattled cybersecurity firm apologized for “the additional work the July 19 incident has caused”
“To express our gratitude, your next cup of coffee or late night snack is on us!” said the email, which included a promo code for Uber Eats and was reportedly sent in the name of CrowdStrike’s chief business officer Daniel Bernard...
https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-apology-gift-card-to-say-sorry-for-outage/
In an email addressed to “CrowdStrike partners” obtained by TechCrunch, the embattled cybersecurity firm apologized for “the additional work the July 19 incident has caused”
“To express our gratitude, your next cup of coffee or late night snack is on us!” said the email, which included a promo code for Uber Eats and was reportedly sent in the name of CrowdStrike’s chief business officer Daniel Bernard...
https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-apology-gift-card-to-say-sorry-for-outage/
Are you sure this is not the onion ????????
On Wednesday, some of the people who posted about the gift card said that when they went to redeem the offer, they got an error message saying the voucher had been canceled. When TechCrunch checked the voucher, the Uber Eats page provided an error message that said the gift card “has been canceled by the issuing party and is no longer valid.”
CrowdStrike did not immediately respond to a request for comment.
jlbenedict
2[H]4U
- Joined
- May 22, 2005
- Messages
- 2,468
Perhaps their servers just crashed.
It is conceivable that advanced tracing frameworks like dtrace and eBPF could provide the functionality required here. If not now then you could possibly make them so.
But this goes back to Windows being closed source. In Linux or BSD crowdstrike themselves could expand these frameworks. For Windows their only option is to beg MS and hope for the best. So a kernel module it is.
Delicieuxz
[H]ard|Gawd
- Joined
- May 11, 2016
- Messages
- 1,956
Probably not new information to this thread, but the CEO of CrowdStrike, George Kurtz, was the CTO of McAfee in April, 2010, when it caused a similar, large-scale failure of Windows systems.
View: https://youtu.be/H9RSeDUdkCA
View: https://youtu.be/H9RSeDUdkCA
Legendary Gamer
2[H]4U
- Joined
- Jan 14, 2012
- Messages
- 2,166
The Dell BIOS bricking is still an ongoing issue. I just got removed from that support case by my peers because I'm too much of a hardass with Dell. I have replaced 50 machines en masse in a text chat without jumping though hoops, and this time an executive was in the email chain and thought I was being mean to dell telling them I shouldn't have to send them 10 pictures of the front and backs of 5 Bricked BIOS machines in order to get them replaced and to prove we didn't beat the laptops to death... SO, I am so old school and have zero patience with the foreign tech support teams at Dell that ... I guess I just no longer fit in with modern politics and sentiments in IT departments. I try to get shit done, not sit on my fucking hands for weeks while Dell techs dick around. But I am no longer wanted for that kind of service and support apparently.
As for the Crowdstrike issue, I just had one that didn't manifest until after deployment. The update downloaded during the imaging process and didn't manifest until the user had fully built the machine up and activated it. Once all the software modules loaded and the security profiles hit, took the machine out hard. I just entered the bitlocker code SEVEN times in order to get it to the point where I could reach the command prompt and delete the c-00000291*.sys file. This is the first one in 2 days, so hopefully we got most of em. Though I did see another tech request a bitlocker key a couple min ago...
It was nasty, but we got though it. I need a new job though, I am tried of this place and being an IT contractor here blows.....
Jabroni31169
My Future Son-in-Law
- Joined
- Apr 19, 2000
- Messages
- 10,636
Im with you and I guess im old school too, vendors suck and they should be treated as such until they prove they are not a typical vendor.
That_Sound_Guy
2[H]4U
- Joined
- Apr 29, 2002
- Messages
- 3,261
If this is true, just wow.
"CrowdStrike’s token of appreciation for those vendors: a $10 Uber Eats voucher. We’re not kidding."
https://www.cnn.com/2024/07/24/business/crowdstrike-outage-uber-eats/index.html
"CrowdStrike’s token of appreciation for those vendors: a $10 Uber Eats voucher. We’re not kidding."
https://www.cnn.com/2024/07/24/business/crowdstrike-outage-uber-eats/index.html
Is it
$10 per PC
$10 per organization
$10 per person
Or any other methodology ???
That_Sound_Guy
2[H]4U
- Joined
- Apr 29, 2002
- Messages
- 3,261
Total guess here as my stuff doesn't use CS, but I would assume it's $10 per customer account main contact at CS. I didn't see a copy of this email so I don't know. Per PC/Server or per person would bankrupt them.
This makes AT&T's "here's your $5 credit for not having a cell phone for 10 hours" seem like a deal.
jlbenedict
2[H]4U
- Joined
- May 22, 2005
- Messages
- 2,468
$10 appreciation voucher is a huge f slap in the face. This outage is one of the most unprecedented, large scale outages in IT history.
The losses $$ ... Could they be in the billions?
The losses $$ ... Could they be in the billions?
pendragon1
Extremely [H]
- Joined
- Oct 7, 2000
- Messages
- 62,267
thats been posted a couple times now and the vouchers are invalid
Uber suspected a fraud & deactivated the vouchers
But CS really did pass around vouchers
mullet
2[H]4U
- Joined
- Aug 19, 2004
- Messages
- 2,457
Maybe some of the other IT pro's here in this forum can help you out here? Maybe they could PM you with info. I wish I could, but I'm not an IT pro, and I recently retired. Things move so fast these days that my knowledge of the job market is way out of date.
Legendary Gamer
2[H]4U
- Joined
- Jan 14, 2012
- Messages
- 2,166
Nice of you to say. The job market is so hard to get hired in these days... I am applying but some of it I am gonna have to wait a while for. Applied at Cleveland Cliffs recently, but I heard you typically don't hear back from them for about a year unless they think you're hot shit. I'm not. I'm just an old, tired, technician that was once a Project Manager and got burnt out doing infrastructure work. Now I work as a field services tech/engineer and help desk at BP in NW Indiana. I hate Chicago, almost nothing can get me to take a job there unless it pays more money than god. The drive is nasty, the trains aren't safe and Chicago is a corrupt, dangerous, shithole that's been run into the ground with bad politics.
I'm looking at something called Field Nation, one of the techs here is making 60 bucks an hour working as a 1099 employee... Doesn't sound too bad, even if I don't make that much. https://fieldnation.com/
I would be moving from stability to whatever contract I could be placed in ... Not sure I want to do that , but money talks and ...
erek
[H]F Junkie
- Joined
- Dec 19, 2005
- Messages
- 12,737
Sad about invalid vouchers
Though, CrowdStrike says hackers are threatening to leak sensitive information on adversaries
https://www.nbcnews.com/tech/securi...ning-leak-sensitive-information-ad-rcna163675
erek
[H]F Junkie
- Joined
- Dec 19, 2005
- Messages
- 12,737
“Delta CEO Says CrowdStrike-Microsoft Outage Cost the Airline $500 Million
msmash 5 hours ago
60
Delta Air Lines CEO Ed Bastian said the massive IT outage earlier this month that stranded thousands of customers will cost it $500 million. From a report:Bastian said the figure is representative of not just the lost revenue, but "the tens of millions of dollars per day in compensation and hotels" over a period of five days. The airline canceled more than 4,000 flights in the wake of the outage, which was caused by a botched CrowdStrike software update and took thousands of Microsoft systems around the world offline. The company had to manually reset 40,000 servers, Bastian said. Further reading: Delta Seeks Damages From CrowdStrike, Microsoft After Outage.”
They can seek compensation, that's in their right, but they won't be able to.
The fact it took them so long is a failure on their part, they should have had better management tools in place.
Cloudstrike is going to lose that client either way.
Not sure how any of this is their fault. Client PC's at airports aren't just readily accessible to an IT team. What would your solution be lol
Plenty of solutions exist for that, why are the machines not equipped with an inline IPMI, iDRAC, iLO, BMC, etc... interface?
The PC could be off but as long as it has power you can access it over a secured connection, for remote management, complete with virtual USB, keyboard, and mouse input, you can access system files and BIOS.
From there they could re-image the whole machine if they needed too, but using it to get system file access to delete the one file they needed would be simple.
There are cluster managers available for those interfaces, they take command line input, use the manager to log into all the PC's contained in the cluster script out the removal of that file, and push it out to all the devices in the cluster.
I might only have a few dozen servers, but I have had plenty of updates over the last 2 decades that have mass-borked machines and I can't imagine not using an IPMI interface to deal with those sorts of situations.
Desktops I have the bulk of them doing this through Intel Management Engine, thought that is very limited.
Last edited:
If they do end up court, I don't think cloudstike defense would be 'u should have been better prepared yourself for us to screw up!' lol
But yea they won't have to pay much due to terms/contract etc I would guess. Maybe some refund on what they already paid
And yet they were willingly running Windows with complex extra software inside the kernel.
From a system architecture standpoint it was a matter of when, not if, something like this happened.
Also, Unix like systems would only need a serial console (or a virtual one) to recover from a boot failure due to a panic in an early loaded kernel module, Just making the system headless with no console at all and hoping for the best is not a winning strategy.
Last edited:
Right but what if a windows update had done this, or a routine driver update, or any sort of software update?
What if a huge zero day was discovered and they needed to do mass patching.
This pointed out who does and doesn’t have good disaster recovery tools and procedures in place.
Lots of places got caught with their pants down.
Cloud strikes reputation is basically garbages now, and they are going to lose a lot of customers. But I doubt any of their clients can get much from them. Refund on their remaining contract term and a wave of the early cancellation fees certainly but unlikely anything beyond that.
I mean, if anybody understands a minor change fucking over travel plans it should be them. You would think the airlines would be a lot more understanding of the situation….
There was a Windows update that wiped people's hard drives. Even though CrowdStrike is full of morons, you're right that people need to have better disaster recovery implementations in place. Microsoft famously has horrendous update issues.
erek
[H]F Junkie
- Joined
- Dec 19, 2005
- Messages
- 12,737
CrowdStrike is sued by shareholders over huge software outage
https://www.reuters.com/legal/crowd...holders-over-huge-software-outage-2024-07-31/
The Shareholders hired Pomerantz LLP, and they are investigating claims on behalf of investors of CrowdStrike Holdings, Inc.
The investigation concerns whether CrowdStrike and certain of its officers and/or directors have engaged in securities fraud or other unlawful business practices.
The class action is to recoup costs for CrowdStrike's investors, but it has nothing to do with their clients.
Clients are going to get the shaft, more so now.
Legendary Gamer
2[H]4U
- Joined
- Jan 14, 2012
- Messages
- 2,166
Yeah, I know it hammered BP Globally. We had techs working round the clock trying to unlock systems and restore the business. I am STILL getting walk-ins with Crowdstrike issues as latecomers and remote users show up with their laptops "I have a frowny face on my PC".... Jesus... Normies will be the death of me. He's talking about a BSOD. lol
erek
[H]F Junkie
- Joined
- Dec 19, 2005
- Messages
- 12,737
Always tax payers pay
https://hardforum.com/threads/intel...-during-decompression.2035841/post-1045927935
H/t Red Falcon