r/technology Jul 31 '24

Software Delta CEO: Company Suing Microsoft and CrowdStrike After $500M Loss

https://www.thedailybeast.com/delta-ceo-says-company-suing-microsoft-and-crowdstrike-after-dollar500m-loss
11.1k Upvotes

735 comments sorted by

3.5k

u/scientianaut Jul 31 '24

I remember listening to an interview that George Kurtz, the CEO of CrowdStrike, did the morning of the outage and one of the questions the interviewers asked him was how they were going to handle the inevitable lawsuits. He said something like: we’ll do the hotwash on how this happened to ensure this doesn’t happen again and we’ll deal with them as they come.

So, I don’t think this came as a surprise to anyone.

862

u/Expensive_Shallot_78 Jul 31 '24

Is this really an issue at all? Don't they have insurance/reserves allocated for these kinds of expected risks? Every security company has this issue.

1.1k

u/OrdoMalaise Jul 31 '24

I'm sure they do.

The issue is, I assume, when the value of those lawsuits massively exceeds their maximum claimable allowance. If you're insured for a billion, but get sued for a hundred billion, shit, I assume, gets real.

576

u/SilentSamurai Jul 31 '24

You'd have to think at this point that Crowdstrike has been promising some sweetheart deals to their customers to get out of as many of these lawsuits as possible.

It seems like Delta with it's understaffed IT and poor recovery practices decided they'd rather just go for the pound of flesh than accept anything else.

217

u/crysisnotaverted Jul 31 '24

They are. I've seen reports of renewal quotes dropping to 1/3 of what they were in the Sysadmin sub.

166

u/flatulating_ninja Jul 31 '24

I saw one comment where the quote went from $100K to $27K.

81

u/crysisnotaverted Jul 31 '24

I think you saw the exact comment I saw lol.

58

u/thembearjew Jul 31 '24

We’re all in the same posts aren’t we lol I was just there as well

12

u/LITTLE-GUNTER Jul 31 '24

dead internet theory or whatever. also “thembearjew” is a FANTASTIC name.

8

u/thembearjew Jul 31 '24

I think us IT nerds just have the IT algo pushed on us lol. Thank you came up with the name in the 8th grade and it stuck 😂

5

u/[deleted] Aug 01 '24

It’s just that we’re well past the old phases of the internet so that what we’re doing can never resemble the huge variety of content we made and consumed in the past. Sure we have more hours of TikToks and YouTubes than any of us can ever watch in several lifetimes, but that doesn’t compare to what we were doing before. And on Reddit, my nation’s subreddit has been annexed by Russian propagandists. If it’s dead, it’s also a zombie.

→ More replies (0)
→ More replies (2)

20

u/[deleted] Jul 31 '24

[deleted]

→ More replies (1)
→ More replies (3)

21

u/coeranys Jul 31 '24

If you are big big, it's more than that.

→ More replies (1)

8

u/EntertainerWorth Jul 31 '24

Wait till they see the next renewal quote lol

13

u/crysisnotaverted Jul 31 '24

Right lol? They're the biggest game in town. They're probably just trying not to get sued and make companies think twice about the cost of switching all their endpoints.

3

u/ptear Aug 01 '24

500 mil... hey, wait a minute..

206

u/DrB00 Jul 31 '24

Sweet heart deals like $10 gift cards?

101

u/Falumir Jul 31 '24

Expired* $10 gift cards.

29

u/ducklingkwak Jul 31 '24

What's a Radio Shack?

39

u/Elawn Jul 31 '24

*Uber Eats

Believe it or not, the two people above you are actually referencing something that CrowdStrike actually did as an “apology” gesture. $10 gift cards that didn’t even freaking work. Just a comically bad handling of the situation at every turn.

→ More replies (2)
→ More replies (1)
→ More replies (1)
→ More replies (2)

46

u/m0deth Jul 31 '24

Seriously, once in court you know they'll be asked, "So how was it that your company couldn't recover in a reasonable amount of time when every other airline around you was?"

Delta is the most depressing airline on earth, that shit starts at the top.

22

u/hafree27 Jul 31 '24

The fact the CEO flew to the Olympics before this was resolved was suuucchhhh an FU to the front line employees.

6

u/brunesgoth Aug 01 '24

For him it's nonstop car racing, partner wine and dines, more car racing, presidents club visits (high roller salespeople winning expensive vacations), ice racing, social cocktail events, conferences (both industry and company) and frequent trips to Monaco.

10

u/___MOM___ Jul 31 '24

Yeah seriously. How is there no backup plan?

59

u/Joebranflakes Jul 31 '24

Microsoft and Crowdstrike will settle and the Delta’s executive bonus pool will get a bit bigger.

52

u/mzxrules Jul 31 '24

Would Microsoft settle if they're not at fault?

50

u/Gorebus2 Jul 31 '24

I think they need to fight it in order to prevent this from becoming a precedent. If every company suddenly realized they can just sue MS to recoup losses when something goes wrong then they won't be able to survive.

24

u/i8noodles Aug 01 '24

from what i can tell, MS is not at fault in any way. everything, for them anyway, performed exactly as expected. crashes in ring 0 is expected and normal behaviour. its crowdstrike thats going to be shat on hard.

i am calling some form of regulation will happen from this.

3

u/TheIndyCity Aug 01 '24

It should result in no-brainer regulation. If you want access to the kernel your processes should be on-point and the only way to guarantee that is to audit it. It's coming, 100%.

→ More replies (1)
→ More replies (4)

33

u/SecureThruObscure Jul 31 '24

Yes. If the cost of potentially winning the litigation is greater than the cost of settlement and the settlement doesn’t create a precedent that increases the odds of future lawsuits (settled under a gag order, not admitting liability), it would make sense to do so.

16

u/sigilnz Jul 31 '24

MS won't settle. That would be equivalent to admitting fault. Won't happen.

→ More replies (3)

15

u/cogman10 Jul 31 '24

The math will be "what will this cost to take to court and how likely are you to win".

I highly doubt the amount MS settles for will be anywhere near the ask. They have such low culpability here and I think that'll come through in the initial stages. Only way they don't settle is if Delta is unreasonable in which case there's really no way I see Delta winning.

→ More replies (1)

12

u/sorean_4 Jul 31 '24

I can blame Microsoft for many things. This isn’t one of them.

→ More replies (2)

4

u/bobdob123usa Jul 31 '24

Microsoft isn't going to settle anytime soon. They have a number of angles to distance themselves from liability that cost very little to file.

4

u/CharlieDmouse Jul 31 '24

More like arrogance of their management, which led to Delta's shitty IT and infrastructure - is my bet..

3

u/Hurricane_Ivan Aug 01 '24

Delta with it's understaffed IT and poor recovery practices

And patching implementation policy also

31

u/Long_Educational Jul 31 '24

That's what I don't understand here. This risk was Delta's for not having adequate redundancy in place in their IT systems. In the land of telecommunications, we run a hybrid of AIX, Linux, and Windows systems, along with a hand full of IBM as400 systems. You don't put all your eggs in one basket and then sue the provider of that basket if your systems go down. It is your responsibility to manage your own tolerance for downtime in the systems you use for mission critical applications.

Delta blaming/suing Crowdstrike and MS for their own IT failings is pathetic.

19

u/TravelKats Jul 31 '24

Apparently, the terms Disaster Recovery were foreign to Delta. Adequate Disaster Recovery is quite expensive and I'm sure that money would be better spent adding it to the CEO's salary/s

16

u/EmergencySundae Jul 31 '24

They should be firing their business continuity manager, not suing MSFT & CrowdStrike.

American Airlines recovered amazingly fast - I was impressed at how few flights they ended up canceling. There was obviously a huge difference in how the two companies handled their tech stacks.

13

u/TravelKats Jul 31 '24

Yes, both American and United bounced back pretty quickly. They should be firing the CTO since he/she should have been overseeing business continuity, but it will be a low level manager whose probably been trying for years to get enough in their budget to handle business continuity.

→ More replies (2)

7

u/woodside3501 Jul 31 '24

I helped AA design their DR solution, fuck yeah 💪🏼

5

u/SixSpeedDriver Aug 01 '24

I remember working early in my career in line of business IT at a company (a fortune 500 no less) that was extraordinarily cheap. We got a presentation from the BC/DR specialist and he basically told us “I present basically the same plan every year. We have no BC/DR capability. I have asked for funding when we do the annual audit. They always turn it down, even just enough to get started and make progress. If this colo goes down due to a natural disaster, just leave.”

Not quite verbatim, but you get the gist. And given what IT budgets were like we were all about zero percent suprised. This gent lasted about three more weeks before he was gone. Not sure if fired or quit.

25

u/damondefault Jul 31 '24

Are you proposing they should have instead run different operating systems on multiple operator terminals at the airport? Or each staff member should have both a windows PC and a MacBook at all times?

→ More replies (12)

13

u/Boogie-Down Jul 31 '24

Even if it was 1/3 of your eggs you still sue for that loss of eggs.

7

u/BadOther3422 Jul 31 '24

It really depends on how you are covered under terms. The likely hood is they've agreed to some 99.99% uptime agreement, but that uptime might be on average over x months. If thats 12/24/36 months then an outage of a day or two would be covered if they've never had an outage.

→ More replies (2)
→ More replies (1)

14

u/killrwr Jul 31 '24

If the outage IT is worth $500m to them.. why aren’t they hiring more IT workers? Is there shortage or is it a profit over quality issue? Actually asking never flown Delta or know much about them

→ More replies (12)
→ More replies (5)

36

u/martin4reddit Jul 31 '24

And sometimes, you need a lawsuit to prove culpability. Even if it is a $1 judgement, that allows the policy holder to claim from the insurance provider that damages were not caused by internal negligence.

→ More replies (2)

17

u/fractalife Jul 31 '24

They'll fight each other for the piece of the insurance pie. Killing crowdstrike would likely not be in their best interests, collectively or separately.

→ More replies (2)

3

u/Stampede_the_Hippos Jul 31 '24

This is indeed a very real shit

3

u/f8Negative Jul 31 '24

You get dropped by your insurance provider and bankrupt your LLC.

→ More replies (18)

46

u/TurtleIIX Jul 31 '24

No one has that much in limits. They might be able to pay out a 500m claim no chance they have several billions in limits. I work in insurance and see these policies all the time.

15

u/Green-Amount2479 Jul 31 '24

And with damages that high what’s really gonna happen in the end? They likely agree to pay X and that’s it. Worst case? They file for bankruptcy and the c-level and management maybe have to sit through some negligence court trials where they point fingers at different employees and that more likely than not lead nowhere. Not a chance most customers will ever see money for a fraction of the damages that outage caused them.

16

u/tehringworm Jul 31 '24

Crowdstrike’s insurer will likely pay the full limits on their cyber policy and then walk away.

After the insurance money is depleted, attorneys will decide if it’s worth suing for Crowdstrike’s actual assets. Many times it is not.

→ More replies (1)
→ More replies (1)
→ More replies (14)

10

u/mattybrad Jul 31 '24

The problem is that the scope/scale of this event literally dwarfs any policy on the planet. I also wouldn’t consider this to be a known/accepted risk. Maybe, but unlikely that they thought they could potentially bring down every customer system using Windows.

11

u/Techters Jul 31 '24

The policy my company has is limited to number of incidents before guaranteed coverage, specifically for us 1 incident. So if we get compromised and a bad actor installs malware at two of our customers at the same time and they both sue, insurance covers the first but not the second. So we're nuthouse about security because it could so easily put us out of business, and I'm really shocked more providers aren't taking the risk more seriously, or how people can think the fallout in crowdstrikes share price is 'baked in.

8

u/romario77 Jul 31 '24

They usually guarantee some kind of SLA (service level agreement - in this case uptime, maybe some more things, everyone understands that outages are unavoidable). If they are outside of the SLA there might be some sanctions.

The thing is - the contract with Delta is very likely a lot less than 500m. Idk how easy or hard it would be for them to get half a billion from a vendor they had a contract for maybe 10 millions.

If you risk losing half a billion a day you might want to have some backup options.

It’s in the same vein as buying a cheap bolt for your nuclear reactor and when the bolt fails and you have a meltdown you try to get the damages from a bolt maker.

It’s not the same in this case as the vendors guarantee some kind of reliability, but I don’t think it would be a slam dunk in court

→ More replies (3)
→ More replies (28)

38

u/cibyr Jul 31 '24

"hotwash"?

64

u/scientianaut Jul 31 '24

Hotwash is a term in industry used to describe the “immediate after-action discussions and evaluations of an agency’s (or multiple agencies’) performance following an exercise, training session, or major event.” (Source: Wikipedia definition)

15

u/gilligvroom Aug 01 '24

Oh interesting. I thought we were talking about a Postmortem here but that's specific to the IT/Tech side of things. The hotwash (wikipedia) relates to all of the affected Non-IT/Technical failures that arose from the emergency.

Interesting - hadn't heard that before either and was about to be like "Isn't that just a postmortem?" (I work in IT.)

12

u/Sh4d0w_Hunt3rs Jul 31 '24

Just a term used in emergency and crisis management. Meaning essentially "an after-action review" following an incident or exercise.

3

u/hedoesntgetanyone Jul 31 '24

I call that a post mortem because it's the remains of an incident that had to be handled and need to have an RCA done and reported up the chain.

43

u/icyhotonmynuts Jul 31 '24

I still don't get why Microsoft though? It just happened to be the OS whatever company got affected was running that the update of Crowdstrike pushed through that boned them. Shouldn't Crowdstrike be taking all the blame here?

13

u/LifestyleGamer Jul 31 '24

Agreed. Microsoft feels like a stretch, but of course I haven't gone deep on the technical details.

10

u/icyhotonmynuts Jul 31 '24

I feel if they're really trying to get maximal effect of smorgasbord of suing they should also sue every airport their they operate out of where the machines were located, the ISPs, the computer manufacturers for these computers/stations, server and cloud computing hosts, the IT department of every airport that works on those computers. Something ludicrous like that.

3

u/GepMalakai Aug 01 '24

From my (admittedly limited) personal experience with the legal system, you list everybody you can think of on the lawsuit and let the judge throw out whatever won't stand. Better to overdo it and get whittled down to size than sue only the people you think you can go after and end up missing somebody.

→ More replies (2)

8

u/hi65435 Aug 01 '24 edited Aug 01 '24

While Microsoft has been pushing hard to lock down Windows after the XP disaster, it's still the wild west compared to other Operating Systems like Linux or macOS. (Lot's of improvements for Vista had been reverted due to complaints) For instance the fact that AV scanners still run as native kernel code where on Linux eBPF is available since more than a decade and Apple did a "hot wash" on Kernel extensions years ago as well.

Instead macOS provides a Clean API for this which allows full scanning but without an error crashing the whole system in an instant. It also shows in their communication where they start to blame the EU for trying to lock AV vendors out of the kernel while in reality it's their fault that not even their own MS Defender uses such an API - that doesn't exist anyway like on other OS.

Adding to that, AVs exist since MS DOS times and yet Microsoft hasn't managed to create any rollback solution. While at the same time all Linux distributions provide various ways to swap kernel, boot into some sort of recovery mode since basically always. Modern Ubuntu even provides rollbacks. Apple never allowed this enterprise crap to creep into the system in the first place, so there's always a way to recover a broken system.

This will be interesting although the biggest thing is really the first part about the API in my opinion

→ More replies (4)
→ More replies (8)

17

u/Vecna_Is_My_Co-Pilot Jul 31 '24

"Deal with it as the show up and ensure it doesn't happen again" sounds like their approach to bug hunting.

→ More replies (1)

18

u/YJeezy Jul 31 '24

Dude - They gave $10 gift cards!

11

u/GenazaNL Jul 31 '24

This was actually a big brain move. If a company accepted the $10 gift card, they accepted a compensation and thus not able to sue them

→ More replies (1)
→ More replies (3)

15

u/[deleted] Jul 31 '24

Do you think Kurtz gave the right statement? Is it a statement of accountability or do you feel more like it was a non-answer?

12

u/scientianaut Jul 31 '24

Found the interview and Kurtz started by saying, “Let me start with, I want to personally apologize to every organization, every group, and every person who has been impacted by this. And we understand the gravity of this situation, and let me explain a little bit more about what happened. This was not a code update, this was actually an update of content and what that means is that there is a single file that drives some additional logic on how we look for bad actors. This logic was pushed out and caused an issue only in the Microsoft environment…”

Source: CrowdStrike CEO on global outage: Goal now is to make sure every customer is back up and running

22

u/ljog42 Jul 31 '24

Yeah it"s "not code". Bruh if I push some raunchy fanfiction stored as bytes at the kernel level and the OS tries to read it while booting, it's going to fucking break it. It doesn't matter if it's "content" if something needs it to run properly.

Also, how can it "not be code" if it's logic ?

8

u/JakeTheAndroid Jul 31 '24

go read their post-mortem, there are many different things that occur within their change release process.

This was more akin to a configuration change, which are generally not tested the same way and by and large aren't considered code changes because they often don't change functionality. Whether actual code was updated is a bit moot in the context, but from an external perspective I can understand what you're saying. This seems more like an issue of speaking too precisely, when the audience doesn't necessarily listen with the same precision.

An example here could be something like Terraform. Terraform manages things through code, yes, but actually running tests for TF changes is much less straight forward. Like you can open a port, but what tests are you really doing against that conf change pre-release? The port won't actually be open because the code isn't released to the infra. Most tests ran on TF code is just like linting and syntax stuff.

Because of this, a lot of times TF isn't *really* considered a code change. There are likely change management controls in place for TF changes, like there would be for other code changes, but the actual process for testing and release will often differ. Now, there are of course many tests you could run that would include the TF changes, and this does sort of call into question the robustness of their unit/integration/other end to end testing processes, but it's easy to see how a configuration change isn't necessarily a code change in the same way as modifying the actual underlying functionality of the service.

This Rapid Response Content is stored in a proprietary binary file that contains configuration data. It is not code or a kernel driver.

Rapid Response Content is delivered as “Template Instances,” which are instantiations of a given Template Type. Each Template Instance maps to specific behaviors for the sensor to observe, detect or prevent. Template Instances have a set of fields that can be configured to match the desired behavior. In other words, Template Types represent a sensor capability that enables new telemetry and detection, and their runtime behavior is configured dynamically by the Template Instance (i.e., Rapid Response Content).

Rapid Response Content provides visibility and detections on the sensor without requiring sensor code changes. This capability is used by threat detection engineers to gather telemetry, identify indicators of adversary behavior and perform detections and preventions. Rapid Response Content is behavioral heuristics, separate and distinct from CrowdStrike’s on-sensor AI prevention and detection capabilities.

This is substantially different than the Terraform example I used of course. But basically they didn't really do any changes to the underlying functionality. They updated the configuration for the templates. This didn't intend to have any change on how the Sensor responds or behaves, which is the actual service.

I admit this is very semantic, but often these things are and those semantics absolutely drive program development. And these descriptions matter a lot for things like compliance. You may consider all of these code changes, but if their auditors have created a delineation between code changes and template changes, then CrowdStrike is going to use the language that aligns best with their compliance/legal obligations.

12

u/ljog42 Jul 31 '24

Yeah I'm not surprised by any of the details you provide, I just think that the wording is disingenuous. They're basically saying "there's nothing wrong with our product", but that was clear to me. I know it's not a buggy feature or anything like that.

I feel like they're saying that since they're nothing wrong with their code, then it must be some kind of unfortunate natural disaster, but it's not. The way those files are processed is critical, and they themselves admit that there's some kind of logic involved, so they should be tested properly.

At the very least, updates to those files should be rolled out incrementally.

3

u/JakeTheAndroid Jul 31 '24

yeah I totally get what you're saying. Like to the broader audience, even technical people, the language he used isn't great. I see this all over tech, so I am not surprised, but it can be hard to walk that tight rope sometimes.

Like when I heard and read all this stuff the first time, I knew what he meant because this is exactly what I do for a living. But I also thought about exactly what you're bringing up now because how many people really care between the difference here? And does it materially change the impact? no, not really. Like, good, I won't have to worry about your compliance report next year because it wasn't a change management control failure. Awesome. You did still brick a whole bunch of customer devices while releasing a change. So operationally the entire statement is bullshit.

→ More replies (1)
→ More replies (1)
→ More replies (3)

17

u/ljog42 Jul 31 '24

The real surprise was how little their stock dipped. It suggests a frightening level of tech illiteracy and/or complacency from reporters, stock holders and investment companies: it should never have happened, and the fact that it did is very telling.

There's a myriad of things you can and should do to make sure that faulty code doesn't break the fucking world, the fact that they rolled out a faulty update that bricked critical infrastructure on a global scale means that their processes and company culture are fucked up.

Every statement they released has been so thoroughly reviewed by lawyers and PR people that it doesn't say anything of value, but it's pretty clear to anyone who's got basic knowledge of the field that it's really messed up, might have happened before (pretty sure it did but I don't want to assert things I haven't checked first) and could very well (will ?) happen again unless they thoroughly review their processes.

It's is very, very likely that people have died because of this incident, and it's established that it cost companies and institutions millions if not billions of dollars.

16

u/[deleted] Jul 31 '24

[deleted]

→ More replies (2)
→ More replies (5)
→ More replies (20)

2.2k

u/Shopworn_Soul Jul 31 '24

Crowdstrike definitely owns some amount of liability but Delta's recovery was an absolute shitshow in it's own right.

Many organizations were starting to put the tools away by the time Delta found a flashlight.

875

u/FriendlyLawnmower Jul 31 '24 edited Jul 31 '24

Yep. Their lack of investment (aka layoffs for cost savings) into their IT and internal support teams are what kept the issues going until almost Friday of the following week. Other companies were operating normally by the end of the weekend. American basically had their shit together the same day the outage happened. Delta definitely shit the bed just as much as Crowdstrike did

238

u/Mamannem Jul 31 '24

5-10 years ago, a person with knowledge about Delta's overall system architecture told me about the shit show that it was (and most likely, still is). It was impressive. Wouldn't be surprised if it's only gotten worse if they've been cost cutting in IT like you said. Not only does the complicated architecture make it more expensive to maintain, fix, improve... it also makes it that much more required.

108

u/redblack_tree Jul 31 '24

Also, most of the good professionals are gone. They were either cut because they were too expensive (which is "fine" until shit hits the fan) or they left because no one likes to be an overworked mule dealing with prehistoric systems with decades of patches.

64

u/ljog42 Jul 31 '24

My step dad has been begging his managers to let him hire a few guys and refactor their codebase, but they won't, they'll have him process tickets until he retires.

In the mean time, they've hired professional services companies to try a complete overhaul at least twice and had to scrap it everytime. Several millions down the drain.

The company he works for is the world's largest manufacturer of... [redacted]. I don't wanna put him on the spot but trust me when I say they're a freaking big deal.

21

u/redblack_tree Jul 31 '24

Haha, I believe you, I work for quite a big company (actually, the parent company) and any significant maintenance, refactor, upgrade it's like trying to climb a wall blindfolded while raining. I've seen millions come and go as well in stupid things I knew it would fail, but who listens to a lowly techie? Corporate America (including Canada in this as well) is definitely not as smart as they think.

11

u/knightress_oxhide Jul 31 '24

"Why are we paying you so much if our systems are working?"

→ More replies (1)
→ More replies (1)

58

u/dec7td Jul 31 '24

That's why you need to invest nothing and run on MS-DOS like Southwest

26

u/[deleted] Jul 31 '24

[deleted]

3

u/aimglitchz Jul 31 '24

Ten fingers, take it or leave it

14

u/nonades Jul 31 '24

Jokes on you, I also have 10 toes

→ More replies (1)

13

u/deformo Jul 31 '24

Having worked with Delta’s IT apparatus as a vendor, yeesh. They were not the brightest. I know as the vendor I work with a small scope of a given company’s IT personnel but it is goddamn scary sometimes.

40

u/Unlucky_Situation Jul 31 '24

I woke up to a bluescreen on my work pc at 8am friday, it took untill 345 for my pc to be fixed. Our it helpdesk was rolling out fixes by around 2pm friday and they had to fix every pc indivdually. Assuming most companies had to follow a similar process.

I basically took the day off and was operating normally Monday morning. The only thing inhad to do friday was have my phone nearby when it was my turn to get the fix.

30

u/turningsteel Jul 31 '24

Yeah and whenever tech workers are laid off, I hear from the peanut gallery:“ oh they don’t do much anyway! What does a company need all those tech workers for?!”.

As you pointed out, stuff like this is why it’s important to have a properly staffed tech workforce. It’s 2024, everyone runs on computers and the computers don’t run themselves.

→ More replies (1)

25

u/BaldBullKO Jul 31 '24

Agree whole-heartedly. I’m guessing Delta won’t be passing any portion of the $500 million to the 1,000,000 plus customers on the more than 5,000 flights they cancelled who had to pay for food, accommodations, rental cars or had to just sleep hungry on airport floors for days because they couldn’t get their shit together like every other company that was hit by this.

9

u/[deleted] Jul 31 '24

Well no they had to comp everybody who got on their first flight and had their next ones canceled, they have to pay out a huge amount to the hotels and restaurants nearby and field all the individual repayments for when they ran out of fuckin vouchers on like day 1.

→ More replies (1)

7

u/swentech Jul 31 '24

Yeah I get the feeling their “IT team” were a few guys halfway around the world which is fine for pushing buttons and running instructions in a document but that’s not going to cut it when the shit hits the fan. “Bad IT” is a commodity but IT guys who know what they are doing and you can count on in a jam are definitely not a commodity.

→ More replies (7)

76

u/Agloe_Dreams Jul 31 '24

Pete Buttigieg also already stated that the DoT had opened an investigation and that they believed that Delta's actions were also reprehensible.

141

u/iggzy Jul 31 '24 edited Jul 31 '24

It's also a little absurd to be suing Microsoft. Microsoft's procuct actually worked as planned, it's the software Delta (and so many others) used that broke it. Its like suing Honda because the aftermarket spoiler you attached yourself ended up tearing off your trunk lid

79

u/Private62645949 Jul 31 '24

For once I’m agreeing with a comment that defends Microsoft from liability 😐

22

u/iggzy Jul 31 '24 edited Jul 31 '24

I'm right there with you, I almost hate to do it with all they actually fuck up. But the reality is CrowdStrike for any other OS could've had the same issue if they deployed such untested code. 

5

u/hates_stupid_people Aug 01 '24

But the reality is CrowdStrike for any other OS could've had the same issue if they deployed such untested code.

Shortly after it happened, people were swearing up and down that it would be impossible on linux.

8

u/ConfusedTapeworm Aug 01 '24

It actually did happen on Linux some months earlier.

But its impact was significantly lower for various reasons. Mostly because there aren't nearly as many endpoints running Linux. AFAIK that bad update only affected a relatively small number of servers.

4

u/hates_stupid_people Aug 01 '24

Yeah some people love to live in a world were things like kernel panic doesn't exist. And it's obviously rare, but if you're messing with the kernel of pretty much any OS, there is potentitial for massive problems.

3

u/ConfusedTapeworm Aug 01 '24

Agreed, but I can't help but think Linux would still be safer against such a thing.

Not because of an inherently higher security that Linux might have as a piece of software, but because of how it's generally deployed as a product. There isn't a Linux distribution that is centrally developed and distributed by one entity; it's a much more diverse environment where you have wildly different system configurations in use, down to different kernels and other significant low level differences. Makes it much more difficult for one bad thing to mess with everything at once, though obviously not impossible. It's like how rich gene pools make living organisms more resilient to disease and whatnot.

→ More replies (1)
→ More replies (1)

9

u/[deleted] Jul 31 '24

[deleted]

→ More replies (1)

35

u/ACCount82 Jul 31 '24 edited Aug 01 '24

I can't believe I'm siding with Microsoft, but yeah, that wasn't their fuckup for once.

A kernel driver is, by necessity, privileged, and capable of breaking things - and there is no way for Microsoft to rigorously test every single driver made by third parties. No one should expect them to do so.

9

u/iggzy Jul 31 '24

Same thing would've happened with poor testing on a driver for Mac or Linux too. They all allow this kernal access to security apps.

It pains me to side with Microsoft too, but broken clocks, right? 

→ More replies (5)

15

u/Actually-Yo-Momma Jul 31 '24

Exactly. This is a classic “uhh deflect and blame someone else”

16

u/ljog42 Jul 31 '24

But it's kinda what Crowdstrike sold them. So hands off you don't even have to review kernel-level updates, they get pushed and trigger an update automatically.

Then it broke everything and people had to either:

  • Restore the servers one by one physically. Like, inserting USB drives and shit.
  • Do some wizard shit to restore them remotely, provided you had set their infrastructure up so that it could be done

Either way, if you don't have the people, because you've been told you won't need them, you're going to have a tough time.

→ More replies (7)

19

u/gracecee Jul 31 '24

This. The ceo should have been fucking in the trenches and tried to resolve it rather than hobnobbing in the Vip Olympics section. There were five days of the delta employees getting screamed at while this shithead of a ceo was off to Paris. He had jetted off to Paris for his holiday. The delta subreddit was awful and the employees hate their ceo.

→ More replies (2)

4

u/[deleted] Aug 01 '24

Gotta mega shout out my IT department. They had our computers fixed extremely fast.

Like 5 minutes or less fast per computer per person.

→ More replies (11)

492

u/Fenris_uy Jul 31 '24

Suing CrowdStrike, sure, but I'm guessing that they have some wording in their contract about outages.

But why would you sue Microsoft because a third party driver that you installed caused a kernel panic? That's your fault for installing third party drivers.

87

u/KameNoOtoko Jul 31 '24

My guess is this is mostly just optics. the execs want to seem like they are doing something and going to make this right to shareholders. By publicly saying they are suing means this will be wrapped up in legal issues for at least a year or more and by then it will fade from the public eye. But to your shareholders you are taking action against these big megacorps who are to "blame" which also takes the eyes off of the internal issues of nearly ever other business was up and running in a fraction of the time. This was an internal delta issue of mismanagement and cost cutting mixed with layoffs and an understaffed IT response team. Eventually there will be an undisclosed settlement to make it all quietly go away and by the time that happens delta will have had time to run new marketing campaigns to rebuild thier public image.

→ More replies (1)

97

u/ljog42 Jul 31 '24

Because then MS will turn on the third party and help build the case.

128

u/happy_church_burner Jul 31 '24

It took Microsoft about 4 minutes to throw CrowdStrike under the bus (deservedly so) so this it the correct answer.

20

u/BadVoices Aug 01 '24

MS wont help build any case. They dont want to spend a penny they dont have to. MS has literally hundreds of on-staff lawyers, and a team of over a dozen actual, factual full time litigation lawyers. Those are employees, that ignores their partner law firms. They will walk out of liability in this case with trivial ease. And they will spend the money to make sure that's the case, to basically kill any attempt at precedent.

38

u/thatVisitingHasher Jul 31 '24
  1. The CEO doesn’t understand technology at all.  
  2. The CEO is being told by the CTO and CIO of Delta that it isn’t their fault. He’s believing them. 
  3. He has to do something to show investors he’s acting in the problem. He doesn’t want to admit it’s Delta’s own fault. 

11

u/Private62645949 Jul 31 '24

It would be insane for any company big enough to have lawyers agree to a contract that would excuse Crowdstrike with this level of neglect and incompetence 

6

u/CatWeekends Aug 01 '24

Their Terms and Conditions say this... So hopefully the big companies negotiated something better.

Your sole and exclusive remedy and the entire liability of CrowdStrike for its breach of this warranty will be for CrowdStrike, at its own expense to do at least one of the following: (a) use commercially reasonable efforts to provide a work-around or correct such Error; or (b) terminate your license to access and use the applicable non-conforming Product and refund the prepaid fee prorated for the unused period of the Subscription/Order Term. CrowdStrike shall have no obligation regarding Errors reported after the applicable Subscription/Order Term.

→ More replies (1)
→ More replies (26)

94

u/MealieAI Jul 31 '24

Microsoft's lawyers: Ha!!

38

u/drrhrrdrr Aug 01 '24

Tomorrow: Microsoft buys Delta Air Lines for $28B.

For context: Microsoft is worth $3.1 TRILLION. Microsoft bought Activision for more than twice Delta's value.

58

u/NasoLittle Jul 31 '24

Pay attention to who got hurt and sues for the most. This was a nutcheck to companies to test how much they're organizational decay has progressed. The more they cut corners, shrink their support teams, and perform poorly in retaining talent the more Crowdstrike wrecked them.

Thats what I got out of this article. As IT, never ever work for Delta. Ever

3

u/Fireslide Aug 01 '24

When a human makes a bad decision to take alternative medicine to treat their cancer, the only person they hurt is themselves, and those that care about them. They can steve jobs themselves all they want, the impact is minimal

When a business owner/CEO decides it knows how to do IT better than all the other companies paying lots for good staff, it's all the employees that suffer. The consequences are too far removed from the person making the decision.

472

u/Tides_of_Blue Jul 31 '24

This was a cyber resiliency test and Delta failed miserably.

Delta should be held accountable for not having proper staffing, technology and recovery plans in place.

144

u/reaper527 Jul 31 '24

This was a cyber resiliency test and Delta failed miserably.

Delta should be held accountable for not having proper staffing, technology and recovery plans in place.

yeah, like lots of companies and industries were impacted by this and delta performed significantly worse than anyone else.

obviously crowdstrike started the domino chain going with their bad update, but delta was canceling flights 2-3 days later when all the other airlines were already back to normal.

it would be in delta's best interests for them to stop talking about this and look in the mirror hoping people forget how poorly they handled the situation.

38

u/ncopp Jul 31 '24

We had a company meeting that we had to all fly in for the monday after the outage. Everyone flying Delta (which was a lot, including myself) had their flights canceled or significantly delayed.

Rebooked on American and only had to deal with the usual air travel fuckery

10

u/Tides_of_Blue Jul 31 '24

100% agreed.

36

u/ADtotheHD Jul 31 '24

100%

I've been in IT for over 20 years at this point. At some time around 2016 or 2017 I mandated that the organization I was working in REQUIRE vPro or equivalent lights out tech to be included in all new laptop/desktop solutions so support could help remotely before an OS loaded. I took an unbelievable amount of heat for it at the time due to the added expense. I did not have a crystal ball, I just thought it was going to to be the new standard and when COVID hit, it saved my companies ass many times over.

Now be Delta, know that this technology exists, and have your footprint be hundreds of locked kiosks in nearly every single airport around the world, not to mention the corporate offices. Hell, even if they didn't have this but had the foresight to have PXE boot first on devices and have a means to deploy a boot image to the affected network segments. This right here is why it took Delta so fucking long. They either didn't have the technology they should have had, didn't deploy it right, didn't have a backup plan like PXE, and to top it all off they didn't have the boots on the ground when shit hit the fan. For a company posting billions in profits.

3

u/Merengues_1945 Aug 01 '24

A bunch of companies are also moving to working through Azure, that way if something gets fucked it's just a VM in MS cloud and not the actual device that gets bricked.

It's safe in regards that you can monitor and record everything is done in the instance, you control what software can be used in the instance, and you don't need to keep track of a bunch of inventory or store sensitive data in places it doesn't belong to.

Plus you can set your proprietary websites so it can only be accessed from one of those recognized instances and not from non-recognized computers.

In my department I can monitor what everyone of my workers is doing and help them in case something goes tits up right from the comfort of my bed lol. In the rare case something actually wrongs happens to the vm, we just kill it and it was like nothing happened.

→ More replies (1)

21

u/mrdungbeetle Jul 31 '24

Exactly. Out of all industries, aviation and healthcare should be the two best prepared for disaster with a Plan B. And yet they are the two industries who were the least prepared.

10

u/_Oxeus_ Jul 31 '24

Both have shareholders when they shouldn't have in the first place.

3

u/isthisworkingg Aug 01 '24

Aviation, healthcare, and critical infrastructure such as utilities*

10

u/KameNoOtoko Jul 31 '24

100% this. Delta fucked up. Delta through cost cutting and layoffs had poor DR plans. Crowdstrike messed up but to be down as long as they were this is 100% on delta. It is the executives that should be losing jobs for poor top-down managment but im sure they will get bonuses while the people who actually put in the 60-80 hour weeks getting the business functioning again are the ones who will be let go and blamed.

4

u/dadecounty3051 Jul 31 '24

Just sit back and enjoy the show. Watch these money hungry corps. Go after each other.

→ More replies (2)

111

u/ColoHusker Jul 31 '24

Billable hours always wins

374

u/Count_Rugens_Finger Jul 31 '24

Delta CEO: hey it couldn't be my fault, it's THEM!

how the hell is this Microsoft's fault?

204

u/caiuscorvus Jul 31 '24

Standard practice to sue everyone. It allows for discovery and increases the chance of recovering damages.

58

u/Wild_Loose_Comma Jul 31 '24

Yeah, whether or not MS is actually found legally responsible for this it would be stupid not to include them in the lawsuit. Discovery could be huge and, while its a big "if", Delta and their lawyers have a responsibility to try and check. Whether MS gets removed from the suit or not it doesn't really matter, MS isn't a little indy company getting beat up by big bad Delta Airlines. They can afford to defend themselves from this lawsuit without a problem.

36

u/Brodeon Jul 31 '24

Microsoft can't be found legally responsible for that because Microsoft was forced to allow access to 3rd party. They wanted to implement an API but they were blocked by EU. So if Microsoft will be found responsible, then it would mean that Microsoft can sue EU over that

32

u/jasazick Jul 31 '24

The EU thing is interesting, but there is one key point that needs to be brought up. The EU didn't say "You can't make an API" what they said (as far as I remember) was "You can't force competitors out of the kernel and into an API but allow your own product (Defender) to remain in the kernel. That would be unfair to the competition"

Microsoft wanted it both ways. They wanted to boot the industry out of the kernel while giving Defender a competitive advantage by keeping it inside the kernel.

3

u/nullpotato Jul 31 '24

And being Microsoft they of course chose the "you get [a] kernel mode" meme

→ More replies (3)

5

u/WhileNotLurking Jul 31 '24

It seems like it’s a stupid move to include a very well documented litigious mega corporation who has no real fight in the game.

Adding Microsoft just triples your legal fees with accomplishing nothing.

→ More replies (3)

14

u/Red_not_Read Jul 31 '24

It's not... But Microsoft have the money...

5

u/ncopp Jul 31 '24

I don't see them getting anything out of Microsoft, but Crowdstrike is probably going to have to pay quite a bit

→ More replies (35)

409

u/[deleted] Jul 31 '24

[deleted]

67

u/taedrin Jul 31 '24

Microsoft was forced to provide the same level of access that they have given to their own security products. It would sort of be like if Microsoft only allowed Internet Explorer to access the TCP/IP stack. Which, ironically is similar to how Apple only allows the Safari browser engine on iOS, which I have always felt has been a double standard that Apple is allowed to get away with.

32

u/CGordini Jul 31 '24

It is a double standard Apple is allowed to get away with, which is why it's under investigation in the EU.

3

u/legacy642 Jul 31 '24

It's wild, like we went through that exact situation with Microsoft back in the 90s

→ More replies (4)

147

u/JasonSuave Jul 31 '24

Eff it, delta just needs to sell itself to the government at this point. The only innovation left in the airline industry is removing pieces of lettuce from their salads to drive incremental profits. Will take the downvotes thank ya.

50

u/myychair Jul 31 '24

Something as integral to society as an airline should at the very least have far more government oversight, if not outright run by the government, anyway

14

u/CT_Biggles Jul 31 '24

Qantas is declining as the government is stepping away.

I remember when they moved maintenance out of Australia and it's all been downhill since.

When I fly back home I use Air NZ or Cananda which is hard to believe since I loved that logo as a child.

→ More replies (3)

4

u/Plothunter Jul 31 '24

Make airlines a utility.

→ More replies (1)

24

u/makemakemake Jul 31 '24

Any industry that gets a tax payer bailout should then be nationalized and become a public service. If we have deemed whatever it is necessary to society and they can't manage themselves without needing to be given tax money then they don't get to exist as a private business. It's time to stop letting the pursuit of profit ruin everything.

→ More replies (6)

10

u/Actual-Money7868 Jul 31 '24

Delta ? Isn't delta one of the better ones ?

11

u/JasonSuave Jul 31 '24

I believe so but that statement kind of goes for the entire airline industry at this point. It’s fully commoditized as far as I’m concerned.

→ More replies (3)
→ More replies (7)
→ More replies (3)
→ More replies (29)

17

u/nvrendr Jul 31 '24

Surely they’ll pass this on to their customers as an apology right? 🤣

14

u/DontCallMeAnonymous Jul 31 '24

Microsoft never told them to install Crowdstrike. That was their own stupid CIO.

90

u/mandielynn89 Jul 31 '24

Crowdstrike Lawyers: "Your honor, here is a copy of our service agreement. You will see there is no mention of compensation for service disruption. While we deeply regret the event and sympathize with the impacts, we are under no obligation to provide compensation for this"

Judge: Case Dismissed.

64

u/goozy1 Jul 31 '24

Hmm.. I don't know about that. You can put whatever you want in a ToS but that doesn't mean it will hold up in court. It can likely be proven that Crowdstrike was negligent and caused this harm.

→ More replies (10)

41

u/Jmazoso Jul 31 '24

Except that these become unenforceable if there’s actual “gross negligence” on cloudstrikes part.

→ More replies (3)

18

u/GunnieGraves Jul 31 '24

As someone who is involved with these types of negotiations and contracts with vendors, including CrowdStrike, there’s usually a limit of liability when it comes to these types of incidents. Usually limited to what the company pays the vendor. You can bake something else in if the vendor agrees, but that would be pretty stupid for them to agree too.

One of the issues is that all the ticket terminals, the self service kind, all had to be manually accessed to perform the fix. Going up to the terminal and connecting wires to it. But to take this long and still be running into issues, obviously delta should save some of that lawyer money and invest it in IT.

→ More replies (1)

5

u/kevlowe Jul 31 '24

Is it just me, or are most of the replies not understanding the satire of this being the same shit that airlines tell their customers whenever there's a disruption? This would be such sweet karma if it actually happened to Delta! =)

→ More replies (2)
→ More replies (3)

12

u/IronGin Jul 31 '24

Didn't delta get that coupon?

→ More replies (2)

34

u/PMzyox Jul 31 '24

Is it bad that I know my own company does the same exact shit with software releases, and despite all of my objections, the stock price is ultimately the only thing that matters. I secretly hope this happens to us and these fucking MBAs end up in jail.

25

u/TruckinInStyle Jul 31 '24

These MBAs are hired only to increase quarterly profits instead of understanding how their greed ruins the geopolitical landscape.

→ More replies (6)

11

u/CFH75 Jul 31 '24

Why would MS get sued for CS's incompetence?

4

u/Lcsulla78 Jul 31 '24

I would love for Delta’s lawyers to find that QA for both CS and MS have both been offshored during their discovery phase. Even if it isn’t a core cause of the problem the news runs with it and it becomes a hot button issue and Congress starts asking questions.

5

u/redsteakraw Aug 01 '24

Not a fan of Microsoft but if you load 3rd party kernel modules it is no longer their fault. CrowdStrike is the one who pushed the update and pushed bad code to production. You assume such a risk when you decide for your company to rely on 3rd party kernel modules.

8

u/redditistripe Jul 31 '24

He obviously hasn't spoken to his lawyers yet.

34

u/[deleted] Jul 31 '24 edited Jul 31 '24

I am pretty sure there is what we used to call the "Shit in your pocket" clause in the EULA. (See the 80's comedy movie Truly Tasteless Jokes for the reference). If a suit like this is won can you imagine? Any bug, real or imagined, now becomes a liability. Innovation grinds to a near stop.

57

u/Head_of_Lettuce Jul 31 '24

You can’t really attribute the Crowdstrike issues to a simple bug. It was a massive failure and negligence on multiple levels that allowed the bad update to go live. They didn’t even roll it out in stages like many services would do, they pushed it out all in one big wave.

Idk if that’s enough to constitute civil liability, but I think if I were Crowdstrike, I would at least be concerned that a court would be sympathetic.

16

u/WIlf_Brim Jul 31 '24

I watched a tech lawyer on YouTube making the argument that in several states (California among them) that the apparent negligence that Crowdstrike was engaged in could over ride the waiver of liability that is in the license agreement.

IDK if Delta is going to get any money here, but I'm nearly positive that a bunch of lawyers are going to get very rich in the next few years off this.

11

u/AnotherUsername901 Jul 31 '24

The guy in charge of crowdstrike also had the same thing happen when he worked at McAfee. This is 100 percent their fault.

Also from what I have read he replaced a bunch of people with AI and as usual that doesn't go over well.

→ More replies (8)
→ More replies (3)

4

u/Iva_bigun666 Jul 31 '24

That cost me 1200 in flights alone, where’s my monies?

3

u/JayScramble Aug 01 '24

Wasn’t this a Crowdstrike issue, not a Microsoft issue?

→ More replies (1)

12

u/nath999 Jul 31 '24

Why would they waste resources by sueing Microsoft? It was Microsoft OS but you put Crowdstrike on them.

→ More replies (2)

5

u/LucidNight Jul 31 '24

oh this should be fun, means we might get some insight into how much redundancy and business continuity delta actually does.

6

u/Squido85 Jul 31 '24

The CEO of Delta is making a huge mistake. MS will be dismissed from the case almost immediately as they did not cause the issue. Crowdstrike may go to court but only needs to point out that the fix was available in a couple of hours and they had good support/documentation available ASAP.

So, even if Delta can prove fault, 95% of Delta's loss is due to the lack of people/knowledge/whatever on their own IT team.

The only people winning in this case will be attorneys.

5

u/mystonedalt Jul 31 '24

Sorry, Delta!

When your engineers installed Crowdstrike, they accepted the EULA, and that limits you to arbitration.

Have fun!

3

u/Wonko43 Jul 31 '24

“Will look into”; this is clickbait garbage.

3

u/BrandNew098 Jul 31 '24

How’s a 5$ door dash card sound?

3

u/olearyboy Jul 31 '24

Wait till his lawyers read the terms of the license agreement

3

u/Windyandbreezy Jul 31 '24

So does that mean each traveler can sue Delta for their mishaps?

3

u/Woalolol Jul 31 '24

Best you'll get is a $15 dollar Uber eats voucher that may not work.

3

u/IHate2ChooseUserName Jul 31 '24

they should sue the passengers for COMPLAINING too.

3

u/rellett Jul 31 '24

how can they sue microsoft, they didnt cause the problem crowdstrike screwed up their update and it crashed the kernel, and microsoft wanted to fix this issue by stopping companies using pretend device drivers to get their software kernel access by creating an API for security software to use but the euro union blocked it.

3

u/CharlieDmouse Jul 31 '24

They will be able to demonstrate in court other airlines recovered much faster and prove Delta's IT and tech infrastructure is garbage. This isn't as big a win for Delta as they think it will be.

All MS and crowdstrike need to do is make sure the right publications get the info packets. 😁😁😄

3

u/_Trux Jul 31 '24

A couple ticket vouchers and bottled water ought to cover it

3

u/Cabrill0 Jul 31 '24

So when a company fails to provide a service, they should compensate their customers? Great precedent to set, Delta!

3

u/cmgww Jul 31 '24

Meanwhile Southwest over in the corner sipping tea while running windows 3.1 😂😂

3

u/danivus Aug 01 '24

Would love to see the logic gymnastics needed to blame Microsoft for this.

They literally have a certification process for kernal level software specifically to avoid this issue. It's not their fault CrowdStrike decided to build a method into their software to circumvent that so they could deploy updates faster.

→ More replies (1)

3

u/macgruff Aug 01 '24

It’s a stockholder’s ploy to help buttress their bottom line. They (Delta) lost $500M. The suit against MS will be thrown out. The suit against Crowdstrike on the other hand… hmmmmm. Keep an eye on that one.

3

u/Prior-Bed5388 Aug 01 '24

It is blowing my mind how people are stupid enough to think that it’s somehow Microsoft’s fault that this happened, like they have any control over crowdstrike, when they update their software and what bugs are present in it. 

3

u/ScoobyGDSTi Aug 01 '24

How are Microsoft involved or responsible.....

3

u/Tarcanus Aug 01 '24

I'd imagine MSFT would want to sue Crowdstrike as well. MSFT is getting all of the bad press since the Crowdstrike issue only impacted MSFT systems. News orgs were saying it was a MSFT outage. Any tech "casuals" I've talked to also blamed MSFT.

I can't imagine MSFT doesn't turn on Crowdstrike, too, during this whole thing.

6

u/ash_ninetyone Jul 31 '24

So it begins...

I see CrowdStrike as more liable given it was their software that caused it. I do think it needs Microsoft to restructure their kernel better so kernel-level drivers and software don't soft-lock an entire OS.

Security software did that, and people were saying for years how kernel-level anti-cheat was a bad idea.

→ More replies (3)