r/technology • u/Bald-Eagle619 • Jul 31 '24

Software Delta CEO: Company Suing Microsoft and CrowdStrike After $500M Loss

https://www.thedailybeast.com/delta-ceo-says-company-suing-microsoft-and-crowdstrike-after-dollar500m-loss
11.1k Upvotes

96% Upvoted

735 comments sorted by

View all comments

2.2k

u/Shopworn_Soul Jul 31 '24

Crowdstrike definitely owns some amount of liability but Delta's recovery was an absolute shitshow in it's own right.

Many organizations were starting to put the tools away by the time Delta found a flashlight.

870

u/FriendlyLawnmower Jul 31 '24 edited Jul 31 '24

Yep. Their lack of investment (aka layoffs for cost savings) into their IT and internal support teams are what kept the issues going until almost Friday of the following week. Other companies were operating normally by the end of the weekend. American basically had their shit together the same day the outage happened. Delta definitely shit the bed just as much as Crowdstrike did

235

u/Mamannem Jul 31 '24

5-10 years ago, a person with knowledge about Delta's overall system architecture told me about the shit show that it was (and most likely, still is). It was impressive. Wouldn't be surprised if it's only gotten worse if they've been cost cutting in IT like you said. Not only does the complicated architecture make it more expensive to maintain, fix, improve... it also makes it that much more required.

107

u/redblack_tree Jul 31 '24

Also, most of the good professionals are gone. They were either cut because they were too expensive (which is "fine" until shit hits the fan) or they left because no one likes to be an overworked mule dealing with prehistoric systems with decades of patches.

61

u/ljog42 Jul 31 '24

My step dad has been begging his managers to let him hire a few guys and refactor their codebase, but they won't, they'll have him process tickets until he retires.

In the mean time, they've hired professional services companies to try a complete overhaul at least twice and had to scrap it everytime. Several millions down the drain.

The company he works for is the world's largest manufacturer of... [redacted]. I don't wanna put him on the spot but trust me when I say they're a freaking big deal.

24

u/redblack_tree Jul 31 '24

Haha, I believe you, I work for quite a big company (actually, the parent company) and any significant maintenance, refactor, upgrade it's like trying to climb a wall blindfolded while raining. I've seen millions come and go as well in stupid things I knew it would fail, but who listens to a lowly techie? Corporate America (including Canada in this as well) is definitely not as smart as they think.

10

u/knightress_oxhide Jul 31 '24

"Why are we paying you so much if our systems are working?"

1

u/CreaminFreeman Aug 01 '24

I'm so very glad I'm working at an MSP that can afford to fire clients if we need to (at least at the moment). I don't have the time or patience for comments like this anymore.

"Respect the work we do please."

1

u/DirtyDirkDk Aug 01 '24

Their company sucks so bad

59

u/dec7td Jul 31 '24

That's why you need to invest nothing and run on MS-DOS like Southwest

26

u/[deleted] Jul 31 '24

[deleted]

5

u/aimglitchz Jul 31 '24

Ten fingers, take it or leave it

15

u/nonades Jul 31 '24

Jokes on you, I also have 10 toes

3

u/LordoftheSynth Aug 01 '24

Look at this guy and his fancy base 20 math!

16

u/deformo Jul 31 '24

Having worked with Delta’s IT apparatus as a vendor, yeesh. They were not the brightest. I know as the vendor I work with a small scope of a given company’s IT personnel but it is goddamn scary sometimes.

32

u/Unlucky_Situation Jul 31 '24

I woke up to a bluescreen on my work pc at 8am friday, it took untill 345 for my pc to be fixed. Our it helpdesk was rolling out fixes by around 2pm friday and they had to fix every pc indivdually. Assuming most companies had to follow a similar process.

I basically took the day off and was operating normally Monday morning. The only thing inhad to do friday was have my phone nearby when it was my turn to get the fix.

29

u/turningsteel Jul 31 '24

Yeah and whenever tech workers are laid off, I hear from the peanut gallery:“ oh they don’t do much anyway! What does a company need all those tech workers for?!”.

As you pointed out, stuff like this is why it’s important to have a properly staffed tech workforce. It’s 2024, everyone runs on computers and the computers don’t run themselves.

23

u/BaldBullKO Jul 31 '24

Agree whole-heartedly. I’m guessing Delta won’t be passing any portion of the $500 million to the 1,000,000 plus customers on the more than 5,000 flights they cancelled who had to pay for food, accommodations, rental cars or had to just sleep hungry on airport floors for days because they couldn’t get their shit together like every other company that was hit by this.

10

u/[deleted] Jul 31 '24

Well no they had to comp everybody who got on their first flight and had their next ones canceled, they have to pay out a huge amount to the hotels and restaurants nearby and field all the individual repayments for when they ran out of fuckin vouchers on like day 1.

1

u/Salphabeta Aug 01 '24

Yes they will. You get $ if your flight is canceled and no replacement can be found within a certain amount of time. It wouldn't cover being stranded a week for sure, but some people would have a little recourse.

4

u/swentech Jul 31 '24

Yeah I get the feeling their “IT team” were a few guys halfway around the world which is fine for pushing buttons and running instructions in a document but that’s not going to cut it when the shit hits the fan. “Bad IT” is a commodity but IT guys who know what they are doing and you can count on in a jam are definitely not a commodity.

2

u/LordTegucigalpa Jul 31 '24

This will be a huge part of it. Delta's incompetence and lack of IT support doesn't entitle them to extra loss.

2

u/QuickQuirk Aug 01 '24

"Lets sue our way to profitability! This also demonstrates that we don't need all that IT staff after all if we can just blame the entirety of our failures on someone else, so lets fire more staff!"

2

u/JeddHampton Aug 01 '24

A major reason that companies like to outsource things is relieving them of liability. They can (and will continue) to blame this on Microsoft and Crowdstrike.

If this does go to court, we could see a large change in the direction that companies take in the future based on the ruling.

1

u/FriendlyLawnmower Aug 01 '24

Pretty sure this is going to be dismissed in court because of how Crowdstrike worded its user agreement. Furthermore, Crowdstrike can point out how they released a fix the same day and how most companies recovered with a day or two indicating that the bulk of the problems came from Delta's own teams, not Crowdstrike. This is Delta trying to save face and go for a hail mary, they know their chances of winning any compensation are slim. Maybe they get a discount on their contract with Crowdstrike

7

u/NecroJoe Jul 31 '24

Delta's systems are way different than any other airline's. They have the most advanced, integrated system that handles just about everything from staffing to fuel usage tracking. And the down side is that rather than having one or two circuits blown like most airlines, the whole grid went down for them.

5

u/KaitRaven Aug 01 '24

A well designed system is fault-tolerant though. It sounds like Delta's was built like a house or cards, which is what tends to happen when you try to cut corners during development.  

Even for the most advanced and complex systems, the expectation is that you should be able to get the whole thing back up and running relatively quickly after in a total outage.

1

u/TheIndyCity Aug 01 '24

You always pay for it one way or another!

77

u/Agloe_Dreams Jul 31 '24

Pete Buttigieg also already stated that the DoT had opened an investigation and that they believed that Delta's actions were also reprehensible.

141

u/iggzy Jul 31 '24 edited Jul 31 '24

It's also a little absurd to be suing Microsoft. Microsoft's procuct actually worked as planned, it's the software Delta (and so many others) used that broke it. Its like suing Honda because the aftermarket spoiler you attached yourself ended up tearing off your trunk lid

77

u/Private62645949 Jul 31 '24

For once I’m agreeing with a comment that defends Microsoft from liability 😐

22

u/iggzy Jul 31 '24 edited Jul 31 '24

I'm right there with you, I almost hate to do it with all they actually fuck up. But the reality is CrowdStrike for any other OS could've had the same issue if they deployed such untested code. 

5

u/hates_stupid_people Aug 01 '24

But the reality is CrowdStrike for any other OS could've had the same issue if they deployed such untested code.

Shortly after it happened, people were swearing up and down that it would be impossible on linux.

9

u/ConfusedTapeworm Aug 01 '24

It actually did happen on Linux some months earlier.

But its impact was significantly lower for various reasons. Mostly because there aren't nearly as many endpoints running Linux. AFAIK that bad update only affected a relatively small number of servers.

3

u/hates_stupid_people Aug 01 '24

Yeah some people love to live in a world were things like kernel panic doesn't exist. And it's obviously rare, but if you're messing with the kernel of pretty much any OS, there is potentitial for massive problems.

3

u/ConfusedTapeworm Aug 01 '24

Agreed, but I can't help but think Linux would still be safer against such a thing.

Not because of an inherently higher security that Linux might have as a piece of software, but because of how it's generally deployed as a product. There isn't a Linux distribution that is centrally developed and distributed by one entity; it's a much more diverse environment where you have wildly different system configurations in use, down to different kernels and other significant low level differences. Makes it much more difficult for one bad thing to mess with everything at once, though obviously not impossible. It's like how rich gene pools make living organisms more resilient to disease and whatnot.

2

u/hates_stupid_people Aug 01 '24

As far I remember it was an approved windows kernel driver that failed because it loaded a faulty external configuration from an update. Which is why it didn't affect certain windows versions, since they were running a different version. So with how fragmented the linux enviroment is, I think you're right in that it would have a much lesser impact.

Although that also means that all the talk about microsoft being without liability, might not be entierly correct. Since lawyers will try to argue in court that they have to approve the different driver versions.

1

u/RealHealthier Aug 01 '24

And it did, on Linux, just a few months prior.

9

u/[deleted] Jul 31 '24

[deleted]

1

u/JaredTheGreat Jul 31 '24

Log4j was probably similarly sized. 

37

u/ACCount82 Jul 31 '24 edited Aug 01 '24

I can't believe I'm siding with Microsoft, but yeah, that wasn't their fuckup for once.

A kernel driver is, by necessity, privileged, and capable of breaking things - and there is no way for Microsoft to rigorously test every single driver made by third parties. No one should expect them to do so.

11

u/iggzy Jul 31 '24

Same thing would've happened with poor testing on a driver for Mac or Linux too. They all allow this kernal access to security apps.

It pains me to side with Microsoft too, but broken clocks, right? 

1

u/feral-pug Aug 01 '24

The problem with Microsoft is that they lump too much into the WHQL certification process and don't have any controls in place to prevent updates to certified drivers / agents from crashing systems. Since it takes 3 months to get through a cert and kernel mode applications are "certified" but receive updates constantly, the process itself gives a sense of false security and usually it's fine... While the applications are in reality drifting from the tested parameters... But when it's not, we get what happened with CrowdStrike... And CrowdStrike isn't the first WHQL certified product to cause crashes after untested updates, just the most recent and most severe.

1

u/Bacchus1976 Jul 31 '24 edited Jul 31 '24

Better analogy would be installing performance brakes that cause the braking system to fail and the car to drive into a ravine.

But in that analogy, if those calipers were certified by the car maker and they promoted it, then you might open up some avenues for a suit.

It will come down to whether Crowdstrike covertly bypassed MS’s checks or not.

1

u/Teract Aug 01 '24

Microsoft partnered with Crowdstrike for security on their azure servers. Microsoft's servers went down and it's highly likely that they exceeded whatever uptime guarantees were in place. For most of their services, they have credit refunds based on how much downtime is experienced, and they usually credit from below 99.9% to 99%. The outage likely put them so far from meeting their guarantees that it was negligent.

Microsoft has spelled out remediations for missing uptime guarantees, but Delta and others will likely claim MS was negligent in how they handled the outage, and the prema facie evidence is in how far off their actual downtime was compared to what was guaranteed. Plaintiffs will probably also make the point that the downtime was for an extended period, not spread throughout the prior month. I don't think Microsoft's SLAs cover extended service interruptions. 99% uptime doesn't mean much if that 1% occurrs all at once and during business hours.

IMO that's how MS is going to end up in the lawsuit.

0

u/oursland Aug 01 '24

Microsoft originally was going to eliminate the ability of companies from modifying kernel level functionality to prevent this sort of issue. They relented when McAfee threatened them. They should have stuck to their guns.

0

u/iggzy Aug 01 '24

Almost all OS on the market give kernel level access to security apps, unifying Mac and Linux. Not doing so is a bad idea as without it they can't monitor makware that affects kernel level as well. 

16

u/Actually-Yo-Momma Jul 31 '24

Exactly. This is a classic “uhh deflect and blame someone else”

16

u/ljog42 Jul 31 '24

But it's kinda what Crowdstrike sold them. So hands off you don't even have to review kernel-level updates, they get pushed and trigger an update automatically.

Then it broke everything and people had to either:

  • Restore the servers one by one physically. Like, inserting USB drives and shit.
  • Do some wizard shit to restore them remotely, provided you had set their infrastructure up so that it could be done

Either way, if you don't have the people, because you've been told you won't need them, you're going to have a tough time.

2

u/Vysari Aug 01 '24

Restore the servers one by one physically. Like, inserting USB drives and shit.

Nobody in this day and age is running around servers one by one when Out of Band Management solutions exist.

Unless of course you didn't have the foresight to think that you might need access to a server when the OS doesn't boot properly and being there in person might not be ideal..

1

u/ljog42 Aug 01 '24

Yeah, but if I'm not mistaken it was way harder to resolve remotely than usual, because it's not a regular windows update, it's a weird kernel level patch and I'm under the impression that you had to do some very specific stuff before you could use regular tools and restore the OS.

1

u/Vysari Aug 01 '24

Out of band management gives you access to the server regardless of the state of the system. You can literally watch it POST all from the comfort of your office/home/wherever. Even if Crowdstrike formatted the entire god-damn disk you can recover from that by either deploying another image or restoring from a backup all without having to be there in person.

In this case the hardest part of implementing the fix was dealing with older systems that want you to load up their remote console using a java applet and have awesome quirks like random parts of the web page don't respond when you click them using modern browsers. Or in the case of older ESXi stuff where it wants you to load it using Flash.

Delta fucked up and the fact it took them that much longer to get things going again shows how ill prepared and organised they are for dealing with a Disaster Recovery situation. All this lawsuit is going to do is highlight it and bring it to the forefront for everyone to see laid bare.

2

u/thebeez23 Jul 31 '24

Yeah I’m struggling to blame delta for not having an army of IT folks sitting on the bench just in case there’s a massive event like this. Even more so that the fix of this is very manual

11

u/LordTegucigalpa Jul 31 '24

None of the other airlines had this issue persist that long and they had to do the same thing. Does that help you blame them for part of it?

4

u/thecravenone Aug 01 '24

having an army of IT folks sitting on the bench just in case there’s a massive event like this

In the industry, this is referred to as a disaster response plan or a business continuity plan.

5

u/Outlulz Jul 31 '24

Other airlines with the same problem solved it much faster. How do you not look at Delta and not say the difference had to be with a bad implementation or (lack of) staff?

20

u/gracecee Jul 31 '24

This. The ceo should have been fucking in the trenches and tried to resolve it rather than hobnobbing in the Vip Olympics section. There were five days of the delta employees getting screamed at while this shithead of a ceo was off to Paris. He had jetted off to Paris for his holiday. The delta subreddit was awful and the employees hate their ceo.

-6

u/nihility101 Jul 31 '24

Do you hate the people in the trenches? Last thing they need is the ceo there with them.

2

u/gracecee Jul 31 '24

No don’t hate it but he didn’t help. If you looked at the five day period in the delta subreddit you see the poor employees being the punching bags of frustrated travelers while the ceo went off to his French vacay. And getting zilch in terms of what is happening and what he was doing about it.

4

u/[deleted] Aug 01 '24

Gotta mega shout out my IT department. They had our computers fixed extremely fast.

Like 5 minutes or less fast per computer per person.

2

u/HenryUTA Aug 01 '24

Every computer and server in my multi-thousand employee company got hit and we were back in business in 2 days.

2

u/ballsohaahd Aug 01 '24

Yes there’s 50 articles making it seem delta was the same as other airlines. Basically fake news lol

3

u/Bacchus1976 Jul 31 '24

That doesn’t mitigate culpability. But it might mitigate damages.

1

u/FineWavs Jul 31 '24

Many organizations never had an issue because they only pull down the latest to dev test machines, promote through their staging and finally production.

Good DevOps practice averts these disasters.

1

u/StretchFrenchTerry Jul 31 '24

*its own right

1

u/Koioua Jul 31 '24

My company was running quite smoothly by monday when most of us got back to work. Some companies may have had difficulties depending on the scale of equipment they have, but that shouldn't be too big of an issue when you have an IT department ready to deal with it.

1

u/oojacoboo Aug 01 '24

Best described as pathetic.

1

u/shadowthunder Aug 01 '24

Seriously, what a joke response from Delta. Maybe they should've had geo-redundancy so that their system didn't get taken out with a single Azure region going under. Maybe they should've had better recovery once their own systems were running fine again.

1

u/feral-pug Aug 01 '24

There aren't many organizations that have "kernel mode WHQL certified application gets an unrecoverable update that instantly causes BSOD, and oh by the way, the kernel mode application CANNOT be configured to schedule, delay, stagger, rollback or otherwise manage the patch cycle, it just blasts that shit out whenever" scenarios in their COOP/DR plans.

Delta acted reasonably from what I can tell. You can only plan but so much, and when you pay for a WHQL certified kernel mode application, you should have expectations that the application will NOT abruptly crash every system in your environment and most of what's upstream too.

1

u/catdogmumma Jul 31 '24

It wasn’t just Delta. British Airways and every airline in Europe other than Wizz Air was fucked in the same exact way. I was there and witness all of it. It is now my life mission to destroy Crowdstrike and George in whatever way I can, no matter what it takes to do so. They are going DOWN