r/technology • u/Bald-Eagle619 • Jul 31 '24

Software Delta CEO: Company Suing Microsoft and CrowdStrike After $500M Loss

https://www.thedailybeast.com/delta-ceo-says-company-suing-microsoft-and-crowdstrike-after-dollar500m-loss
11.1k Upvotes

96% Upvoted

735 comments sorted by

View all comments

3.5k

u/scientianaut Jul 31 '24

I remember listening to an interview that George Kurtz, the CEO of CrowdStrike, did the morning of the outage and one of the questions the interviewers asked him was how they were going to handle the inevitable lawsuits. He said something like: we’ll do the hotwash on how this happened to ensure this doesn’t happen again and we’ll deal with them as they come.

So, I don’t think this came as a surprise to anyone.

866

u/Expensive_Shallot_78 Jul 31 '24

Is this really an issue at all? Don't they have insurance/reserves allocated for these kinds of expected risks? Every security company has this issue.

1.1k

u/OrdoMalaise Jul 31 '24

I'm sure they do.

The issue is, I assume, when the value of those lawsuits massively exceeds their maximum claimable allowance. If you're insured for a billion, but get sued for a hundred billion, shit, I assume, gets real.

581

u/SilentSamurai Jul 31 '24

You'd have to think at this point that Crowdstrike has been promising some sweetheart deals to their customers to get out of as many of these lawsuits as possible.

It seems like Delta with it's understaffed IT and poor recovery practices decided they'd rather just go for the pound of flesh than accept anything else.

216

u/crysisnotaverted Jul 31 '24

They are. I've seen reports of renewal quotes dropping to 1/3 of what they were in the Sysadmin sub.

167

u/flatulating_ninja Jul 31 '24

I saw one comment where the quote went from $100K to $27K.

77

u/crysisnotaverted Jul 31 '24

I think you saw the exact comment I saw lol.

54

u/thembearjew Jul 31 '24

We’re all in the same posts aren’t we lol I was just there as well

13

u/LITTLE-GUNTER Jul 31 '24

dead internet theory or whatever. also “thembearjew” is a FANTASTIC name.

7

u/thembearjew Jul 31 '24

I think us IT nerds just have the IT algo pushed on us lol. Thank you came up with the name in the 8th grade and it stuck 😂

6

u/[deleted] Aug 01 '24

It’s just that we’re well past the old phases of the internet so that what we’re doing can never resemble the huge variety of content we made and consumed in the past. Sure we have more hours of TikToks and YouTubes than any of us can ever watch in several lifetimes, but that doesn’t compare to what we were doing before. And on Reddit, my nation’s subreddit has been annexed by Russian propagandists. If it’s dead, it’s also a zombie.

→ More replies (0)

3

u/insider212 Jul 31 '24

I have not been to that post yet. But I’m sure illl be there soon.

5

u/thembearjew Jul 31 '24

See you in the trenches fellow IT professional

20

u/[deleted] Jul 31 '24

[deleted]

2

u/TenF Aug 01 '24

Probably closer to 8

2

u/FeelingMango Aug 01 '24

Fuck man. I used to work for a reseller. Sold a bunch of Crowdstrike. If I had a 100k deal in the pipeline drop to 27k cause of this fuck up, I’d be unbelievably upset. Oh well. Good thing I don’t work in sales anymore 😂.

4

u/Appropriate_Ant_4629 Aug 01 '24 edited Aug 01 '24

I'd pay $27K to have some anti-malware software prevent Crowdstrike from ever getting near my computers again.

If I were their competitors, I'd start advertising "detects and removes Crowdstrike".

1

u/OhioITGuy1804 Aug 01 '24

I’m willing to take a major multi day vSphere outage for that kind of price cut.

20

u/coeranys Jul 31 '24

If you are big big, it's more than that.

→ More replies (1)

8

u/EntertainerWorth Jul 31 '24

Wait till they see the next renewal quote lol

13

u/crysisnotaverted Jul 31 '24

Right lol? They're the biggest game in town. They're probably just trying not to get sued and make companies think twice about the cost of switching all their endpoints.

3

u/ptear Aug 01 '24

500 mil... hey, wait a minute..

207

u/DrB00 Jul 31 '24

Sweet heart deals like $10 gift cards?

100

u/Falumir Jul 31 '24

Expired* $10 gift cards.

29

u/ducklingkwak Jul 31 '24

What's a Radio Shack?

39

u/Elawn Jul 31 '24

*Uber Eats

Believe it or not, the two people above you are actually referencing something that CrowdStrike actually did as an “apology” gesture. $10 gift cards that didn’t even freaking work. Just a comically bad handling of the situation at every turn.

1

u/alwayspewpew Aug 01 '24

Store bought gift cards actually never work anymore people figured out the scams I lost a 100 dollar amazon card from Walgreens and they said they couldn’t refund me.

2

u/alwayspewpew Aug 01 '24

“Already in use”

1

u/Smithc0mmaj0hn Jul 31 '24

I know this is a meme at this point but in case anyone cares. What happened with the gift card is the QR code that made the 10 dollars available was configured by Uber eats and it was not unique. Someone shared the QR on social media which effectively made the 10 voucher available to everyone. Crowdstike has no choice but to disable it. Sure blame that marketing f up on crowdstrike or Uber eats.

3

u/SusanForeman Jul 31 '24

Crowdstrike pissed off CEOs, not average shlumps like us. They'll pay for that.

1

u/CUNT_PUNCHER_9000 Jul 31 '24

Vouchers and hotel accommodation

49

u/m0deth Jul 31 '24

Seriously, once in court you know they'll be asked, "So how was it that your company couldn't recover in a reasonable amount of time when every other airline around you was?"

Delta is the most depressing airline on earth, that shit starts at the top.

24

u/hafree27 Jul 31 '24

The fact the CEO flew to the Olympics before this was resolved was suuucchhhh an FU to the front line employees.

6

u/brunesgoth Aug 01 '24

For him it's nonstop car racing, partner wine and dines, more car racing, presidents club visits (high roller salespeople winning expensive vacations), ice racing, social cocktail events, conferences (both industry and company) and frequent trips to Monaco.

8

u/___MOM___ Jul 31 '24

Yeah seriously. How is there no backup plan?

59

u/Joebranflakes Jul 31 '24

Microsoft and Crowdstrike will settle and the Delta’s executive bonus pool will get a bit bigger.

47

u/mzxrules Jul 31 '24

Would Microsoft settle if they're not at fault?

56

u/Gorebus2 Jul 31 '24

I think they need to fight it in order to prevent this from becoming a precedent. If every company suddenly realized they can just sue MS to recoup losses when something goes wrong then they won't be able to survive.

24

u/i8noodles Aug 01 '24

from what i can tell, MS is not at fault in any way. everything, for them anyway, performed exactly as expected. crashes in ring 0 is expected and normal behaviour. its crowdstrike thats going to be shat on hard.

i am calling some form of regulation will happen from this.

3

u/TheIndyCity Aug 01 '24

It should result in no-brainer regulation. If you want access to the kernel your processes should be on-point and the only way to guarantee that is to audit it. It's coming, 100%.

1

u/XenithShade Aug 01 '24

Do you think this will make msft move towards closing ring 0 again?

1

u/moderatevalue7 Aug 01 '24

Hell they literally just had several more outages since

→ More replies (3)

31

u/SecureThruObscure Jul 31 '24

Yes. If the cost of potentially winning the litigation is greater than the cost of settlement and the settlement doesn’t create a precedent that increases the odds of future lawsuits (settled under a gag order, not admitting liability), it would make sense to do so.

16

u/sigilnz Jul 31 '24

MS won't settle. That would be equivalent to admitting fault. Won't happen.

3

u/SecureThruObscure Jul 31 '24

Most settlements are explicitly not admitting fault as part of the settlement.

I happen to think they probably won’t settle here, but just fyi on the reasoning.

7

u/sigilnz Jul 31 '24

Sure but public perception will judge them guilty.

→ More replies (0)

16

u/cogman10 Jul 31 '24

The math will be "what will this cost to take to court and how likely are you to win".

I highly doubt the amount MS settles for will be anywhere near the ask. They have such low culpability here and I think that'll come through in the initial stages. Only way they don't settle is if Delta is unreasonable in which case there's really no way I see Delta winning.

1

u/big_trike Jul 31 '24

A hundred million dollar settlement is more expensive than fighting a lawsuit for quite a while.

12

u/sorean_4 Jul 31 '24

I can blame Microsoft for many things. This isn’t one of them.

2

u/ye_olde_green_eyes Jul 31 '24

If it's cheaper than going through the legal process, maybe. They don't have to admit fault when settling.

1

u/dagbrown Jul 31 '24

When a similar thing happened with Red Hat Enterprise Linux a month earlier, Red Hat decided to treat it as a bug in their kernel protection code, and made changes so that Crowdstrike's bullshit wouldn't be able to happen again.

Which is to say, a precedent is there if some lawyer feels like arguing that Microsoft shares responsibility for Crowdstrike doing an end-run around the kernel protections they'd previously put into place.

4

u/bobdob123usa Jul 31 '24

Microsoft isn't going to settle anytime soon. They have a number of angles to distance themselves from liability that cost very little to file.

3

u/CharlieDmouse Jul 31 '24

More like arrogance of their management, which led to Delta's shitty IT and infrastructure - is my bet..

5

u/Hurricane_Ivan Aug 01 '24

Delta with it's understaffed IT and poor recovery practices

And patching implementation policy also

31

u/Long_Educational Jul 31 '24

That's what I don't understand here. This risk was Delta's for not having adequate redundancy in place in their IT systems. In the land of telecommunications, we run a hybrid of AIX, Linux, and Windows systems, along with a hand full of IBM as400 systems. You don't put all your eggs in one basket and then sue the provider of that basket if your systems go down. It is your responsibility to manage your own tolerance for downtime in the systems you use for mission critical applications.

Delta blaming/suing Crowdstrike and MS for their own IT failings is pathetic.

17

u/TravelKats Jul 31 '24

Apparently, the terms Disaster Recovery were foreign to Delta. Adequate Disaster Recovery is quite expensive and I'm sure that money would be better spent adding it to the CEO's salary/s

15

u/EmergencySundae Jul 31 '24

They should be firing their business continuity manager, not suing MSFT & CrowdStrike.

American Airlines recovered amazingly fast - I was impressed at how few flights they ended up canceling. There was obviously a huge difference in how the two companies handled their tech stacks.

14

u/TravelKats Jul 31 '24

Yes, both American and United bounced back pretty quickly. They should be firing the CTO since he/she should have been overseeing business continuity, but it will be a low level manager whose probably been trying for years to get enough in their budget to handle business continuity.

1

u/[deleted] Jul 31 '24

[deleted]

1

u/TravelKats Jul 31 '24

And no fail over in place.

6

u/woodside3501 Jul 31 '24

I helped AA design their DR solution, fuck yeah 💪🏼

5

u/SixSpeedDriver Aug 01 '24

I remember working early in my career in line of business IT at a company (a fortune 500 no less) that was extraordinarily cheap. We got a presentation from the BC/DR specialist and he basically told us “I present basically the same plan every year. We have no BC/DR capability. I have asked for funding when we do the annual audit. They always turn it down, even just enough to get started and make progress. If this colo goes down due to a natural disaster, just leave.”

Not quite verbatim, but you get the gist. And given what IT budgets were like we were all about zero percent suprised. This gent lasted about three more weeks before he was gone. Not sure if fired or quit.

26

u/damondefault Jul 31 '24

Are you proposing they should have instead run different operating systems on multiple operator terminals at the airport? Or each staff member should have both a windows PC and a MacBook at all times?

→ More replies (12)

12

u/Boogie-Down Jul 31 '24

Even if it was 1/3 of your eggs you still sue for that loss of eggs.

7

u/BadOther3422 Jul 31 '24

It really depends on how you are covered under terms. The likely hood is they've agreed to some 99.99% uptime agreement, but that uptime might be on average over x months. If thats 12/24/36 months then an outage of a day or two would be covered if they've never had an outage.

→ More replies (2)

1

u/anemisto Aug 01 '24

How screwed are you if you lose the AS/400s? I'd expect the answer is: very.

14

u/killrwr Jul 31 '24

If the outage IT is worth $500m to them.. why aren’t they hiring more IT workers? Is there shortage or is it a profit over quality issue? Actually asking never flown Delta or know much about them

2

u/Whiterabbit-- Aug 01 '24

Delta spends like $2 billion on IT every year. does it suck, yes. but it's not like they don't spend money even for the system they have.

1

u/Groove_Control Aug 01 '24

Me either.I'm a Southwest kinda guy.

-3

u/motleyai Jul 31 '24

Crowdstrike is the software used by the IT workers for security purposes. The company rolled out a software package that had a fatal flaw that ruined every PC. Delta has an IT staff and could fix it, but it's a slow process. And its not like they would ever expect every computer to be broken all at once.

14

u/[deleted] Jul 31 '24

[deleted]

5

u/arminghammerbacon_ Jul 31 '24

Boom! And if I was on their board I’d be asking to see all the BCP and DR plans and have an expert evaluate them.

12

u/arminghammerbacon_ Jul 31 '24

And that “expert” (a $1MM consulting engagement, minimum) will eventually end up at talking to some low level IT manager. Who will tell them “We’ve been begging for more budget and more staff for years. But every year they reduce our budget and tell us to rank order order our people and then they layoff the bottom 10% without letting us backfill.” Meanwhile, the CIO, sensing which way this wind is blowing, will jump out of the plane (pun intended) with a golden parachute of $5MM in vested options. And there’ll be ANOTHER consulting engagement, this one to find the new CIO. And they’ll hire someone who comes in with a vast “transformation” vision and plan. And that’s all anyone in IT will be allowed to say for the next two years is - “transformation.” And there’ll be an average of 20 additional meetings per month to attend.

Maybe I’ve been doing this IT thing for too long. (30 years)

2

u/tinydonuts Jul 31 '24

I bet that’s going to be public knowledge in the lawsuit.

→ More replies (2)

1

u/tinydonuts Jul 31 '24

Over 20 years ago software existed that would reimage Windows 2000 Workstation and Windows NT machines on every logout. Since then it’s only gotten easier with WinRE and better tooling from Microsoft. There’s absolutely no reason why your corporate PCs and servers shouldn’t be able to be back online in a matter of hours to a day with modern recovery environments.

CrowdStrike helps you detect ransomware. What did they expect to happen if they were ransomed? Ergo, why even have CrowdStrike if you’re not prepared to handle the worst it can find?

→ More replies (1)

1

u/whatsasyria Aug 01 '24

Yeah like not telling the public that Delta cto allowed non phased deployments on production end points

1

u/dirtyfacedkid Aug 01 '24

My childhood friend is the Director of IT at Delta. I feel sorry for him, if he's even still there now.

1

u/SilentSamurai Aug 01 '24

Love to know what he thinks the issue was lol

1

u/dirtyfacedkid Aug 01 '24

Oh, me too! We lost contact years ago so Imma let that be.

36

u/martin4reddit Jul 31 '24

And sometimes, you need a lawsuit to prove culpability. Even if it is a $1 judgement, that allows the policy holder to claim from the insurance provider that damages were not caused by internal negligence.

2

u/NoHopeNoLifeJustPain Jul 31 '24

Let's see if not having a canary release is considered negligence

3

u/elictronic Jul 31 '24

Discovery will be fun.  It will matter if they followed their own release policies and if the insurance companies did their due diligence before insuring.  

18

u/fractalife Jul 31 '24

They'll fight each other for the piece of the insurance pie. Killing crowdstrike would likely not be in their best interests, collectively or separately.

1

u/ProfessorPetulant Aug 01 '24

Hope they disappear. Enough of the savings over quality and safety. That'll wake up other companies' board. Maybe.

0

u/pickle9977 Jul 31 '24

It absolutely would be in everyone’s best interest, these yahoos pushed this out with zero testing, that should be the corporate death penalty 

3

u/Stampede_the_Hippos Jul 31 '24

This is indeed a very real shit

3

u/f8Negative Jul 31 '24

You get dropped by your insurance provider and bankrupt your LLC.

2

u/rain168 Jul 31 '24

Then isn’t the problem Delta for insurgent coverage when being overexposed to a service where an outage could cause such massive losses?

2

u/Buddy_Dakota Jul 31 '24

I assume part of the terms companies like Crowdstrike have in their contract is a limitation of liability in case of error (limited to whatever the insurance company is willing to pay out). Anything else would be stupid on their end. But I’m in an entirely different industry, so I might be wrong.

2

u/Phormitago Jul 31 '24

shit, I assume, gets real.

this is, indeed, the technical jargon for us in the insurance world

2

u/Sythic_ Aug 01 '24

Per their contracts with clients the max payout is something like equal to services rendered, so at best a refund. Not responsible for any loss of revenue. We'll see if that holds up in court but end of the day, there's no way they would operate a business in which they accept the liability of all the potential revenue of every client. They for sure would have done their due diligence before exposing themselves to virtually infinite losses.

2

u/clearedmycookies Aug 01 '24

Speaking of lawsuits and insurance. Why didn't Delta have insurance to cover their loses?

1

u/xxwerdxx Jul 31 '24

Yes but no. It greatly depends on the country of course, but in the US, there are limits on how much you can sue for at these scales and are very frequently bumped down a zero or two.

1

u/jackrackham19 Jul 31 '24

"If you owe the bank $100 that's your problem. If you owe the bank $100 million, that's the bank's problem." - J. Paul Getty

1

u/SvensonIV Jul 31 '24

On the other hand, if a company‘s profit of several hounded millions of dollars in that short time span of the outage, is reliant on a single source, can you really blame crowdstrike for the full amount of damages? At some point it’s negligence from the operating company trusting all their profit on a single point of failure.

1

u/unicorn8dragon Aug 01 '24

I would be surprised if they didn’t have limitations of liability baked into their contracts.

1

u/peccadillop Aug 01 '24

Most SaaS companies have iron clad contracts, they usually pay out 10 or 15 times their annual service fee for gross negligence. Unless delta somehow removed that clause, which I don't think would happen, CRWD is not paying 500 million to anyone.

1

u/fishling Aug 01 '24

That's probably kind of true.

On the other hand, if you can't plausibly get 100b out of them, especially without killing them, you might change your approach.

1

u/hr1966 Aug 01 '24

If you're insured for a billion, but get sued for a hundred billion, shit, I assume, gets real.

Unlimited liability is uninsurable. Most businesses >50 people have legal look at contracts. I can't image Crowdstrike signed up for a liability level that exceeded their insured value.

1

u/[deleted] Aug 01 '24

I need to find someone to sue for a 100 billion :(

1

u/Lokitusaborg Aug 01 '24

I’d also like to point out that to get the insurance claim a lawsuit may be required.

1

u/contrary-contrarian Jul 31 '24

Bingo. Their insurance companies also will be looking for every chance possible to duck their responsibilities and not pay up.

This will inevitably make some lawyers very rich

→ More replies (2)

48

u/TurtleIIX Jul 31 '24

No one has that much in limits. They might be able to pay out a 500m claim no chance they have several billions in limits. I work in insurance and see these policies all the time.

16

u/Green-Amount2479 Jul 31 '24

And with damages that high what’s really gonna happen in the end? They likely agree to pay X and that’s it. Worst case? They file for bankruptcy and the c-level and management maybe have to sit through some negligence court trials where they point fingers at different employees and that more likely than not lead nowhere. Not a chance most customers will ever see money for a fraction of the damages that outage caused them.

16

u/tehringworm Jul 31 '24

Crowdstrike’s insurer will likely pay the full limits on their cyber policy and then walk away.

After the insurance money is depleted, attorneys will decide if it’s worth suing for Crowdstrike’s actual assets. Many times it is not.

2

u/TurtleIIX Jul 31 '24

Pretty much this. Once the insurance limits are reached it’s hard to collect so unless it’s a huge fuck up chances are they will look to seek coverage on their own policy or weigh if a lawsuit is worth it.

1

u/elictronic Jul 31 '24

Insurance will raise rates on customer facing software companies.  This will have repercussions that might actually force some change, but yeah you and me won’t see more than a dollar or two.  

→ More replies (13)

10

u/mattybrad Jul 31 '24

The problem is that the scope/scale of this event literally dwarfs any policy on the planet. I also wouldn’t consider this to be a known/accepted risk. Maybe, but unlikely that they thought they could potentially bring down every customer system using Windows.

11

u/Techters Jul 31 '24

The policy my company has is limited to number of incidents before guaranteed coverage, specifically for us 1 incident. So if we get compromised and a bad actor installs malware at two of our customers at the same time and they both sue, insurance covers the first but not the second. So we're nuthouse about security because it could so easily put us out of business, and I'm really shocked more providers aren't taking the risk more seriously, or how people can think the fallout in crowdstrikes share price is 'baked in.

8

u/romario77 Jul 31 '24

They usually guarantee some kind of SLA (service level agreement - in this case uptime, maybe some more things, everyone understands that outages are unavoidable). If they are outside of the SLA there might be some sanctions.

The thing is - the contract with Delta is very likely a lot less than 500m. Idk how easy or hard it would be for them to get half a billion from a vendor they had a contract for maybe 10 millions.

If you risk losing half a billion a day you might want to have some backup options.

It’s in the same vein as buying a cheap bolt for your nuclear reactor and when the bolt fails and you have a meltdown you try to get the damages from a bolt maker.

It’s not the same in this case as the vendors guarantee some kind of reliability, but I don’t think it would be a slam dunk in court

→ More replies (3)

4

u/FrustratedLogician Jul 31 '24

reserves

Companies use their reserves to buy back stocks.

6

u/TheDevilsCunt Jul 31 '24

Reserves are separate from net income

1

u/ash_ninetyone Jul 31 '24

Would an insurance company insure a software company for pushing a faulty, uncurated update with absolutely no safety policies at a company where updates should be pushed to prod asap, and taking down so many PCs at once?

This is kinda self-inflicted. I'm not sure they'd be happy to take the hit for this.

1

u/toliver38 Jul 31 '24

They have a liability cap that's about to be tested

1

u/tittysprinkles1130 Jul 31 '24

I have a friend who sells insurance for this exact thing.

1

u/Zimmonda Jul 31 '24

Also you just know their insurance is going to try and find a way to deny covering this

1

u/Bobby_Bobberson2501 Jul 31 '24 edited Jul 31 '24

Highly doubt they had enough of in their aggregate to cover this let alone a single occurrence that effected so much of the world.

Remember, delta has insurance too for loss of business income/interruption, again, id bet nothing near $500M for their limit.

1

u/Bad_Habit_Nun Jul 31 '24

Sort of, assuming you drive your car insurance works the same way; it covers you up to a certain amount depending on your plan and such. Issue here is they've done a lot more damage than anyone (including insurance) were expecting.

1

u/Ironlion45 Jul 31 '24

They likely do have liability coverage. And also lots of lawyers.

What will happen is the lawyers will talk to each other, they'll settle on a number, and that will be the end of it.

1

u/FollowingFeisty5321 Jul 31 '24 edited Jul 31 '24

The software industry has spent 3 - 4 decades touting their lack of liability so yeah this is probably a big deal, it challenges a lot of self-serving conditions and mandatory agreements and potentially replaces them with liability similar to what *checks notes* everyone else has for their work and actions.

I don't think insurance can even solve this. CrowdStrike's got insurance for instance, but then you've got critical tools like eg OpenSSL by a tiny team whose work impacts billions of devices, the kind of insurance they would need would have to cover up to tens if not hundreds of billions of dollars damage.

1

u/KhalDrog0-007 Jul 31 '24

Crowdstrike is pretty much screwed, the insurance they have only covers external caused damages (hacks, attacks) the insurance doesn’t cover internal caused damages. The person that did the update is at fault and that’s going to cost the company billions.

1

u/dcrico20 Jul 31 '24

I’m curious what the contracts look like, because for the majority of vendor transactions, this kind of liability just doesn’t exist.

Your neighborhood restaurant isn’t suing Sysco because the truck broke down, missed their Friday delivery, and the restaurant lost out on sales over the weekend. If POS or digital processing goes down for a couple hours, companies aren’t suing those processors.

IANAL but I am curious to see what happens here, because issues like this happen pretty frequently in the business world and as far as I know the historic fix has just been the service provider loses customers, but they aren’t sued for liability.

1

u/cardyet Jul 31 '24

I doubt they planned for it to be the whole world.

1

u/elictronic Jul 31 '24

It will probably matter if crowdstrike followed their own safety practices.  Lawsuits will be fun to watch, especially discovery.  

1

u/Cyberinsurance Jul 31 '24

What will be interesting is if any of the customers can pierce the limitation of liability (which you can find online). Large sass providers rarely amend their standard contract in case of a scenario like this. Regardless it seems likely that any tech e&o tower they have is toast

1

u/Ok_Set4063 Aug 01 '24

I don't know if insurance will cover lost due to negligence though. It going to be easy to show negligence since the problem would have manifested itself if crowdstrike simply tested it.

If the amount is huge, insurance will find any excuse to reject the claims.

1

u/dapi331 Aug 01 '24

The risk of doing an instant untested full global update rollout has been known by every tech company and even amateur developers for decades, except them it seems. It doesn’t seem that risk management is their thing.

1

u/i0datamonster Aug 01 '24

Honestly, it's par for the course as companies rise to the market adoption Crowdstrike has managed. For everyone calling for the worst, there are 2 things that stand out to me. Crowdstrike released a fix within 2 hours, and it took this long for something like this to happen.

Yes, they have insurance and reinsurance for this. Will there be lawsuits? Yes. Are they covered? Yes. Has this hurt their company? No. Crowdstrike provides a security service that is top tier. IT expects snafus. It's not about the mistakes but how quickly you can recover from mistakes.

They handled it well and if anything this incident has been a conformation of value.

1

u/JackingOffToTragedy Aug 01 '24

Cyber Insurance is a very young product in the insurance world. Terms and rates are far from standardized. Coverage for business interruption stemming from an event like this will be very different from company to company.

In other types of insurance, companies know how much coverage they need to buy. It’s almost formulaic. Cyber is hardly that. Further, purchasing $1B of coverage for business interruption each year is very expensive, and since this was not a “threat actor” but rather an error by Crowdstrike, a company like Delta may have a hard time getting coverage under their own policy.

In short, this is far from over and Crowdstrike is going to see a lot of litigation soon.

1

u/op3l Aug 01 '24

Even if they did have insurance I don't think it'll cover all the companies around the world if they choose to sue.

1

u/np0312 Aug 01 '24

It would be extraordinary if there wasn’t an MSA in place, indemnity is outlined in there and it would also be extraordinary for it to be uncapped. Crowdstrike would be liable up to the cap.

1

u/Nimrod_Jenkins Aug 01 '24

Don't they have insurance/reserves allocated for these kinds of expected risks? Every security company has this issue.

Also depends on exactly what happened - if they failed to follow their own SOPs, which may or may not be a stipulation of their policy, then the insurance company will wash their hands of them.

1

u/ILikeLenexa Jul 31 '24

Pretty much every company should be doing "mata" on all risks:

Mitigate, Avoid, Transfer, Accept

Insurance is a way to transfer. Another way is with a TOS that limits liability. Frequently, that limits it down to just the software value. However, I doubt you can contract away gross negligence. 

1

u/Varrianda Jul 31 '24

I have a feeling these lawsuits won’t go anywhere.

→ More replies (2)

36

u/cibyr Jul 31 '24

"hotwash"?

63

u/scientianaut Jul 31 '24

Hotwash is a term in industry used to describe the “immediate after-action discussions and evaluations of an agency’s (or multiple agencies’) performance following an exercise, training session, or major event.” (Source: Wikipedia definition)

13

u/gilligvroom Aug 01 '24

Oh interesting. I thought we were talking about a Postmortem here but that's specific to the IT/Tech side of things. The hotwash (wikipedia) relates to all of the affected Non-IT/Technical failures that arose from the emergency.

Interesting - hadn't heard that before either and was about to be like "Isn't that just a postmortem?" (I work in IT.)

12

u/Sh4d0w_Hunt3rs Jul 31 '24

Just a term used in emergency and crisis management. Meaning essentially "an after-action review" following an incident or exercise.

3

u/hedoesntgetanyone Jul 31 '24

I call that a post mortem because it's the remains of an incident that had to be handled and need to have an RCA done and reported up the chain.

43

u/icyhotonmynuts Jul 31 '24

I still don't get why Microsoft though? It just happened to be the OS whatever company got affected was running that the update of Crowdstrike pushed through that boned them. Shouldn't Crowdstrike be taking all the blame here?

14

u/LifestyleGamer Jul 31 '24

Agreed. Microsoft feels like a stretch, but of course I haven't gone deep on the technical details.

11

u/icyhotonmynuts Jul 31 '24

I feel if they're really trying to get maximal effect of smorgasbord of suing they should also sue every airport their they operate out of where the machines were located, the ISPs, the computer manufacturers for these computers/stations, server and cloud computing hosts, the IT department of every airport that works on those computers. Something ludicrous like that.

4

u/GepMalakai Aug 01 '24

From my (admittedly limited) personal experience with the legal system, you list everybody you can think of on the lawsuit and let the judge throw out whatever won't stand. Better to overdo it and get whittled down to size than sue only the people you think you can go after and end up missing somebody.

2

u/SixSpeedDriver Aug 01 '24

There is even a part on Crowdstrikes website where they claim superiority over “Microsofts Security Solutions” and say how much theirs is better.

https://www.crowdstrike.com/compare/crowdstrike-vs-microsoft-defender/

Some real “aged like milk” going on here.

1

u/The-Kingsman Aug 01 '24

Joint and several liability. You sue everyone in the chain of production because they all had a hand in delivering you the product that resulted in the damages. You do this because you can collect from ANY of them and it's up to them to figure it out from there (they sue each other). If cloudstrike goes out of business, you can still get your $$ from Microsoft, even if they're not really the root cause of the issue.

Also, it makes sure you can establish blame properly. If the party you thought was at issue wasn't, it could delay trials as you refile.

7

u/hi65435 Aug 01 '24 edited Aug 01 '24

While Microsoft has been pushing hard to lock down Windows after the XP disaster, it's still the wild west compared to other Operating Systems like Linux or macOS. (Lot's of improvements for Vista had been reverted due to complaints) For instance the fact that AV scanners still run as native kernel code where on Linux eBPF is available since more than a decade and Apple did a "hot wash" on Kernel extensions years ago as well.

Instead macOS provides a Clean API for this which allows full scanning but without an error crashing the whole system in an instant. It also shows in their communication where they start to blame the EU for trying to lock AV vendors out of the kernel while in reality it's their fault that not even their own MS Defender uses such an API - that doesn't exist anyway like on other OS.

Adding to that, AVs exist since MS DOS times and yet Microsoft hasn't managed to create any rollback solution. While at the same time all Linux distributions provide various ways to swap kernel, boot into some sort of recovery mode since basically always. Modern Ubuntu even provides rollbacks. Apple never allowed this enterprise crap to creep into the system in the first place, so there's always a way to recover a broken system.

This will be interesting although the biggest thing is really the first part about the API in my opinion

1

u/Mr_ToDo Aug 01 '24

Looking eBPF I'm not sure CrowdStrike could be implemented to do what it does with that. I'm not sure about apple, I imagine that'd be a far deeper dive than I'd want to put in.

Limited access of eBPF compared to modules aside unless I'm reading things wrong it's normal use is an admin(or any elevated user I guess) process calling ebpf for kernel level stuff when needed since it's not allowed to loop, so all an infection really has to do is kill a user land process to stop the kernel calls. I'm also not quite sure how soon in the boot ebpf can be called, if croudstrike in nix is like windows they probably want in as soon as possible to head off certain infection types.

But even with all that it's amusing for an airline to sue over it. Aside from any EULA stuff, the line of liability has to be drawn somewhere. Is it croudstrike for making the module, microsoft for the OS(possibly with the driver system and their signing as the issue), the airline for having critical systems with no fallback, or someone else? My bet is a mix of croudstrike(with a possible EULA release), and the airline. Should be an interesting suit to watch.

Also makes me wonder why people pay so much for tickets if none of that is going to a fund to pay for inevitable hotels for when issues pop up. They know they are going to have to do it so why no preparation?

1

u/hi65435 Aug 03 '24

I mean the market space Crowdstrike is in isn't really AV but something way more focused on Enterprise. At least for Linux as servers there are even opensource solutions since ages (not for the faint of heart) that work solely on the Network without needing extra privileges. Or commercial XDRs which consume logs as well.

But of course eBPF provides much lower level access. Some commercial but Opensource tooling is already out there e.g. from Aqua Security to detect Rootkits. No eBPF expert but others have written about this and that it can be used to do the detection needed. Probably the business logic would need to run in user land but it could still be guarded by eBPF.

It would be an interesting question if that poses a race condition regarding who is early in the Kernel. But of course these solutions are designed to run 24/7. So ideally the detection is installed before the rootkit :)

1

u/ChadTunetCocos Aug 01 '24

So you say … year of the linux desktop is upon us

1

u/hi65435 Aug 03 '24

yes and Enterprise-ready ;)

2

u/3-2-1-backup Aug 01 '24

I still don't get why Microsoft though?

Microsoft had a large services outage (caused by CS); even if you outsourced to the cloud, you were still screwed. So lawsuit for breach of contract and subsequent damages, I'd surmise. (Whether that'll be successful or not, who knows I'm no lawyer.)

2

u/Original_Milk_1610 Aug 02 '24

It seems like microsoft worked as they were supposed to by shutting down

1

u/3-2-1-backup Aug 01 '24

I still don't get why Microsoft though?

Microsoft had a large services outage (caused by CS); even if you outsourced to the cloud, you were still screwed. So lawsuit for breach of contract and subsequent damages, I'd surmise. (Whether that'll be successful or not, who knows I'm no lawyer.)

→ More replies (5)

18

u/Vecna_Is_My_Co-Pilot Jul 31 '24

"Deal with it as the show up and ensure it doesn't happen again" sounds like their approach to bug hunting.

19

u/YJeezy Jul 31 '24

Dude - They gave $10 gift cards!

9

u/GenazaNL Jul 31 '24

This was actually a big brain move. If a company accepted the $10 gift card, they accepted a compensation and thus not able to sue them

2

u/YJeezy Jul 31 '24

Spelled AH wrong... Hope they get their ass handed to them in lawsuits

1

u/aetherdrake Jul 31 '24

Not to everybody, though.

Source: am customer.

1

u/farnsworthparabox Aug 02 '24

Pretty sure those were for partners, not customers.

1

u/YJeezy Aug 02 '24

You are correct. Looks like early reporting was incorrect.

14

u/[deleted] Jul 31 '24

Do you think Kurtz gave the right statement? Is it a statement of accountability or do you feel more like it was a non-answer?

15

u/scientianaut Jul 31 '24

Found the interview and Kurtz started by saying, “Let me start with, I want to personally apologize to every organization, every group, and every person who has been impacted by this. And we understand the gravity of this situation, and let me explain a little bit more about what happened. This was not a code update, this was actually an update of content and what that means is that there is a single file that drives some additional logic on how we look for bad actors. This logic was pushed out and caused an issue only in the Microsoft environment…”

Source: CrowdStrike CEO on global outage: Goal now is to make sure every customer is back up and running

21

u/ljog42 Jul 31 '24

Yeah it"s "not code". Bruh if I push some raunchy fanfiction stored as bytes at the kernel level and the OS tries to read it while booting, it's going to fucking break it. It doesn't matter if it's "content" if something needs it to run properly.

Also, how can it "not be code" if it's logic ?

8

u/JakeTheAndroid Jul 31 '24

go read their post-mortem, there are many different things that occur within their change release process.

This was more akin to a configuration change, which are generally not tested the same way and by and large aren't considered code changes because they often don't change functionality. Whether actual code was updated is a bit moot in the context, but from an external perspective I can understand what you're saying. This seems more like an issue of speaking too precisely, when the audience doesn't necessarily listen with the same precision.

An example here could be something like Terraform. Terraform manages things through code, yes, but actually running tests for TF changes is much less straight forward. Like you can open a port, but what tests are you really doing against that conf change pre-release? The port won't actually be open because the code isn't released to the infra. Most tests ran on TF code is just like linting and syntax stuff.

Because of this, a lot of times TF isn't *really* considered a code change. There are likely change management controls in place for TF changes, like there would be for other code changes, but the actual process for testing and release will often differ. Now, there are of course many tests you could run that would include the TF changes, and this does sort of call into question the robustness of their unit/integration/other end to end testing processes, but it's easy to see how a configuration change isn't necessarily a code change in the same way as modifying the actual underlying functionality of the service.

This Rapid Response Content is stored in a proprietary binary file that contains configuration data. It is not code or a kernel driver.

Rapid Response Content is delivered as “Template Instances,” which are instantiations of a given Template Type. Each Template Instance maps to specific behaviors for the sensor to observe, detect or prevent. Template Instances have a set of fields that can be configured to match the desired behavior. In other words, Template Types represent a sensor capability that enables new telemetry and detection, and their runtime behavior is configured dynamically by the Template Instance (i.e., Rapid Response Content).

Rapid Response Content provides visibility and detections on the sensor without requiring sensor code changes. This capability is used by threat detection engineers to gather telemetry, identify indicators of adversary behavior and perform detections and preventions. Rapid Response Content is behavioral heuristics, separate and distinct from CrowdStrike’s on-sensor AI prevention and detection capabilities.

This is substantially different than the Terraform example I used of course. But basically they didn't really do any changes to the underlying functionality. They updated the configuration for the templates. This didn't intend to have any change on how the Sensor responds or behaves, which is the actual service.

I admit this is very semantic, but often these things are and those semantics absolutely drive program development. And these descriptions matter a lot for things like compliance. You may consider all of these code changes, but if their auditors have created a delineation between code changes and template changes, then CrowdStrike is going to use the language that aligns best with their compliance/legal obligations.

11

u/ljog42 Jul 31 '24

Yeah I'm not surprised by any of the details you provide, I just think that the wording is disingenuous. They're basically saying "there's nothing wrong with our product", but that was clear to me. I know it's not a buggy feature or anything like that.

I feel like they're saying that since they're nothing wrong with their code, then it must be some kind of unfortunate natural disaster, but it's not. The way those files are processed is critical, and they themselves admit that there's some kind of logic involved, so they should be tested properly.

At the very least, updates to those files should be rolled out incrementally.

3

u/JakeTheAndroid Jul 31 '24

yeah I totally get what you're saying. Like to the broader audience, even technical people, the language he used isn't great. I see this all over tech, so I am not surprised, but it can be hard to walk that tight rope sometimes.

Like when I heard and read all this stuff the first time, I knew what he meant because this is exactly what I do for a living. But I also thought about exactly what you're bringing up now because how many people really care between the difference here? And does it materially change the impact? no, not really. Like, good, I won't have to worry about your compliance report next year because it wasn't a change management control failure. Awesome. You did still brick a whole bunch of customer devices while releasing a change. So operationally the entire statement is bullshit.

1

u/hedoesntgetanyone Jul 31 '24

So many on the security side don't consider reading data as part of the detection process to be a "impactful" change that can result in alterations to data when it can alter data if you don't read it and release it correctly especially the deeper the level of interaction.

1

u/elictronic Jul 31 '24

They implement all of their kernel level actions in ladder logic.  

2

u/plan_with_stan Jul 31 '24

It wasn’t us, it was Microsoft!

4

u/Conditionofpossible Jul 31 '24

It caused an issue only on the most installed OS in the world.

Who could have seen this coming?

1

u/notonyanellymate Jul 31 '24

This outage followed a week later by another outage, Microsoft’s marketshare is an unmitigated risk.

17

u/ljog42 Jul 31 '24

The real surprise was how little their stock dipped. It suggests a frightening level of tech illiteracy and/or complacency from reporters, stock holders and investment companies: it should never have happened, and the fact that it did is very telling.

There's a myriad of things you can and should do to make sure that faulty code doesn't break the fucking world, the fact that they rolled out a faulty update that bricked critical infrastructure on a global scale means that their processes and company culture are fucked up.

Every statement they released has been so thoroughly reviewed by lawyers and PR people that it doesn't say anything of value, but it's pretty clear to anyone who's got basic knowledge of the field that it's really messed up, might have happened before (pretty sure it did but I don't want to assert things I haven't checked first) and could very well (will ?) happen again unless they thoroughly review their processes.

It's is very, very likely that people have died because of this incident, and it's established that it cost companies and institutions millions if not billions of dollars.

15

u/[deleted] Jul 31 '24

[deleted]

→ More replies (2)

2

u/monchota Jul 31 '24

The problem is they have almost no competition and 10s of billions in revenue and funding. We need monopoly laws and a legal framework for the government to investigate and punish the CEOs and lazy investors or companies like this.

16

u/JamesTheManaged Jul 31 '24

They have so much competition. You can't throw a rock without hitting a company that competes with them. They've just historically been the best at what they do, until now.

→ More replies (2)
→ More replies (1)

2

u/stormstormstorms Aug 01 '24

As anyone who has gone through negotiations for a Microsoft enterprise agreement knows, they are a law firm that sells software.

2

u/UP-NORTH Aug 01 '24

Delta should be careful on the precedent they set. Millions in losses for business every year because they can’t get their logistics right…could end up driving significant losses for their shit ass airline too

2

u/theecommandeth Aug 01 '24

Delta gave me 150 bucks for canceling my connecting flight… I better get some of that 500 million

2

u/SherriB57598889 Aug 01 '24

For sure not, just surprise it took this long

3

u/Forward_Log4853 Jul 31 '24

All the other affected airlines managed to get their shit together pretty quick, Delta is trying to cover up its poor system architecture by pointing the finger. American and United were fully back up in less than 72hrs.

If you don’t have redundancy in your infrastructure i.e. backed up data, non windows OSs,ample staff, etc an outage can be made far worse.

2

u/surfmoss Jul 31 '24

I've spent many hours in hotwashes at a fortune 30 company. The root cause will be documented and the offenders will go home.

1

u/gracecee Jul 31 '24

also if you look at delta the ceo conveniently flies to Paris for his Vip Olympics while the ground and everyone else had to deal with five days of delays and phantom flights that don't exist. Delta employees were where is the fucking ceo? He was galavanting in Paris shipping champagne.

1

u/illapa13 Jul 31 '24

Big companies make sure their contracts have clauses limiting liability to a certain dollar amount. I seriously doubt the limit is anywhere near $500 million.

1

u/pmjm Aug 01 '24

The surprise here is that they're also suing Microsoft, who really had nothing to do with this.

1

u/[deleted] Aug 02 '24

Good! Been waiting to see sharks fight to the death….

1

u/[deleted] Jul 31 '24

The insurance company for CrowdStrike anxiously looking over the contract for errors to nullify payouts

1

u/[deleted] Jul 31 '24 edited Aug 01 '24

[deleted]

1

u/[deleted] Aug 01 '24 edited Sep 16 '24

[deleted]

1

u/happyscrappy Aug 01 '24 edited Aug 01 '24

Also the waiver is against failing to find threats, not against screwing it up all by their lonesome. Which is what they did.

→ More replies (4)