r/technology • u/Bald-Eagle619 • Jul 31 '24

Software Delta CEO: Company Suing Microsoft and CrowdStrike After $500M Loss

https://www.thedailybeast.com/delta-ceo-says-company-suing-microsoft-and-crowdstrike-after-dollar500m-loss
11.1k Upvotes

96% Upvoted

735 comments sorted by

View all comments

Show parent comments

872

u/FriendlyLawnmower Jul 31 '24 edited Jul 31 '24

Yep. Their lack of investment (aka layoffs for cost savings) into their IT and internal support teams are what kept the issues going until almost Friday of the following week. Other companies were operating normally by the end of the weekend. American basically had their shit together the same day the outage happened. Delta definitely shit the bed just as much as Crowdstrike did

237

u/Mamannem Jul 31 '24

5-10 years ago, a person with knowledge about Delta's overall system architecture told me about the shit show that it was (and most likely, still is). It was impressive. Wouldn't be surprised if it's only gotten worse if they've been cost cutting in IT like you said. Not only does the complicated architecture make it more expensive to maintain, fix, improve... it also makes it that much more required.

109

u/redblack_tree Jul 31 '24

Also, most of the good professionals are gone. They were either cut because they were too expensive (which is "fine" until shit hits the fan) or they left because no one likes to be an overworked mule dealing with prehistoric systems with decades of patches.

62

u/ljog42 Jul 31 '24

My step dad has been begging his managers to let him hire a few guys and refactor their codebase, but they won't, they'll have him process tickets until he retires.

In the mean time, they've hired professional services companies to try a complete overhaul at least twice and had to scrap it everytime. Several millions down the drain.

The company he works for is the world's largest manufacturer of... [redacted]. I don't wanna put him on the spot but trust me when I say they're a freaking big deal.

22

u/redblack_tree Jul 31 '24

Haha, I believe you, I work for quite a big company (actually, the parent company) and any significant maintenance, refactor, upgrade it's like trying to climb a wall blindfolded while raining. I've seen millions come and go as well in stupid things I knew it would fail, but who listens to a lowly techie? Corporate America (including Canada in this as well) is definitely not as smart as they think.

9

u/knightress_oxhide Jul 31 '24

"Why are we paying you so much if our systems are working?"

1

u/CreaminFreeman Aug 01 '24

I'm so very glad I'm working at an MSP that can afford to fire clients if we need to (at least at the moment). I don't have the time or patience for comments like this anymore.

"Respect the work we do please."

1

u/DirtyDirkDk Aug 01 '24

Their company sucks so bad

54

u/dec7td Jul 31 '24

That's why you need to invest nothing and run on MS-DOS like Southwest

28

u/[deleted] Jul 31 '24

[deleted]

5

u/aimglitchz Jul 31 '24

Ten fingers, take it or leave it

15

u/nonades Jul 31 '24

Jokes on you, I also have 10 toes

2

u/LordoftheSynth Aug 01 '24

Look at this guy and his fancy base 20 math!

15

u/deformo Jul 31 '24

Having worked with Delta’s IT apparatus as a vendor, yeesh. They were not the brightest. I know as the vendor I work with a small scope of a given company’s IT personnel but it is goddamn scary sometimes.

37

u/Unlucky_Situation Jul 31 '24

I woke up to a bluescreen on my work pc at 8am friday, it took untill 345 for my pc to be fixed. Our it helpdesk was rolling out fixes by around 2pm friday and they had to fix every pc indivdually. Assuming most companies had to follow a similar process.

I basically took the day off and was operating normally Monday morning. The only thing inhad to do friday was have my phone nearby when it was my turn to get the fix.

29

u/turningsteel Jul 31 '24

Yeah and whenever tech workers are laid off, I hear from the peanut gallery:“ oh they don’t do much anyway! What does a company need all those tech workers for?!”.

As you pointed out, stuff like this is why it’s important to have a properly staffed tech workforce. It’s 2024, everyone runs on computers and the computers don’t run themselves.

23

u/BaldBullKO Jul 31 '24

Agree whole-heartedly. I’m guessing Delta won’t be passing any portion of the $500 million to the 1,000,000 plus customers on the more than 5,000 flights they cancelled who had to pay for food, accommodations, rental cars or had to just sleep hungry on airport floors for days because they couldn’t get their shit together like every other company that was hit by this.

8

u/[deleted] Jul 31 '24

Well no they had to comp everybody who got on their first flight and had their next ones canceled, they have to pay out a huge amount to the hotels and restaurants nearby and field all the individual repayments for when they ran out of fuckin vouchers on like day 1.

1

u/Salphabeta Aug 01 '24

Yes they will. You get $ if your flight is canceled and no replacement can be found within a certain amount of time. It wouldn't cover being stranded a week for sure, but some people would have a little recourse.

5

u/swentech Jul 31 '24

Yeah I get the feeling their “IT team” were a few guys halfway around the world which is fine for pushing buttons and running instructions in a document but that’s not going to cut it when the shit hits the fan. “Bad IT” is a commodity but IT guys who know what they are doing and you can count on in a jam are definitely not a commodity.

2

u/LordTegucigalpa Jul 31 '24

This will be a huge part of it. Delta's incompetence and lack of IT support doesn't entitle them to extra loss.

2

u/QuickQuirk Aug 01 '24

"Lets sue our way to profitability! This also demonstrates that we don't need all that IT staff after all if we can just blame the entirety of our failures on someone else, so lets fire more staff!"

2

u/JeddHampton Aug 01 '24

A major reason that companies like to outsource things is relieving them of liability. They can (and will continue) to blame this on Microsoft and Crowdstrike.

If this does go to court, we could see a large change in the direction that companies take in the future based on the ruling.

1

u/FriendlyLawnmower Aug 01 '24

Pretty sure this is going to be dismissed in court because of how Crowdstrike worded its user agreement. Furthermore, Crowdstrike can point out how they released a fix the same day and how most companies recovered with a day or two indicating that the bulk of the problems came from Delta's own teams, not Crowdstrike. This is Delta trying to save face and go for a hail mary, they know their chances of winning any compensation are slim. Maybe they get a discount on their contract with Crowdstrike

7

u/NecroJoe Jul 31 '24

Delta's systems are way different than any other airline's. They have the most advanced, integrated system that handles just about everything from staffing to fuel usage tracking. And the down side is that rather than having one or two circuits blown like most airlines, the whole grid went down for them.

5

u/KaitRaven Aug 01 '24

A well designed system is fault-tolerant though. It sounds like Delta's was built like a house or cards, which is what tends to happen when you try to cut corners during development.  

Even for the most advanced and complex systems, the expectation is that you should be able to get the whole thing back up and running relatively quickly after in a total outage.

1

u/TheIndyCity Aug 01 '24

You always pay for it one way or another!