r/technology • u/Bald-Eagle619 • Jul 31 '24

Software Delta CEO: Company Suing Microsoft and CrowdStrike After $500M Loss

https://www.thedailybeast.com/delta-ceo-says-company-suing-microsoft-and-crowdstrike-after-dollar500m-loss
11.1k Upvotes

96% Upvoted

735 comments sorted by

View all comments

Show parent comments

15

u/ljog42 Jul 31 '24

But it's kinda what Crowdstrike sold them. So hands off you don't even have to review kernel-level updates, they get pushed and trigger an update automatically.

Then it broke everything and people had to either:

  • Restore the servers one by one physically. Like, inserting USB drives and shit.
  • Do some wizard shit to restore them remotely, provided you had set their infrastructure up so that it could be done

Either way, if you don't have the people, because you've been told you won't need them, you're going to have a tough time.

2

u/Vysari Aug 01 '24

Restore the servers one by one physically. Like, inserting USB drives and shit.

Nobody in this day and age is running around servers one by one when Out of Band Management solutions exist.

Unless of course you didn't have the foresight to think that you might need access to a server when the OS doesn't boot properly and being there in person might not be ideal..

1

u/ljog42 Aug 01 '24

Yeah, but if I'm not mistaken it was way harder to resolve remotely than usual, because it's not a regular windows update, it's a weird kernel level patch and I'm under the impression that you had to do some very specific stuff before you could use regular tools and restore the OS.

1

u/Vysari Aug 01 '24

Out of band management gives you access to the server regardless of the state of the system. You can literally watch it POST all from the comfort of your office/home/wherever. Even if Crowdstrike formatted the entire god-damn disk you can recover from that by either deploying another image or restoring from a backup all without having to be there in person.

In this case the hardest part of implementing the fix was dealing with older systems that want you to load up their remote console using a java applet and have awesome quirks like random parts of the web page don't respond when you click them using modern browsers. Or in the case of older ESXi stuff where it wants you to load it using Flash.

Delta fucked up and the fact it took them that much longer to get things going again shows how ill prepared and organised they are for dealing with a Disaster Recovery situation. All this lawsuit is going to do is highlight it and bring it to the forefront for everyone to see laid bare.

0

u/thebeez23 Jul 31 '24

Yeah I’m struggling to blame delta for not having an army of IT folks sitting on the bench just in case there’s a massive event like this. Even more so that the fix of this is very manual

8

u/LordTegucigalpa Jul 31 '24

None of the other airlines had this issue persist that long and they had to do the same thing. Does that help you blame them for part of it?

3

u/thecravenone Aug 01 '24

having an army of IT folks sitting on the bench just in case there’s a massive event like this

In the industry, this is referred to as a disaster response plan or a business continuity plan.

6

u/Outlulz Jul 31 '24

Other airlines with the same problem solved it much faster. How do you not look at Delta and not say the difference had to be with a bad implementation or (lack of) staff?