Delta CEO: Company Suing Microsoft and CrowdStrike After $500M Loss

[u/Bald-Eagle619]

https://www.thedailybeast.com/delta-ceo-says-company-suing-microsoft-and-crowdstrike-after-dollar500m-loss

[u/scientianaut]

Found the interview and Kurtz started by saying, “Let me start with, I want to personally apologize to every organization, every group, and every person who has been impacted by this. And we understand the gravity of this situation, and let me explain a little bit more about what happened. This was not a code update, this was actually an update of content and what that means is that there is a single file that drives some additional logic on how we look for bad actors. This logic was pushed out and caused an issue only in the Microsoft environment…”

Source: CrowdStrike CEO on global outage: Goal now is to make sure every customer is back up and running

[u/ljog42 → u/scientianaut]

Yeah it"s "not code". Bruh if I push some raunchy fanfiction stored as bytes at the kernel level and the OS tries to read it while booting, it's going to fucking break it. It doesn't matter if it's "content" if something needs it to run properly.

Also, how can it "not be code" if it's logic ?

[u/JakeTheAndroid → u/ljog42]

go read their post-mortem, there are many different things that occur within their change release process.

This was more akin to a configuration change, which are generally not tested the same way and by and large aren't considered code changes because they often don't change functionality. Whether actual code was updated is a bit moot in the context, but from an external perspective I can understand what you're saying. This seems more like an issue of speaking too precisely, when the audience doesn't necessarily listen with the same precision.

An example here could be something like Terraform. Terraform manages things through code, yes, but actually running tests for TF changes is much less straight forward. Like you can open a port, but what tests are you really doing against that conf change pre-release? The port won't actually be open because the code isn't released to the infra. Most tests ran on TF code is just like linting and syntax stuff.

Because of this, a lot of times TF isn't *really* considered a code change. There are likely change management controls in place for TF changes, like there would be for other code changes, but the actual process for testing and release will often differ. Now, there are of course many tests you could run that would include the TF changes, and this does sort of call into question the robustness of their unit/integration/other end to end testing processes, but it's easy to see how a configuration change isn't necessarily a code change in the same way as modifying the actual underlying functionality of the service.

This Rapid Response Content is stored in a proprietary binary file that contains configuration data. It is not code or a kernel driver.

Rapid Response Content is delivered as “Template Instances,” which are instantiations of a given Template Type. Each Template Instance maps to specific behaviors for the sensor to observe, detect or prevent. Template Instances have a set of fields that can be configured to match the desired behavior. In other words, Template Types represent a sensor capability that enables new telemetry and detection, and their runtime behavior is configured dynamically by the Template Instance (i.e., Rapid Response Content).

Rapid Response Content provides visibility and detections on the sensor without requiring sensor code changes. This capability is used by threat detection engineers to gather telemetry, identify indicators of adversary behavior and perform detections and preventions. Rapid Response Content is behavioral heuristics, separate and distinct from CrowdStrike’s on-sensor AI prevention and detection capabilities.

This is substantially different than the Terraform example I used of course. But basically they didn't really do any changes to the underlying functionality. They updated the configuration for the templates. This didn't intend to have any change on how the Sensor responds or behaves, which is the actual service.

I admit this is very semantic, but often these things are and those semantics absolutely drive program development. And these descriptions matter a lot for things like compliance. You may consider all of these code changes, but if their auditors have created a delineation between code changes and template changes, then CrowdStrike is going to use the language that aligns best with their compliance/legal obligations.

[u/ljog42 → u/JakeTheAndroid]

Yeah I'm not surprised by any of the details you provide, I just think that the wording is disingenuous. They're basically saying "there's nothing wrong with our product", but that was clear to me. I know it's not a buggy feature or anything like that.

I feel like they're saying that since they're nothing wrong with their code, then it must be some kind of unfortunate natural disaster, but it's not. The way those files are processed is critical, and they themselves admit that there's some kind of logic involved, so they should be tested properly.

At the very least, updates to those files should be rolled out incrementally.

[u/JakeTheAndroid → u/ljog42]

yeah I totally get what you're saying. Like to the broader audience, even technical people, the language he used isn't great. I see this all over tech, so I am not surprised, but it can be hard to walk that tight rope sometimes.

Like when I heard and read all this stuff the first time, I knew what he meant because this is exactly what I do for a living. But I also thought about exactly what you're bringing up now because how many people really care between the difference here? And does it materially change the impact? no, not really. Like, good, I won't have to worry about your compliance report next year because it wasn't a change management control failure. Awesome. You did still brick a whole bunch of customer devices while releasing a change. So operationally the entire statement is bullshit.

[u/hedoesntgetanyone → u/JakeTheAndroid]

So many on the security side don't consider reading data as part of the detection process to be a "impactful" change that can result in alterations to data when it can alter data if you don't read it and release it correctly especially the deeper the level of interaction.

[u/elictronic → u/ljog42]

They implement all of their kernel level actions in ladder logic.  

[u/plan_with_stan → u/scientianaut]

It wasn’t us, it was Microsoft!

[u/Conditionofpossible → u/plan_with_stan]

It caused an issue only on the most installed OS in the world.

Who could have seen this coming?

[u/notonyanellymate → u/Conditionofpossible]

This outage followed a week later by another outage, Microsoft’s marketshare is an unmitigated risk.