The problem is the monoculture. We are fucking addicted to convenience and efficiency at all costs.

A diverse ecosystem, if a bit more work to manage, is much more resilient, and wouldn’t have been this catastrophe.

Our technology is great, but our processes suck. Standardization. Just in time. These ideas create incredibly fragile organizations. Humanity is so short sighted. We are screwed.

This happened to me in December 2022/January 2023. Pretty similar problem. Just a regular Windows update caused it. Weirdly it didn’t affect everyone (and I’m not on any sort of beta channels). Installing KB5021233 keeps causing BSOD 0xc000021a.

After installing KB5021233, there might be a mismatch between the file versions of hidparse.sys in c:/windows/system32 and c:/windows/system32/drivers (assuming Windows is installed to your C: drive), which might cause signature validation to fail when cleanup occurs.

I’m not sure you can blame the CEO. As much as I despise C-level execs this seems like a failure at a much lower level. Now the question of whether this is a culture failure is a different story because to me that DOES come from the CEO or at least that level.

How difficult would it be for companies to have staged releases or oversee upgrades themselves? I mostly just use Linux but upgrading itself is a relatively painless processing and logging into remote machines to trigger an update is no harder. Why is this something an independent party should be able to do without end user discretion?

I read his comment more about the kernel level access more than the one company.

Crowdstrike was near ubiquitous because it was the best tool out there.

I understand the reason for it, but that ubiquity comes with potential dangers, as we saw on Friday. But, no, I don’t think the solution is “five different cyber security solutions” at every site. However, different cyber security solutions for different industries might not be such a bad idea. Or, I suppose the root of the problem might be the ubiquity of the OS. Should every PC be running the same jack of all trades but master of none OS?

It’s not the best tool out there. It’s the laziest one that works. It’s perfectly possible to securely operate without a rootkit hacked into your kernel.

Modern approaches involve running an ebpf module on rootless immutable images that are scanned on build. My org is PCI, SOC2, and HITRUST and we didn’t go down because we would never take such a sad lax approach to hand off responsibility for security to a third party. The trade off is your head of compliance and security need to actually learn things and work hard to push alternatives with auditors and consultants and most companies put an MBA who can’t critically think their way out of an empty room at the helm.

They haven’t moved everything off windows, just as much as they could.

I’m not on that team. For all I know, our password manager might be on a random window server. Or some middleware.

In a major company where each team does their own thing and communicate through endpoints, It’s impossible to know every configuration.

Was thinking bout this (a week later).

Following up, our partner/affiliate sites were down. Each partner connects to us to submit data, and half were government contracts that were down. It didn’t affect our systems, but it affected how we provide services to them.

So it was a mild shit show.

The issue, in this case, is more about Crowdstrike’s broad usage than Microsoft’s. The update that crippled everything was to the Crowdstrike Falcon Sensor software, not to the OS.

Funnily enough, they had a similar issue with an update to the Linux version of the software a few months ago, that didn’t have these broad-reaching consequences largely due to the smaller Linux user base. Which means this is starting to look like a pattern, and there are going to need to be some serious process changes at Crowdstrike to prevent things like this in the future.

Anybody’s guess if those changes happen or not.

Should be done even when things go right.