CrowdStrike Windows BSOD

Forums › Microsoft Windows › CrowdStrike Windows BSOD

1 | ... | 3 | 4 | 5 | 6 | 7 | 8 | 9

281 posts

Ultimate Geek
+1 received by user: 168

ID Verified

Trusted

#3262703 22-Jul-2024 12:03

NickMack:

As you state, the foundational cause is the architectural decisions made by Microsoft around the kernel and how applications interact/control plain allowing an application to brick and O/S vs. terminate/disallow. Dont be fooled into believing this is only a MS risk/issue..... Architecture always is a trade off of many things.

There's multiple lessons here for all, from Vendors, customers, MSPs etc. - As always Security is about defense in depth, layers, acceptable risk, $$, culture, diversity of AV products etc.

Nick

Completely agree and definitely not just a MS risk/issue. As an example, Crowdstrike has caused issues previously on Linux but on a much smaller scale:

https://old.reddit.com/r/debian/comments/1c8db7l/linuximage61020_killed_all_my_debian_vms/

In the current case I've been interested by the several analyses that have shown the channel update file was full of nulls. However, Crowdstrike have stated that wasn't the issue. I'm trying to understand how a file that is referenced just containing null values couldn't be a problem.

neb

11294 posts

Uber Geek
+1 received by user: 10018

Trusted

Lifetime subscriber

#3262887 22-Jul-2024 13:49

K8Toledo:
neb: Not necessarily. Some systems map the zero page to valid memory so NULL pointer derefs can be managed in a way other than producing a segfault.

Well I would hope all systems map the zero page to valid memory, because if not - something is very, very wrong. :)

I didn't feel like typing a three-page essay on the semantics of pointers... what I meant was that they map NULL (all-zero) pointers to valid memory at address zero so that you can dereference one without the program crashing.

This was a situation where staggering amounts of money were involved and they couldn't have any downtime or outages. The code also dated back to before memory protection was much of a thing.

K8Toledo

1018 posts

Uber Geek
+1 received by user: 311

#3262939 22-Jul-2024 16:01

neb:

I didn't feel like typing a three-page essay on the semantics of pointers... what I meant was that they map NULL (all-zero) pointers to valid memory at address zero so that you can dereference one without the program crashing.

This was a situation where staggering amounts of money were involved and they couldn't have any downtime or outages. The code also dated back to before memory protection was much of a thing.

Still wrong, I'm afraid. :) Null points to nothing.

Behodar

11094 posts

Uber Geek
+1 received by user: 6071

Trusted

Lifetime subscriber

#3262946 22-Jul-2024 16:03

I'm not familiar with the low-level stuff in Windows, but I know in at least one other OS there's a "zero page" at address zero with some dummy data along with the ASCII text "NULL POINTER DEREFERENCE" so that if a null pointer is dereferenced and then output to the screen/printer it doesn't crash, and you find out about it :)

K8Toledo

1018 posts

Uber Geek
+1 received by user: 311

#3262948 22-Jul-2024 16:07

Behodar:

I'm not familiar with the low-level stuff in Windows, but I know in at least one other OS there's a "zero page" at address zero with some dummy data along with the ASCII text "NULL POINTER DEREFERENCE" so that if a null pointer is dereferenced and then output to the screen/printer it doesn't crash, and you find out about it :)

You may be right.

It's been ages since I read Windows Internals but the answers should be under Memory Management section 5.

Also has anyone pointed out that bad hardware can cause an access violation with Null pointer as the bucket ID when debugging?

https://archive.org/details/windows-internals-part1-7th

KiwiSurfer

1722 posts

Uber Geek
+1 received by user: 993

ID Verified

Lifetime subscriber

#3262951 22-Jul-2024 16:15

zenourn:

Completely agree and definitely not just a MS risk/issue. As an example, Crowdstrike has caused issues previously on Linux but on a much smaller scale:

https://old.reddit.com/r/debian/comments/1c8db7l/linuximage61020_killed_all_my_debian_vms/

Don't get me wrong, this isn't 100% entirely MS's fault, that is a perfectly fine position to take. I agree with that.

However it is interesting to observe how other OSes tend to have these sort of "security" software running in user space. The little impact on Linux/MacOS shows that type of software is normally run in user space so a failure is fairly minimal when it does happen. The Linux example above is very rare in the Linux ecosystem -- it is very much not standard practice in Linux to inject untrusted code from outside Linux into the kernel. On Windows however it seems to be standard practice allow external untrusted software from e.g. "security" vendors to run in kernel space -- something that would be a big no-no in most Linux/MacOS/Unix/etc environments.

I think MS need to have a good look at whether it makes sense to allow external code to run within their kernel space. They did a great job many years ago of taking graphic drivers out of kernel space and running graphic driver outside the kernel -- it led to a massive improvement in reliability. I think I read somewhere >90% of BSOD in the Win9x era was simply due to crappy graphics drivers taking down the entire OS. A graphic driver crash now just results in the screen going blank for ~1 sec (which you can also observe during a driver install/update). This is largely why we rarely see a BSOD in recent versions of Windows. MS should look at doing the same for AV software.

So my position is that it is at least partially a MS risk/issue -- there are certainty steps MS can take to mitigate/reduce or even eliminate the risk of untrusted "security" type software from taking the entire OS down. They can look in-house at how they dealt with graphics drivers as a good example of how to manage this risk better.

There is something everyone can do to avoid this happening in the future -- MS included.

/ends my 2 cents

Lego

Shop now for Lego sets and other gifts (affiliate link).

neb

11294 posts

Uber Geek
+1 received by user: 10018

Trusted

Lifetime subscriber

#3262966 22-Jul-2024 16:51

K8Toledo: Null points to nothing.

And that's why I avoided getting into a three-page writeup about the convoluted semantics of NULL pointers, which are typically phrased in such a way that no implementation, no matter how bizarre, can be said to be non-compliant (the standards committee members keep coming up with new corner cases and oddities and rephrasing the wording to accommodate them until it can mean almost anything). In most sane architectures a C NULL pointer is all zeroes, (void *)0, which means it points to whatever's at (or mapped at) address zero. That's usually not accessible or usable in order to generate a SIGSEGV, but in this case they made it so to avoid crashes and outages, so you could dereference a NULL pointer without interrupting operations.

neb

11294 posts

Uber Geek
+1 received by user: 10018

Trusted

Lifetime subscriber

#3263680 23-Jul-2024 19:08

In case this is still of use to anyone, there's a cloud-based way to recover which only requires an internet connection rather than manual intervention. It also explains why many reboots may be necessary.

msukiwi

2443 posts

Uber Geek
+1 received by user: 2126

Lifetime subscriber

#3263693 23-Jul-2024 19:51

An explanation......that almost anyone can understand.....

Rickles

3107 posts

Uber Geek
+1 received by user: 445

Trusted

#3263717 23-Jul-2024 20:30

@msukiwi ... I do like the Mr Robot reference inserts 😁

neb

11294 posts

Uber Geek
+1 received by user: 10018

Trusted

Lifetime subscriber

#3263722 23-Jul-2024 20:42

And another random snippet, for people who have been gleefully pointing out that it only affects Windows:

CrowdStrike broke Debian and Rocky Linux months ago, but no one noticed.

Falcon Pro: Massive issues on macOS

So Crowdstrike can take out anything, any time. It just happened to be Windows this time.

Geekzone support

Support Geekzone with one-off or recurring donations Donate via PressPatron.

lxsw20

3689 posts

Uber Geek
+1 received by user: 2174

Subscriber

#3263723 23-Jul-2024 20:47

It's a very powerful security suite, it's not surprising it can take out any OS.

From an admin perspective, it's actually a really good product. I know it's all well and good to get pitchforks out, and it's deserved, but there is a reason so many big companies use it.

MadEngineer

4591 posts

Uber Geek
+1 received by user: 2570

Trusted

#3263736 23-Jul-2024 21:15

Anyone on the coalface for fixing this? I suspect a few machines (1-3% of fleet) were suffering from this fault a few weeks ago. BSOD then randomly fixing themselves after numerous reboots and power cycles. Sample size unfortunately has been too small and no details of the BSOD error.

You're not on Atlantis anymore, Duncan Idaho.

freitasm

BDFL - Memuneh
80646 posts

Uber Geek
+1 received by user: 41030

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#3263838 24-Jul-2024 08:16

Having a drink with friends at a Wellington bar last night, heard the bartender explaining this even to a couple of patrons as "a bad Windows Update, that usually comes on Wednesdays but was released on Friday."

Hmmm. No.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

Tinkerisk

4798 posts

Uber Geek
+1 received by user: 3660

#3263848 24-Jul-2024 08:49

freitasm:

Having a drink with friends at a Wellington bar last night, heard the bartender explaining this even to a couple of patrons as "a bad Windows Update, that usually comes on Wednesdays but was released on Friday."

Hmmm. No.

Doesn't that ultimately matter to the end user? Some even claim that Windows itself is a virus. 😁

- NET: FTTH & VDSL, OPNsense, 10G backbone, GWN APs
- SRV: 12 RU HA server cluster, 0.1 PB storage on premise
- IoT: thread, zigbee, tasmota, BidCoS, LoRa, WX suite, IR
- 3D: two 3D printers, 3D scanner, CNC router, laser cutter

1 | ... | 3 | 4 | 5 | 6 | 7 | 8 | 9