CrowdStrike Windows BSOD

Forums › Microsoft Windows › CrowdStrike Windows BSOD

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

1722 posts

Uber Geek
+1 received by user: 993

ID Verified

Lifetime subscriber

#3261823 20-Jul-2024 14:26

Oblivian: Kiosks that get reimaged nearly daily are the reason you see the PoS/kiosks and supermarkets up so fast.

Wasn't the case at Woolworths -- I was there this morning and I noticed their strategy was seemed to be getting their IT vendor (probably outsorced) come in and fix every second POS machine and take off to the next supermarket where I assume they did the same thing and so on. Presumely they're doing that to try and get a minimal number of POS machines back into action and come back later to sort out the rest of the fleet.

Farmers were wrose, they had their entire shop closed today -- they weren't letting anyone in even to browse. Not sure why they couldn't just get someone in to reset even one or two POS machines. shrugs

Most other shops operating normally though. Ironically those not using kiosks were all fine from what I could see.

KiwiSurfer

1722 posts

Uber Geek
+1 received by user: 993

ID Verified

Lifetime subscriber

#3261830 20-Jul-2024 14:41

Rikkitic:

This is exactly the reason I avoid automatic updates and only very selectively allow minimal others after waiting a decent time to make sure they aren't going to kill something important. So far it has worked for me.

My confidence in autoupdates depends on how the autoupdates are managed by the software vendor. I have no issues having autoupdates enabled on Debian stable as they have a fairly robust testing process before updates are pushed out to the stable distribution -- basically updates must progress through their testing and unstable distributions without any red flags within a certain timeframe before it is allowed into the stable distribution.

Other autoupdates mechanisms which don't have a clear process for testing with a small group before being pushed out to the majority I agree are best avoided.

Some vendors have a strategy of pushing out to 1%, then 5%, then 10%, progressing up to 100% so they have the opportunity to roll-back if needed. Better to roll back at 1% than after you've pushed it out to 100% in one go. CrowdStrike is probably regretting not doing something like this for their autoupdate system. When this is all over I hope they do a review of their testing methology as testing should absolutely have picked this up before the go-ahead to push this broken update to everyone.

BarTender

3629 posts

Uber Geek
+1 received by user: 2572

ID Verified

Trusted

Lifetime subscriber

#3261833 20-Jul-2024 14:59

KiwiSurfer:My confidence in autoupdates depends on how the autoupdates are managed by the software vendor. I have no issues having autoupdates enabled on Debian stable as they have a fairly robust testing process before updates are pushed out to the stable distribution -- basically updates must progress through their testing and unstable distributions without any red flags within a certain timeframe before it is allowed into the stable distribution.

Other autoupdates mechanisms which don't have a clear process for testing with a small group before being pushed out to the majority I agree are best avoided.

Some vendors have a strategy of pushing out to 1%, then 5%, then 10%, progressing up to 100% so they have the opportunity to roll-back if needed. Better to roll back at 1% than after you've pushed it out to 100% in one go. CrowdStrike is probably regretting not doing something like this for their autoupdate system. When this is all over I hope they do a review of their testing methology as testing should absolutely have picked this up before the go-ahead to push this broken update to everyone.

Which is all well and good until a zero day drops and you get popped because no one was working the weekend to apply that update.

To me many of the responses here show people have never worked in large enterprises with tens or hundreds of thousands of staffers and many thousands of servers facing the internet.

The compliance, auditors and risk management people have sucked all the joy of leaving unpatched servers in your infrastructure.

networkn

Networkn
32862 posts

Uber Geek
+1 received by user: 15453

ID Verified

Trusted

Lifetime subscriber

#3261837 20-Jul-2024 15:14

Rikkitic:

This is exactly the reason I avoid automatic updates and only very selectively allow minimal others after waiting a decent time to make sure they aren't going to kill something important. So far it has worked for me.

There is a balance to be struck. You aren't in charge of keeping dozens or hundreds of users with varying degrees of IT Competency safe from themselves and all the people who want to do bad things.

Security is rarely convenient, and as someone who is responsible for MANY users, I'd rather this type of occasional massive inconvenience, over not having the protection offered.

I'd like to think that this particular issue was somewhat mitigatable with a little QA, but there is a balance to be had between the amount of testing you do, and the time that takes, over how quickly you are addresses loop holes or gaping security issues.

Tinkerisk

4798 posts

Uber Geek
+1 received by user: 3660

#3261855 20-Jul-2024 16:24

Not affected. Neither my customers nor I personally, neither directly nor indirectly. I would only be affected by an outage if the basic internet infrastructure were to fail, but then I would certainly have completely different problems. If the POS systems fail, I simply pay with cash, that's it. I say it again and again, even now: don't become dependent on clouds and tech giants, even if it is practical and convenient.

I have been responsible for safety and testing all my professional life. My thesis that maintenance and tests account for 1/3 of the development costs of a technical product could not be refuted by any smart manager in decades - in the end it was true.

- NET: FTTH & VDSL, OPNsense, 10G backbone, GWN APs
- SRV: 12 RU HA server cluster, 0.1 PB storage on premise
- IoT: thread, zigbee, tasmota, BidCoS, LoRa, WX suite, IR
- 3D: two 3D printers, 3D scanner, CNC router, laser cutter

Rikkitic

Awrrr
19062 posts

Uber Geek
+1 received by user: 16302

Lifetime subscriber

#3261858 20-Jul-2024 16:30

I'm only speaking as an individual user of course and I'm not advocating anything for others, but it really annoys me when I allow an update to proceed unimpeded it breaks something I have come to rely on and I have to spend hours or days finding fixes or workarounds.

Plesse igmore amd axxept applogies in adbance fir anu typos

New World

Shop on-line at New World now for your groceries (affiliate link).

Oblivian

7345 posts

Uber Geek
+1 received by user: 2117

ID Verified

#3261865 20-Jul-2024 16:43

Tinkerisk: If the POS systems fail, I simply pay with cash, that's it. I say it again and again, even now: don't become dependent on clouds and tech giants, even if it is practical and convenient.

Spoken like someone who hasn't read what actually was effected?

Cash was unable to be accepted at many a place also. Depending on the level of product tracking required for the product or service you were engaging in an agreement with.

It wouldn't have helped you at many public transport stations either because the system the tickets would normally be sold with that cash. Which is why they had to go free.

This wasn't just relying on a tech giant. This is one of the companies in place that arguably actually stop this sort of thing happening on scale.

Behodar

11094 posts

Uber Geek
+1 received by user: 6071

Trusted

Lifetime subscriber

#3261869 20-Jul-2024 16:52

Tinkerisk:

Not affected. Neither my customers nor I personally, neither directly nor indirectly. I would only be affected by an outage if the basic internet infrastructure were to fail, but then I would certainly have completely different problems. If the POS systems fail, I simply pay with cash, that's it. I say it again and again, even now: don't become dependent on clouds and tech giants, even if it is practical and convenient.

I have been responsible for safety and testing all my professional life. My thesis that maintenance and tests account for 1/3 of the development costs of a technical product could not be refuted by any smart manager in decades - in the end it was true.

Same sort of thing here. Personally not affected at all. I'm aware that EFTPOS outages happen occasionally so I try to carry a little cash. We went out for a family dinner last night and when we arrived we asked whether their EFTPOS was working. This resulted in a bit of a blank look and a "yes, why wouldn't it be?", but of course restaurant workers wouldn't have been looking at the news during rush hour!

I'd left something in the office on Friday and popped in this morning to pick it up. No sign of any issues: my PC looked just how I left it.

Tinkerisk

4798 posts

Uber Geek
+1 received by user: 3660

#3261871 20-Jul-2024 17:13

Oblivian:
Tinkerisk: If the POS systems fail, I simply pay with cash, that's it. I say it again and again, even now: don't become dependent on clouds and tech giants, even if it is practical and convenient.

Spoken like someone who hasn't read what actually was effected?

… This wasn't just relying on a tech giant.

Apparently not fully awake, it's about ANY dependency that you as a person CAN avoid. My grocery store around the corner worked, it's the giants that are vulnerable like dinosaurs. Just because EVERYONE does or doesn't do a certain thing doesn't mean it's right. Afterwards, the howling is all the greater and the words “should have, would have, ...” come too late.

Tinkerisk

4798 posts

Uber Geek
+1 received by user: 3660

#3261876 20-Jul-2024 17:32

If the medical, transportation and payment infrastructure of a country collapses due to a brain fart of a small developer in the USA, someone still hasn't understood something fundamental. And if it's said with a typical admin attitude that the (IT) world won't change for a long time because of this, then you just change your behavior as an individual. And many individuals are a whole myrade ... if they (would) think.

We can talk about deterministic systems (whereby Windows is immediately ruled out because it is not deterministic), which are used as operating systems in critical military and civil aviation systems, for good reason. Why is that? It wasn't the airplanes, stores and banks themselves that failed, but the stupid infrastructure, which shouldn't even bother with Windows BECAUSE it is critical.

neb

11294 posts

Uber Geek
+1 received by user: 10018

Trusted

Lifetime subscriber

#3261883 20-Jul-2024 17:57

Batman:
msukiwi:
Reanalyse:.....I am relieved to see what services still work despite this massive issue.
Thankfully Geekzone is still up.

But how? Curious.

There's a great circa-2000 User Friendly comic where they get asked why they're still up and running during another one of MS's global Windows crash events, and Pitr responds "Because we are not running crap on our servers". I think the same applies here.

Samsung

Shop now on Samsung phones, tablets, TVs and more (affiliate link).

freitasm

BDFL - Memuneh
80646 posts

Uber Geek
+1 received by user: 41030

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#3261885 20-Jul-2024 17:59

neb:

Batman:
msukiwi:
Reanalyse:.....I am relieved to see what services still work despite this massive issue.
Thankfully Geekzone is still up.

But how? Curious.

There's a great circa-2000 User Friendly comic where they get asked why they're still up and running during another one of MS's global Windows crash events, and Pitr responds "Because we are not running crap on our servers". I think the same applies here.

I'm not running CrowdStrike, but I'm running Windows.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

neb

11294 posts

Uber Geek
+1 received by user: 10018

Trusted

Lifetime subscriber

#3261887 20-Jul-2024 18:01

BarTender: I do firmly blame much of the situation on Microsoft being unable to supply an OS with sufficient hardening steps that a separate vendor is required for EDR rather than the OS being sufficient.

It's not really Microsoft's fault, the "security" products hook into the OS at very low levels using undocumented internal APIs and hacks, there's not much MS can do about that. And in terms of "a separate vendor required", they already landed in enough legal hot water when they started adding their own AV to Windows in competition with established AV vendors, trying to do what the various security vendors do as part of Windows would be lawsuit suicide.

neb

11294 posts

Uber Geek
+1 received by user: 10018

Trusted

Lifetime subscriber

#3261888 20-Jul-2024 18:04

Posted by a friend after being up all night working on this:

The process for recovery:

0: if bitlocker_key == unobtainium goto 0
1: if useriq == room_temp goto 1

Tinkerisk

4798 posts

Uber Geek
+1 received by user: 3660

#3261892 20-Jul-2024 18:15

neb:

Posted by a friend after being up all night working on this:

The process for recovery:

0: if bitlocker_key == unobtainium goto 0
1: if useriq == room_temp goto 1

🤣

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9