Waikato DHB IT Outage

Forums › IT Pro and developers › Waikato DHB IT Outage

1 | 2 | 3 | 4 | 5 | 6

BlinkyBill

1443 posts

Uber Geek
Inactive user

#2713984 27-May-2021 16:08

wellygary:

"The health board was working through about 680 computer servers that needed to be sanitised, restored, and brought back online, he said."

https://www.stuff.co.nz/national/politics/125260696/waikato-dhb-cyber-attack-it-could-take-weeks-to-resurrect-nearly-700-computer-servers

680 Severs?? that sounds like a huge number... but it sounds too many for a server count?? but too small as a workstation count...

That number is similar to the number of servers at the 5 DHB’s my company does work for. At these 5, which are among the larger ones, the servers would be 10-20% utilised. It’s almost as if they’ve never heard of virtualisation!

The reason for this sprawl is a plethora of technologies used to deliver the multitude of applications used; and a horrible build-up of technical debt - no effort to consolidate or modernise.

One of these DHB’s has 1,200 different applications - and for some of those they don’t know if the application is used by anyone! I asked once if they shouldn’t just turn it off and see if anyone noticed and they said no, in case they’re used once a year!! Of course, there is software that figures that out, but no they can’t find time or dollars to manage their software assets.

cruxis

482 posts

Ultimate Geek

#2713991 27-May-2021 16:29

wellygary:

cruxis:

I agree it won't happen this year. Maybe in 2022 or 2023. I hope it becomes just like buying a flu shot is today.

Flu shots are recorded on the National Immunisation Register.. even walk ups

https://www.influenza.org.nz/recording-influenza-vaccinations-nir

You dont have to be. As I have walked up multiple years, and had no problems buying a shot. Just answered few health questions, about allergies. There is a little box/field you tick to opt off the NIR

Beccara

1469 posts

Uber Geek

ID Verified

#2714043 27-May-2021 18:05

I'd assume it's VM's which is completely reasonable for a place with 8000 staff with specialist applications thrown into the mix. Just a proper AD setup for that size would be 8-10 VM's from memory

Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated

CB_24

366 posts

Ultimate Geek

#2714120 27-May-2021 21:22

wellygary:

"The health board was working through about 680 computer servers that needed to be sanitised, restored, and brought back online, he said."

https://www.stuff.co.nz/national/politics/125260696/waikato-dhb-cyber-attack-it-could-take-weeks-to-resurrect-nearly-700-computer-servers

680 Severs?? that sounds like a huge number... but it sounds too many for a server count?? but too small as a workstation count...

(From experience working in DHBs)

Most services delivered by IT will have Dev, Test (multiple servers), Prod (multiple servers) environments so around 680 sounds about right to me.

I'm guessing they wont be too worried about rebuilding Dev (and maybe Test) right now though.

billgates

4705 posts

Uber Geek

Trusted

#2714122 27-May-2021 21:23

680 servers are likely VM’s as mentioned above. There are companies in NZ with 8 to 10 times bigger than this in terms of numbers of VM’s to give you a scale of things. Backing up machines incrementally is one component that most are happy with on a 1gbps network link that you do not notice or think about during planning stages but having to restore all your backups over that same 1gbps link takes a lot of time.

Do whatever you want to do man.

Batman

Mad Scientist
29768 posts

Uber Geek

Trusted

Lifetime subscriber

#2715393 30-May-2021 10:01

Something about more investment in cybersecurity needed. Someone can explain if article is right or wrong, I have no idea in this field

https://i.stuff.co.nz/business/125180968/ministry-of-health-abandoned-cybersecurity-system-for-waikato-and-other-dhbs-due-to-budget-issues

Beccara

1469 posts

Uber Geek

ID Verified

#2715594 30-May-2021 17:31

Maybe? There's going to be alot of people pushing products (well moreso than usual) but the sometimes you're just going to get hit if you've got a nation state aimed at you, look at SolarWinds where you monitoring and security system itself was hacked. CrowdStrike is one of the new comers to the "nextgen" AV space and has a good rep. That said there's plenty of mass-encryption detecting options in AV land so why this got so far without being halted or seen is unknown right now. I always try and keep in mind the scale of some of the hack's we've seen and a prime example is Maersk being globally taken down by a similar type of ransomware

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

Technofreak

6530 posts

Uber Geek

Trusted

#2730555 18-Jun-2021 10:56

Over four weeks now since the ransomware attack and still counting. I need some information from the DHB before I can return to work and have been told it may still be several weeks before the system containing that information is up and running.

I doubt I am alone in my situation. It's very frustrating especially when you see articles that infer things are getting back to normal yet you know there's still a long way to go. You have to wonder how deep the penetration was and how much data has potentially been lost altogether.

You really hope other institutions have got their act together with their IT defence systems.

Sony Xperia XA2 running Sailfish OS. https://sailfishos.org The true independent open source mobile OS
Samsung Galaxy Tab S6
Dell Inspiron 14z i5

networkn

Networkn
32354 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#2730559 18-Jun-2021 10:59

Technofreak:

Over four weeks now since the ransomware attack and still counting. I need some information from the DHB before I can return to work and have been told it may still be several weeks before the system containing that information is up and running.

I doubt I am alone in my situation. It's very frustrating especially when you see articles that infer things are getting back to normal yet you know there's still a long way to go. You have to wonder how deep the penetration was and how much data has potentially been lost altogether.

You really hope other institutions have got their act together with their IT defence systems.

That sounds pretty tough, sorry to hear it.

https://www.geekzone.co.nz/forums.asp?forumid=86&topicid=285793&page_no=5#2713702

Should explain why it's not really as easy as just restoring from backup. They will need to be *very* sure they aren't taken down again by stuff left over or embedded in prior backups, otherwise this gets so much worse.

Technofreak

6530 posts

Uber Geek

Trusted

#2730580 18-Jun-2021 11:48

networkn:

That sounds pretty tough, sorry to hear it.

https://www.geekzone.co.nz/forums.asp?forumid=86&topicid=285793&page_no=5#2713702

Should explain why it's not really as easy as just restoring from backup. They will need to be *very* sure they aren't taken down again by stuff left over or embedded in prior backups, otherwise this gets so much worse.

Thanks.

I don't work in IT but I understand the issues involved especially around restoring from back ups and the high probability of "restoring" the embedded ransomware. It's just the time it's taking that's frustrating.

Also I have to wonder about how far back they have to go to find a copy that isn't corrupted (assuming they have historical copies) and how much data between then and now might have been lost, or is there a way to extract up to date data from a compromised backup?

My fear is that as my information was very recent data that it may be lost altogether. If it's lost it can be "recreated" and then I might as well be going down that path now rather than waiting for the system to be brought back online.

Sony Xperia XA2 running Sailfish OS. https://sailfishos.org The true independent open source mobile OS
Samsung Galaxy Tab S6
Dell Inspiron 14z i5

networkn

Networkn
32354 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#2730586 18-Jun-2021 11:52

Sure. It's somewhat likely they may be taking the approach of removing the OS for the servers and only restoring the data. Much more time consuming, especially since that would likely involve many outside vendors with their own timeframes and requirements. They likely are also taking the time to work out what the better structure for IT is, to minimize the attack surface again.

I can totally understand the frustration.

frankv

5680 posts

Uber Geek

Lifetime subscriber

#2730609 18-Jun-2021 12:41

Technofreak:

My fear is that as my information was very recent data that it may be lost altogether. If it's lost it can be "recreated" and then I might as well be going down that path now rather than waiting for the system to be brought back online.

I don't see that as a real risk, except for the data since the last database backup, and even then there are log files, so should be restorable up to a few seconds before the shutdown.. The database itself is separate from the applications that read it, so they should be able to install the application code (from scratch if necessary), then restore the database.

Batman

Mad Scientist
29768 posts

Uber Geek

Trusted

Lifetime subscriber

#2736258 29-Jun-2021 11:57

data now being teased online

https://www.stuff.co.nz/business/125592089/ransomware-attack-waikato-dhb-documents-appear-to-have-been-released-online

tehgerbil

1102 posts

Uber Geek

ID Verified

Subscriber

#2736311 29-Jun-2021 14:00

Batman:

data now being teased online

https://www.stuff.co.nz/business/125592089/ransomware-attack-waikato-dhb-documents-appear-to-have-been-released-online

As if stopping sick people getting medical treatment isn't sadistic enough, what kind of evil person do they have to be to then release their private information?

1 | 2 | 3 | 4 | 5 | 6