The big fibs the European Central Bank (ECB) told us about Cyprus being a special case with the Russian mafia money only lasted 3 days before the truth came out that other PIGS countries’ (Portugal, Ireland, Italy, Greece and Spain) depositors could also be on the hook for the reckless gambling of their banks.
The golden rule of banking over the last century has been that deposit accounts were private property, and if a bank collapsed, then the shareholders and bond holders would lose their money first. The bank would be wound up, inquiries would be held, the assets sold, and the last people to lose money were the depositors. This process could take years.
A depositors’ bank account was private property, and governments worked hard to make sure this money was guaranteed up to a certain amount (usually $100,000 to $250,000 in most OECD banks).
Investors on the other hand are treated very differently, as these investments (such as the share market, or managed funds) are accepted as more risky but of course have better returns.
Helen Clarke and the Labour Government stepped up to the mark during the financial crisis of 2008, stopping a potential bank run by assuring depositors’ that their bank accounts were guaranteed.
Our ex-Merrill Lynch and US Federal Reserve banker, John Key, along with other leaders around the world have changed this golden rule to make sure the bankers, shareholders and bond holders don’t lose any money, but as an “investor” – note this wording change – the money in your everyday bank account could shrink 10%-80% overnight.
No big inquiries, no bankers losing money. Just your savings. Overnight. Gone. That’s what Mr. English and Mr. Key are planning to have by July.
So, where are we at in this plan to steal your money overnight if you are fortunate enough to have worked hard, and put away a nest egg?
From NZ Herald
“…New Zealand banks are readying their IT systems for Open Bank Resolution, a Reserve Bank policy that in extreme cases like insolvency would see a bank's losses shouldered in part by its shareholders and creditors - including everyday depositors…”
“…The Reserve Bank has the power to freeze bank deposits but up to now has lacked the technical infrastructure to implement it - hence their requirement for banks with retail deposits of more than $1 billion to change their systems and meet their requirements by July 1…”
[link to www.nzherald.co.nz]
Four out of five of New Zealand’s major banks (excluding National Bank) have had scheduled outages of their online banking systems and ATM access in the last 60 days. Has the new Cyprus style Open Bank Resolution software updates already been installed at these banks and the Reserve Bank of New Zealand?
Kiwibank 24th Feb – 3 ½ Hour Scheduled Maintenance Internet Banking
For all you night-owls out there, a heads up that we'll be doing some scheduled maintenance on internet banking from 1.30am on Sunday morning (24th February). We aim to be finished by 5am. Internet banking won't be available during that time. Thanks in advance for your patience. ^RM
[link to www.facebook.com (secure)]
ANZ 10th March 2 Hour Scheduled Maintenance Internet Banking
While we make this update, ANZ Internet Banking, goMoney for Android and iBank will be unavailable from 11:30 pm, Friday 08 March until 6:00 am, Saturday 09 March. They will also be unavailable due to scheduled maintenance from 1:00 am till 3:00 am on Sunday 10 March.
[link to comms.anz.co.nz (secure)]
WestPac 21st March 2 Hours Scheduled Website/Trading Platform Maintenance
The trading website will be unavailable for placing trades, managing orders and checking holdings between 17:00 and 19:00 (Sydney time), Thursday 21 March 2013 whilst we perform scheduled maintenance. We apologise for any inconvenience this may cause.
[link to securities.westpac.com.au (secure)]
ASB Bank 8th March 1 Hour Scheduled Maintenance Internet & Phone Banking
Internet & mobile banking will be unavailable between 2am – 3am on Saturday due to scheduled maintenance.
[link to webcache.googleusercontent.com]
BNZ Bank 18th January 5 Hours Scheduled Maintenance Internet and Phone Banking
We're doing maintenance from 12-5am tomorrow morning which will make Internet, Mobile & Phone banking unavailable. Transfer your cab $ now.
[link to twitter.com (secure)]
Cooperative Bank 18th January 1 ½ Hours Scheduled Maintenance Internet, Phone and ATM Access
On Friday 18 January 2013 there is a scheduled outage to allow for maintenance to the Loaded Everyday product.
This means that between the hours of 5.30am -7.00am on 18 January those members with a Loaded Everyday card won’t be able to:
[link to webcache.googleusercontent.com]
Reserve Bank of New Zealand – Partial List of Scheduled Maintenance for NZ Bank Clearing Systems (non-SWIFT)
ESAS-NZClear change release Outages on 2nd March, 23-24 March, 27-28 March, 20-21 July, 23-24 November
- ESAS is the Reserve Bank of New Zealand’s Exchange Settlement Account System which is used by banks and other approved financial institutions to settle their obligations on a Real-Time Gross Settlement (RTGS) basis.
- NZClear is a real-time settlement system which members access from their premises. This system was formerly known as Austraclear.
[link to www.rbnz.govt.nz]
How does New Zealand compare to Cyprus, which is already under lockdown with the longest bank holiday in history?
The total New Zealand debt of NZ$150 billion is equivalent to about EU$21,682 (NZ$33,624) per New Zealander.
Cyprus is sitting at EU$18 billion is equivalent to about EU$16,120 per Cypriot.
So, the big banks as well as the New Zealand Reserve Bank have been busy pressing update on their software, and we owe more per person than Cyprus which has experienced a dramatic lose of depositors funds.
The question I am still pondering, is why Mr. Key and the National Government risk potentially triggering a catastrophic bank run with these policies when kiwi's realize their mattress won't have a catastrophic failure and take 30% of your money overnight, unless National knows something we aren't privy to.
I'll leave you with a comment from a industry expert
“… it is disgraceful that out government is placing the onus of bank support on the public of New Zealand with zero debate and very little reporting. Bill English must be very upset with the events in Cyprus raining on his parade, of course it is a year behind in its implementation as the banks did not know how to achieve what the Reserve Bank was wanting the to achieve in only one days Bank Holiday. It will be interesting to see what transpires.”
Quote from a anonymous opponent of OBR in the precious metals sector…
All us humble geeks in the IT industry ask is that you don’t forget about that most loyal member of your team during these festive times.
You know, the one who always volunteers to work through the Christmas break, slaving over the incoming e-mails and random jobs clients’ demands during those non-stat days? The one who spends summer stuck in the office, gazing absent-mindedly out the window at deserted streets, wishing they were at Piha with the fresh wind blowing in their face instead of in the stifling heat, facing a breakdown of epic proportions upon everyone’s arrival back at work.
That’s right, I'm talking about your humble network server.
With the hopes of a record summer heat wave, good workmates don’t let their servers drink and cook. So, before you leave for the year, please check with your building staff to ensure the office air conditioning is going to be left on (or at least the air conditioning in the server room, if you have one). That way, your faithful colleague won’t slowly keel over while you’re away (or worse, go completely postal before shutting down altogether), and you’ll be able to ease back into work by putting your feet up and slowly checking those hundreds of new e-mails without a painful and expensive week long server outage.
Us geeks will, as always, will be standing by for those forgotten passwords and water logged devices indicative of a great summer break upon your return :-)
Otherwise, we wish you the very best for the holiday season. Have a cold one for us, and maybe another for that guy working 24/7 in your office while you bring in the new year.
...And remember to thank grandma for those beautiful new handkerchiefs.
The announcement, while being a surprise, was not entirely unexpected with Apple's cancelling the Xserve RAID and transitioning everyone to the Promise VTrak SAN in February 2008. Apple have never been afraid to exit markets they are not the premium product in, or are not making much money in. Their focus is consumer electronics', which is where the Mac Mini and Mac Pro sit. They are the premium brand in these markets for Phones/music players/computers/tablets and command a premium for their products. If I was Steve I would rather have those production lines churning out iPad's then a few xserve's.
On the other hand, the units of xserve's they have been shipping (predictions are in the range of 10K to 20K a quarter) have been dwarfed by their other product lines, but I imagine need just as much R&D, and a far greater commitment to support resources. Apple never really got their head around the requirements of Enterprise support - dropping your server off at a Apple Authorised Repair centre is not an acceptable solution.
The server market is in a race to commoditization and consolidation around standard technology for compute and storage resources to drive per instance virtualization costs down. Steve Jobs said in the D8 interviews that Apple must control the key technologies in the markets it competes in otherwise it gets thumped, and that's not a reality in the server hardware market.
Pound for pound, their existing Xserve offers 5 less drives and 144GB less memory per 1U form factor than comparable solutions from HP and IBM. Ouch, sounds like a thumping to me.
This seems to be Apple's thinking as they have appeared to buy IBM and SUN servers for their billion dollar cloud data centre which will run a mix of IBM/AIX, Linux, SUN/Solaris and OS X. One does wonder if Apple will now use Mac Mini's or Mac Pro towers in their data centre (tongue firmly in cheek), or virtualize OS X.
OS X does not currently perform very well on a virtualized platform, but Apple have spent the last two OS releases optimising the nuts and bolts of the OS with Snow Leopard and Lion (out soon), and I imagine are also retooling OS X during this to take advantage of virtualization. Microsoft went through the same retooling with the last two versions of Windows to make it perform well in a virtualised environment.
The companies building the biggest data centres are the ones with the biggest cloud ambitions, and Apple's 500,000 square foot one is as big as they come with room to plonk another one right beside it. I can see Apple releasing a slew of cloud based subscription services out of this massive data centre for their consumer electronics' products, such as a cloudy iTunes, online file sharing/backup and now with iTV Streaming but the interesting question then becomes of what happens to their stranded enterprise and post-production video solutions like Final Cut Server, as this will not be a feasible cloud based service.
Final Cut Server is a fantastic solution for post production video workflow for TV Stations, video editing (e.g. ad agencies) and anyone with broadcast ambitions. What does Apple have in mind for these clients? What is their future in this non-Xserve world?
Some people believe that Apple will license OS X Server for deployment on specific non-mac hardware in the same way as Apple partnered with Promise for the XSAN solutions, others see Apple offering OS X Server for virtual appliances which enterprises can deploy to their existing private clouds. Both of which removes Apple's enterprise SLA problems, but still allows them to control the key technologies in this market.
What do you think Apple is planning? Love to hear your thoughts in the comments below especially if you have a current or planned Xserve deployment.
Until then, I leave you with this.
Or alternatively "how to make your high end RAID array perform worse than a laptop drive". If you have ever wondered how bad the performance on a RAID array performs with no battery backed write cache, the hard numbers are below and it's not pretty. It's not even Pretty woman pretty. Hide your wife and kids people - these numbers get ugly.
I've hit this problem numerous times within the last year, as IBM abandoned Adaptec RAID controllers (rebranded as ServeRAID) and moved to LSI Logic, and IBM were shipping them in the first year without cache batteries. A casual glance over the pricing and product descriptions does not reveal the missing cache backup batteries, and as the new range are more expensive than the old cards, the cache batteries then double the cost of the RAID cards.
HP's kit doesn't allow you to create RAID5 arrays on controllers without a cache battery - but they allow RAID5 arrays to continue running when the cache battery fails. A smarter approach, I reckon.
When you don't have the magic cache battery, the RAID controllers cache lurches into 100% read caching only, like a dodgy gearbox stuck in second. As the numbers below show, this creates horrific performance problems on RAID Arrays, allowing a single desktop hard drive to out perform a 4 disk 10,000 RPM SAS array by up a factor of 21 on write operations. Hope no one brought one of these for Exchange or SQL or Virtualisation!
I'm no hardware guy, sure I've been building clusters and arrays and servers for longer than my belly is round, but even I was stunned by these numbers - I knew performance would be worse, knew the missing write cache was a performance bottleneck, but didn't expect read operations to also take a hit as well. This doesn't sit right with me, but these are the numbers. I did a santa and checked them thrice, and Big Blue did too. They thought it was a driver issue and made me rebuild the whole server was their total confusion at such low numbers.
1. The desktop hard drive was a 3 year old 7200RPM drive.
2. The Tool used for testing is RD II (I prefer it over iometer)
3. The OS the testing was done on was a clean install of Windows Server 2008 R2 x64 SP0.
Single 7200RPM Desktop Hard Drive
IBM x3450 R2 With 256MB Cache Backed Battery (4 SAS Drives 10K RPM 2.5")
IBM x3450 R2 With 256MB no Cache backed battery (4 SAS Drives 10K RPM 2.5")
HP DL380 G5 with 512MB Cache Backed Battery (3 SAS Drives 10K RPM 2.5")
Many of these new boxes (as least for us) went in for Virtualization projects, as they were the first of the Nehalem based servers, which love a bit of VM goodness, but we were having quite lumpy performance.
We have now rolled out cache batteries to our fleet, and are surprisingly picking up a bit of new business due to other IT companies falling into the same trap and us spotting it.
Check your caches out there kids, and be safe on your writes.
I thought that hiring a new staff member would be easy, our company is doing well, we have lots of big interesting high profile projects in the pipeline and there is a surplus of workers due to these trying times. Boy was I ever wrong. Repeat after me, Real geeks interview badly. We nearly employed someone which 'interviewed' really well had real world experience, skills and certifications flowing over pages of their CV but could not walk three hours in our existing engineers shoes. We nearly missing out on a super smart amazingly fast learner who didn't have 'experience' and was not as confident as the study bunnies which know how to press flesh and smile while spouting off IT war stories to impress the bosses.
Since we have been bitten before by a bad staff hire, someone who's CV and skills didn't match up in any way to what we saw every day I decided to rig up a technical exam for the position. We needed something our engineers face on a daily basis, as real world as possible, a day in the life so to speak, without the bad Chinese food. I finalised on a exam which consisted of a Exchange Server with database corruption and a client request to enable Outlook Web Access with SSL/Forms Based authentication. We also broke DNS by adding incorrect hosts file entries and blamed it on the old IT guys which we were 'taking' the work over from. The candidates were allowed to use google and the internet for research and troubleshooting just like our engineers do every day and were given a maximum of 3.5 hours to complete this task.
Oh, and it wouldn't be a real world test without the demands of a Account Manager as well, so we also asked them to list out for the Account Manager what's wrong with the server before we 'took over' the responsibility of this server. We expected the candidates to find things like the Antivirus being 5 years out of date, no Windows Updates (ever), the blue screens of death, no UPS, no backups etc. You know, the basics. I wrote a exam document and marking check sheet to make sure we were marking fairly and consistently.
We had about 40 CV's submitted, and whittled that down to about ten, half were telephone interview only and half were in person interviews, that was the plan.
Apart from some fellow turning up late and then talking on his mobile during the interview, they all interviewed well, and one in particular, lets call her Jenny, had pages full of Microsoft qualifications and massive rollout experience overseas, Jenny was the only one qualified and that also had real world experience with Windows Server 2008 and Exchange Server 2007. Jenny was our favourite, our pick. Smart. Driven. Qualified. I imagined her working here, sitting at her desk, fixing the worlds problems, laughing with the other engineers over bad Chinese food.
We were left with 4 top notch candidates. We forgot about the phone candidates, including a guy which we'll call Johnny.
By this stage I had finally got the exam environment and exam ready and invited them all back to sit it. We made one of our existing engineers sit the test to set the bar and setup a remote viewer for the exam server so I could watch them in my office while they sat the exam, watched how they approached problems, their mythologies, how quick they were moving around, the use of shortcuts and what tools they used. It was amazing, I learnt more about those candidates during that exam time than any face / question time I had with them before or after. Incidentally we did tell them we were watching, but only after the exam finished so they wouldn't panic or feel any more pressure.
I watched people flounder around and focus on the wrong things, do damage to Active Directory, completely ignore important tools and errors and I also watched others literally learn software on the fly and resolve problems they had never seen, trained for or had qualifications on. We even had a guy which walked out five minutes into the exam after admitting he was out of his depth and bluffed his way in, but again interviewed quite well.
I was stunned. Absolutely shocked. Jenny our star, my pick, our most qualified candidate, floundered around and scored the lowest of all, she didn't try to repair the database, got fixated on benign windows errors a seasoned engineer knows to ignore, keep trying to start the services over and over without actually fixing anything in between and took the longest of everyone before we had to simply stop her as she ran the clock out.
Johnny, the guy we dismissed with not enough skills in the phone interview and had to frantically arrange a interview for along with some others hit it out of the park and bet everyone. He barnstormed through the exam. He took no prisoners while learning exchange server on the fly, figured out how the databases worked, found the eseutil Jenny, our star candidate, ignored completely. He made backups and tried different things with streaming and db files, checked complex raw eseutil logs, had it mastered and he did it in nearly half the time that Jenny took to not fix the database on a program she trained on, was qualified for. That Microsoft said she was up to snuff on.
Johnny bet everyone by a large margin. We nearly missed having the opportunity to have the best, smartest, quickest learning candidate and in a few years geek god, because we believed that those which interview well, that can memorise answers and are good at interviews would perform best in their jobs which involve none of those skills. I nearly made the same mistakes we made in the past hiring staff. That those with the best war stories and interview nous make the best engineers.
What I have actually found out is that the best engineers, are well, the best at engineering. It took us a while to work this out and actually know how to interview for it, but since we have it's opened my eyes to the vast difference in talking the talk and walking the walk, and more importantly in measuring the ability for someone to walk 3 hours in our engineers shoes.
Oh, and as for Johnny, he will be eating bad Chinese food and if your lucky working on your server soon.
I was excited to put this new server to the test, and consolidate various aging servers onto it. I wanted to have a great first blog post about it, but sadly I cannot rave about this server. It has brought me pain.
Processors / Memory
The new Nehalem processors are fast, very very fast. But reports of serious bugs with the timers/interrupts Microsoft went to publish a kb article about recommending not using these for Hyper-V before Intel stepped in and stopped them. The workaround was to disable the power saving features (turning off cores) but now there is a hotfix out which works around the processor bugs.
Slashdot | Microsoft Advice Against Nehalem Xeons Snuffed Out
The server is stacked with 16 RAM slots (8 per processor, i.e. if you don’t have a second processor then only 8 slots are usable.)
If you have ever installed RAM in one of these and are puzzled by the odd RAM installation numbering it’s to do with cooling.
IBM has had some serious data loss bugs with the ServeRAID 7k/8k RAID cards, and so IBM changed course and dumped Adaptec which made the old ServeRAID controllers and brought out their new RAID controllers which use LSI chips.
There’s three downfalls, the first is the famous, lovely, easy to use ServeRAID software has been replaced with a uglier, slow, buggy cousin "MegaRAID", and the price of the RAID card has increased by 50% (%100 if you count the cache battery.)
The other gotcha which caught me out is that the old RAID cards came with cache batteries (without which RAID runs at speeds 10x less than desktop hard drives in some operations, and the cache batteries are half the cost of the RAID card again. HP got this right, they make their own RAID chipsets (from what I can work out), and their RAID controllers on even their cheap servers come with RAID5 built in. With IBM you throw away the bundled RAID controller which only does RAID 1 and buy a new controller, and the cache battery. I hear the newer higher end RAID controllers will now ship with the batteries again. This was a silly mistake on IBM's part. The reasoning IBM give is that the batteries are now classified as consumables.
UEFI - The Replacement for the 20 year old BIOS
The X3650 M2 is one of the first servers from IBM with the UEFI (Unified Extensible Firmware Interface) which is the replacement for the aging BIOS. I love the concept, and it needed to be done, you can configure ALL your adapter’s in the main ‘BIOS’ screen, including RAID cards, network adapter firmware settings etc instead of waiting for the correct time in the boot sequence and choosing the specific adapter firmware to configure.
The problem is IBM's implementation is painfully slow. It takes around 4 minutes between plugging power into the server and being able to turn it on (the power light flashes rapidly during this period), it then takes approximately another 4 minutes to go through the firmware initialisation before you can then boot the OS. The ‘BIOS’ Raid Configuration screens are also so painfully slow that I was stunned how it was possible to take such a fast server and make it display graphics so slowly! The shipping version of the "WebBIOS" for the RAID also had bugs where if you didn't click on a button the whole BIOS locked up hard. Time for another 4 minute power cycle. It was like playing Minesweeper!
I have a feeling there will be many firmware updates to improve the performance of the UEFI of this server over its lifecycle.
There is also a discussion going on about the finer points and other problems found such as the IMM Windows interface (which is USB) dropping domain controllers firewalls to Public mode causing all kinds of grief.
First real world experiences with IBM’s x3650 M2
Integrated Management Module (IMM)
The IMM is much improved, and now has a web interface standard, instead of having to upgrade to the RSA II Adapter, it has some cool features like taking a ‘picture’ of the server as it blue screens which it keeps a few of in the IMM so you can diagnose those hard to find problems, and a ploretha of data such as temp and fan speeds etc.
To note you still have to purchase the key to unlock remote screen control using the IMM when deploying servers remotely. This is still the same price as the RSA II adapters.
To note, we did have a problem with the cooling in our server room a few weeks ago, and the x3650 M2 is the only server which locked up hard. Older IBM boxes and Apple xserve's kept ticking over fine. The x3650 M2 likes it cool and freaks out when things get a little bit warm.
Microsoft Hyper-V VHD Bugs
Another issue we have stuck is a bug in Microsoft's vhdmp.sys which handles VHD files which blue screens on backup using Volume Shadow Copy.
We have one big data drive on our Hyper-V box with all the VHD's, we ALSO have a VHD which boots the physical Hyper-V machine so any software updates, windows upgrades are easier. This is something Microsoft supports but has a critical bug. There are discussions over here
vhdmp.sys BSOD 0x000000ca - StorageCraft Support Center
at the ShadowProtect forums and Microsoft said it was on the top 5 list for that product group which usually means a hotfix is out within 7 days but we are 3 weeks and counting.
While I wouldn't claim this is IBM's finest hour, the x3650 M2 is still a solid box, I think they x3650 M2 will get better over time with firmware updates but right now there are a lot of rough edges with this box.
When this server is deployed with the correct RAID cards and cache batteries and enough RAM it makes a great Hyper-V virtualisation platform. We have had ours in production which has half a dozen in production servers and about that again test machines and screams along.
My next post will cover some of the Hyper-V and Virtualization goodness.