Telecom advises that it has restored service to the vast majority of customers, but is monitoring the network closely and urgently investigating the underlying cause of the problem.
Telecom's CEO Retail Chris Quin says customers expect a reliable and robust broadband service from Telecom and rightly so given the role the internet now plays as a major communications tool.
"We have a comprehensive review underway to ascertain how these issues can be minimised in the future and apologize unreservedly to customers for any disruption these outages caused in the interim."
Telecom apologises to customers affected by these incidents and the inconvenience they have caused.
The first outage occurred early yesterday morning with service restored by 8.15am.
Another outage last night was identified at 9.45pm with service restored by 11.35pm (although it took up to an hour or more for some customer modems to be reconnected).
The first outage related to a problem with a software upgrade on an Auckland-based server that is part of a system which handles just over 300,000 customers, representing approximately two-thirds of Telecom?s broadband customer base. The problem meant customer modems were failing to authenticate when trying to communicate with the server. The Auckland server normally runs in parallel with a Christchurch based server and service was initially restored by removing the Auckland server from the system and running solely on the Christchurch server.
Yesterday afternoon service resumed on both parallel servers, but last evening further problems were experienced with the Christchurch server overloading due to issues synchronising information between the two servers. To address this issue, the Christchurch server was taken off the system late last night, with internet traffic all going through the Auckland server.
Close monitoring overnight indicated that some customer modems were having intermittent problems connecting to broadband, which is believed to have been caused by the capacity pressure on the single Auckland server. The Christchurch server was brought back up just after 9.00am this morning and the number of internet connections improved rapidly from this time, with 90%+ active connections by 9.30am.
Telecom helpdesks are also experiencing higher than normal levels of calls for this time of the day, but this may be due to customers being unable to connect because they need to reboot their modems first. Although most modems will reconnect automatically, a minority need to be turned off and back on again.