jnimmo: Hmm now the Maxnet / FYX DNS servers seem to be unusable.. switched to Google and it is working again...
I've lost Geekzone avatars using Fyx DNS (they were working earlier). I also don't have a Post button so hopefully clicking in that general area will post this :)
After getting a briefing this morning, here is what happened.
We actually two separate events.
1. a Dos attack 2. A firewall change from which BGP didn't recover automatically and needed manual intervention. This is what caused the DNS server to not be available.
We'd like to apologise for the problems last night and thank you for your responses and engagement. We are working on finding the root causes of these issues and will implement changes to ensure events like these are do no occur again.
Thanks for the explanation Andrew. Just as a suggestion for the future, it would be great if customers felt like we had a way to get in contact or to hear from Fyx/Maxnet after hours in the event of something similar. You were probably made aware of the problems by your monitoring systems, but it probably helps the customer experience if we at least feel like we have some official channel to go down if the internet breaks in the evening. (Even if it was just a case of something triggering in your monitoring system if there was sudden influx of mentions @FyxNZ messages on twitter!). Cheers
I was actually made aware of these issues through Twitter/Geekzone etc. I've found that these are a more accurate picture of real world problems, and the techie graphs etc (which are always up for flavours of interpretation in large, complex designs). Also, I'm far more interested in the user experience than what the internal guys SAY the user experience should be.
I've set up additional personal alerts for geekzone posts etc, so I'll be extra responsive here also!
And finally, after some additional digging around we have found a few things can tweak to get extra performance out of the network for out xDSL customers and will be implementing these in the coming days.
Fair call, just hope your weekends don't get too interrupted :) General web browsing has been fine since the other day, but finding streaming media is constantly buffering (from US), which had only happened to me once or twice on previous ISP. Sometimes it has been working great but very intermittent. Speed tests seem good, 2Mbps to LA, but seems like there is high latency and jitter when the problems are happening. http://www.pingtest.net/result/64150650.png I realise I can't isolate problems from ISP to streaming service, but just know that it hadn't been an issue until switching..
We will be shuffling users between BUBA and EUBA depending on traffic profiles in the coming days do we get the best performance for all users. Metrics shown today give us a clear indication that this will SIGNIFICANTLY improve the network bottlenecks causing these issues.
That, plus a whole lot of new hardware and a lot more international bandwidth and we will be steaming along as quick as ever.
HTe time has come! ... As a small/specialist ISP we've needed to be careful not to bugger up the BUBA dimentioning for the 20ish% of customers in NZ who can't get EUBA.
We have been using EUBA for specific business requirements on a one off basis for a number of years, so nothing new here, just some additional effort going into ISP in the past few months means some more changes to the way we do things technically so we can grrepaid still remain kickarse.
Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly
to your computer or smartphone by using a feed reader.
Trusted
