Spark DNS Issues - Amazing - Broadband Service Alert

Forums › Spark New Zealand (including Skinny and BigPipe) › Spark DNS Issues - Amazing - Broadband Service Alert

1 | ... | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21

44 posts

Geek

#1125576 10-Sep-2014 09:28

Sfitz: From Spark this morning ... it looks like you may be using a Netcomm router, is that correct? If so, some of these modems have been
blocked from checking DNS due to being vulnerable to attacks like we had on the weekend. Applying DNS to each device
should still allow each device to get online.

My problem with Spark is they haven't published that they have blocked users and provide more detailed information. Instead they have just left users not working and announce in the media all is good.
Setting Google DNS in the router rather than Spark ones does not resolve the problem as it would appear the router itself is blocked from DNS however setting up your own internal DNS or setting DNS on each device does work.

Mines a Dynalink RTA 1025W and I'm in the same boat, so it's not just the Netcomms. I 100% agree without about the announcement that it's all working again, while some users are just sitting in internet darkness.

Same findings about the end device DNS, though to be honest - normal service could be restored but since I've already set all my devices to Google DNS, I haven't bothered testing.

Talkiet

4793 posts

Uber Geek

Trusted

#1125580 10-Sep-2014 09:34

toxicbubble:
Sfitz: From Spark this morning ... it looks like you may be using a Netcomm router, is that correct? If so, some of these modems have been
blocked from checking DNS due to being vulnerable to attacks like we had on the weekend. Applying DNS to each device
should still allow each device to get online.

My problem with Spark is they haven't published that they have blocked users and provide more detailed information. Instead they have just left users not working and announce in the media all is good.
Setting Google DNS in the router rather than Spark ones does not resolve the problem as it would appear the router itself is blocked from DNS however setting up your own internal DNS or setting DNS on each device does work.

Mines a Dynalink RTA 1025W and I'm in the same boat, so it's not just the Netcomms. I 100% agree without about the announcement that it's all working again, while some users are just sitting in internet darkness.

Same findings about the end device DNS, though to be honest - normal service could be restored but since I've already set all my devices to Google DNS, I haven't bothered testing.

While using the Google DNS servers is a perfectly reasonable short term fix, you will likely be directed offshore for Akamai content, although Youtube content should still come from an optimal location.

With the tiny number of users currently using non Telecom DNS servers, it makes next to no difference for our international link capacity, but your user experience will be worse.

On top of that your DNS responses are now probably at least 45-70ms, instead of 5-30ms for the Spark DNS servers. (Depending on where i NZ you are and your DSL profile)

TL;DR: It's worth setting the Spark DNS servers again at some point.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Jarsky

142 posts

Master Geek

#1125587 10-Sep-2014 09:46

toxicbubble: Mines a Dynalink RTA 1025W and I'm in the same boat, so it's not just the Netcomms..

Dynalink = Netcomm ;)

Sfitz

22 posts

Geek

#1125595 10-Sep-2014 09:52

Mine is also a Dynalink RT 1025. Luckily I had a Raspberry Pi available and installed dnsmasq which includes DNS caching.
Actually response time is now visibly faster when loading pages like NZ Herald with many different links for advertising, than using Spark DNS (even when it was working).

Once again, blaming the modems still doesn't answer the question why only Spark affected. Still seems like we are not getting the full story.

Talkiet

4793 posts

Uber Geek

Trusted

#1125596 10-Sep-2014 09:55

Sfitz: Mine is also a Dynalink RT 1025. Luckily I had a Raspberry Pi available and installed dnsmasq which includes DNS caching.
Actually response time is now visibly faster when loading pages like NZ Herald with many different links for advertising, than using Spark DNS (even when it was working).

Once again, blaming the modems still doesn't answer the question why only Spark affected. Still seems like we are not getting the full story.

1) We're still working on it. Giving the full story is less important than resolving all the issues.

2) Yep, if someone wants to install their own caching DNS server then that's great and you will see better performance... Sub millisecond DNS lookups for cached entries are better than even 10ms lookups to our servers. I personally run a caching DNS server at home as well.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

martinjward

68 posts

Master Geek

#1125602 10-Sep-2014 10:19

Sfitz: From Spark this morning ... it looks like you may be using a Netcomm router, is that correct? If so, some of these modems have been
blocked from checking DNS due to being vulnerable to attacks like we had on the weekend. Applying DNS to each device
should still allow each device to get online.

My problem with Spark is they haven't published that they have blocked users and provide more detailed information. Instead they have just left users not working and announce in the media all is good.
Setting Google DNS in the router rather than Spark ones does not resolve the problem as it would appear the router itself is blocked from DNS however setting up your own internal DNS or setting DNS on each device does work.

Yep, totally agree with this, the media release should state; we are still under attack/dealing with this issue, however we have managed to largely mitigate the issue, unfortunately our mitigation has inadvertently/(purposefully) blocked a small amount of users from even connecting to our network, in which case DNS server changes wont help them and they need to contact a Spark technician to work through why they have been blocked...

cyril7

9058 posts

Uber Geek

ID Verified

Trusted

Subscriber

#1125698 10-Sep-2014 12:25

Hi, just come from a school on Telecom Business Fibre that has been cut off, after an age on the line talking to folk with clearly limited knowledge I finally get told that the modem (sic) needs replacing as the current one has a virus. Obviously not happy to have to replace the Mikrotik I ask what it will be replaced with, then ask how I will terminate my VPN's on that. Obviously my immediate concern was that the Mikrotiks DNS server/cache was providing out of LAN scope DNS recursion, I have checked with others I have installed on other ISPs and they dont, clearly I cannot test this one as Telecom have blocked it.

Anyone know what options there are, it seems another two days of no internet for a school wanting to do online exams.

Cyril

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

martinjward

68 posts

Master Geek

#1125712 10-Sep-2014 12:52

cyril7: Hi, just come from a school on Telecom Business Fibre that has been cut off, after an age on the line talking to folk with clearly limited knowledge I finally get told that the modem (sic) needs replacing as the current one has a virus. Obviously not happy to have to replace the Mikrotik I ask what it will be replaced with, then ask how I will terminate my VPN's on that. Obviously my immediate concern was that the Mikrotiks DNS server/cache was providing out of LAN scope DNS recursion, I have checked with others I have installed on other ISPs and they dont, clearly I cannot test this one as Telecom have blocked it.

Anyone know what options there are, it seems another two days of no internet for a school wanting to do online exams.

Cyril

Hi Cryril, you've pretty much summed up what I have experienced over the past two days. I will PM you.

Talkiet

4793 posts

Uber Geek

Trusted

#1125715 10-Sep-2014 12:54

cyril7: Hi, just come from a school on Telecom Business Fibre that has been cut off, after an age on the line talking to folk with clearly limited knowledge I finally get told that the modem (sic) needs replacing as the current one has a virus. Obviously not happy to have to replace the Mikrotik I ask what it will be replaced with, then ask how I will terminate my VPN's on that. Obviously my immediate concern was that the Mikrotiks DNS server/cache was providing out of LAN scope DNS recursion, I have checked with others I have installed on other ISPs and they dont, clearly I cannot test this one as Telecom have blocked it.

Anyone know what options there are, it seems another two days of no internet for a school wanting to do online exams.

Cyril

PM me details.

When you say business fibre, please be specific about the product, including any CLNE supplied.

Regards
Neil G

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

pristle

158 posts

Master Geek

Subscriber

#1125751 10-Sep-2014 13:30

Well further to my post on Monday. I did visit the family on Monday. Installed a 'loan' TP-Link ADSL2+ router. Ran anti malware software across their PCs (which turned up a lot of junk). Once I was comfortable in the knowledge all was clear I tried contacting Spark to get their IP unblocked. Oh the joy.

My first call disconnected after 35 minutes. I figured this was going to be a mission, so instead of waiting at their house interrupting their family dinner time, I left. I began my next call in the car.

This "approximately 1 hour wait time" call got to ±115 minutes when I got to speak to somebody. I gave the requisite 'incident number' and asked that she pull it up and use it as reference. I explained what I had done and was now asking that the block be lifted. She then insisted that I go onto the net and see if I could browse to certain addresses. I explained that beside the fact I was no longer onsite the clients IP was being blocked by Spark and none of her online tests would be possible. She was quite insistent. I let rip and asked that I speak to the next tier, and was put on hold. After 2-3 minutes she came back on the line and said that she had now read through the notes relating to the incident number with her supervisor and could now understand what I trying to say.

At ±120 minutes I was told that the team that could remove the block had gone home about hour earlier. Nice. There's a major crisis on, and senior tech people get to go home? I've been doing this stuff for over 30 years, you never go home until it's fixed. Anyway I was then advised that it's been treated as a priority and the customer would have service between 9-10am Tuesday.

Nothing happened Tuesday morning bar the arrival of a shiny new Huawei HG630b. I advsed that they install it, which they did. A rep from Spark called me at around 14:00 to check if there was service yet, if not the I would be put through to the 'specialist' team. When that didn't happen, she advised someone would call be back in 20-30 minutes. No-one did.

Well Tuesday has been and gone and the customer is still waiting for service. I left a voice message for a call back this morning. Sent an email. And also engaged in a 'chat' session with the Broadband Team, first Sarah who upon reading of the customers dilemma, quickly put me through to Colin in Billing. Very helpful. Anyway Colin was quite positive, he took my details and assured me someone will be in touch within 10-15 minutes. That was around 10am, still waiting.

I eventually got a call back from someone at 12:04. They weren't very concise about the delays. Something about how 'some other stuff had to be unblocked before their IP could be released'. "Perhaps sometime in the next 2 to 3 hours".

Then 20 minutes ago the customer recieved the following txt: "From Spark Support: Your Broadband issue has been resolved for 095*****2. If you still experience issues please call 0800225598 using case #2312*******9"

It's still not functioning. She's had to go elsewhere to attend to her internet requirements of banking and account payment. Hopefully on her return there'll be some change.

I can't help think this whole issue could've been handled better. Blaming the non 'Telecom"/"Spark" routers is a bit of a cop out. If there was a vulnerablility in them, why was this not picked up in the Telepermit process?

Keef

20 posts

Geek

#1125772 10-Sep-2014 14:32

I have had to talk 8 sites through changing the DNS on their PC so far, last one was 2 minutes before writing this post. restarting the router wasn't effecting a solution.
fortunately we only have 1 PC per site and they are all win7 so it's follow the same steps, job done.
strange thing, spark is telling me the problem is fixed, waiting for the official response
all our routers are dlink couple different varieties.

cyril7

9058 posts

Uber Geek

ID Verified

Trusted

Subscriber

#1125794 10-Sep-2014 14:58

Just like to update that with the help of Spark staff who frequent here (thanks guys) the schools router was removed from the blacklist, it would seem that a machine in the school seems to be infested with malware that took part in an amplification attack, naturally I have requested the site admin take a look into that aspect.

Cheers
Cyril

pristle

158 posts

Master Geek

Subscriber

#1125797 10-Sep-2014 15:01

Nearly fell of my chair....3 minutes and I was speaking to somebody.

He assured me that progress is being made. But it will be another 3-4 hours before the client sees service. This clients connection didn't suddenly drop off on Saturday, but has been intermittent since early last week already. Spark, mad that disconnection effectively permanent over the weekend.

The client has already asked about other provider options.

insane

3240 posts

Uber Geek

ID Verified

Trusted

#1125813 10-Sep-2014 15:11

pristle:

The client has already asked about other provider options.

If the client is actively taking part in the cause of the issue, then perhaps they should be looking inwards before looking outwards? As far as I know every ISP has rights to kick users off who are affecting their service/core infrastructure.

Unless I've misunderstood what your issue is.

hio77

12999 posts

Uber Geek

ID Verified

Trusted

Lizard Networks

#1125816 10-Sep-2014 15:13

insane:
pristle:

The client has already asked about other provider options.

If the client is actively taking part in the cause of the issue, then perhaps they should be looking inwards before looking outwards? As far as I know every ISP has rights to kick users off who are affecting their service/core infrastructure.

Unless I've misunderstood what your issue is.

but it was working fine before! it cant have been their fault! isp must have made the mistake!

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

1 | ... | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21