Spark DNS Issues - Amazing - Broadband Service Alert

Forums › Spark New Zealand (including Skinny and BigPipe) › Spark DNS Issues - Amazing - Broadband Service Alert

1 | ... | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | ... | 21

197 posts

Master Geek
+1 received by user: 9

#1124564 8-Sep-2014 17:29

Talkiet:
pohutukawa: snip

I don't know whether it's deliberate, but your questions are coming across as pretty aggressive and inflammatory.

I can promise you, Spark is doing a hell of a lot to improve the response to any future attacks of this nature. I can also promise you that there's no way we're going to tell random people on message boards exactly what we are doing from a resiliency and security design point of view.

Cheers - N

That's interesting, as I was thinking that all the Spark comments here were pretty aggressive towards anyone remotely critical of Spark, both in tone, and numbers of Spark employees here pulling rank.

What exactly is "aggressive" and "inflammatory". Please be specific so I know what you're talking about.

My guess is that you will reply saying that you don't have time, don't want to get into a slinging match, etc.

Let me be clear: I am not attacking anyone personally, but I *am* questioning the picture that has been painted by Spark to date. It's too easy to say people have been sucked in by malware pretending to be porn, oh those silly end-users.

Back to the issue...my questions are simply: 1. If Spark is going a hell of a lot more to improve the situation in the future, why wasn't this not done previously? The DNS DDoS amplification vector is not a new phenomenon. And it's not just a fibre/VDSL thing. and 2. Were Spark modems amongst the group of modems you previously mentioned?

Thanks for your up-front responses.

pohutukawa ... turning with the seasons ...

gished

276 posts

Ultimate Geek
+1 received by user: 10

#1124586 8-Sep-2014 17:45

Actually scratch what I said before, it's still broken. I re-enabled the auto dns settings on my router when I got home from work and still a no go. Rebooted that, tried again, no go. Manually entered in ns1 and ns2 still no go.

After speaking with the online helpdesk their suggestion was to leave the google dns settings till later in the week and then call the broadband helpdesk. It's functional but not ideal.

pohutukawa

197 posts

Master Geek
+1 received by user: 9

#1124587 8-Sep-2014 17:47

gished: Actually scratch what I said before, it's still broken. I re-enabled the auto dns settings on my router when I got home from work and still a no go. Rebooted that, tried again, no go. Manually entered in ns1 and ns2 still no go.

What happens if you manually update the DNS primary and secondary on your laptop/box to 122.56.237.1 and 210.55.111.1 (and reboot)?

You're not entering ns1. ... etc. in your DNS are you? (sorry, just had to ask!)

pohutukawa ... turning with the seasons ...

cbrpilot

964 posts

Ultimate Geek
+1 received by user: 555

Trusted

Spark NZ

#1124588 8-Sep-2014 17:50

gished: Actually scratch what I said before, it's still broken. I re-enabled the auto dns settings on my router when I got home from work and still a no go. Rebooted that, tried again, no go. Manually entered in ns1 and ns2 still no go.

What type of router are you using please?

My views are my own, and may not necessarily represent those of my employer.

gished

276 posts

Ultimate Geek
+1 received by user: 10

#1124595 8-Sep-2014 17:58

pohutukawa:
gished: Actually scratch what I said before, it's still broken. I re-enabled the auto dns settings on my router when I got home from work and still a no go. Rebooted that, tried again, no go. Manually entered in ns1 and ns2 still no go.

What happens if you manually update the DNS primary and secondary on your laptop/box to 122.56.237.1 and 210.55.111.1 (and reboot)?

You're not entering ns1. ... etc. in your DNS are you? (sorry, just had to ask!)

Ha, no I'm not entering ns1....

Manually entering it in doesn't make any difference. I spoke to the online helpdesk who have said leave the google settings for a few more days! uh, ok...

Router is a NetComm NB6plus4WN (f/w version 3.104k)

pohutukawa

197 posts

Master Geek
+1 received by user: 9

#1124596 8-Sep-2014 18:01

gished:
pohutukawa:
gished: Actually scratch what I said before, it's still broken. I re-enabled the auto dns settings on my router when I got home from work and still a no go. Rebooted that, tried again, no go. Manually entered in ns1 and ns2 still no go.

What happens if you manually update the DNS primary and secondary on your laptop/box to 122.56.237.1 and 210.55.111.1 (and reboot)?

You're not entering ns1. ... etc. in your DNS are you? (sorry, just had to ask!)

Ha, no I'm not entering ns1....

Manually entering it in doesn't make any difference. I spoke to the online helpdesk who have said leave the google settings for a few more days! uh, ok...

Router is a NetComm 11n (f/w version 3.104k)

Even if you enter it on your computer?

pohutukawa ... turning with the seasons ...

Dell

Shop now for Dell laptops and other devices (affiliate link).

cbrpilot

964 posts

Ultimate Geek
+1 received by user: 555

Trusted

Spark NZ

#1124597 8-Sep-2014 18:04

11n doesn't sound like the full model name...
Is there a model number such as NB6Plus4WN:

E.g. http://www.netcommwireless.com/product/adsl/nb6plus4wn

My views are my own, and may not necessarily represent those of my employer.

jeffnz

2870 posts

Uber Geek
+1 received by user: 666

Trusted

Lifetime subscriber

#1124609 8-Sep-2014 18:09

just wanted to say thanks to the guys on here helping fix the problem and also keeping us updated as best you can, also understand there is no requirement for you to even comment on here.

Personally if was a minor inconvenience, annoying but not enough to make me want to change ISP's as I've had great service from Spark and its employee's in the past, mush the same as I think other ISP's on here do. I find it sad and frustrating that some keep pushing the arguments to the point it seems almost a personal crusade or soapbox when a less aggressive approach would probably achieve more but such are the forums I guess.

keep up all the good work you guys we don't all think you are required to make excuses for the companies you work for.

Galaxy S10

Garmin Fenix 5

gished

276 posts

Ultimate Geek
+1 received by user: 10

#1124612 8-Sep-2014 18:13

cbrpilot: 11n doesn't sound like the full model name...
Is there a model number such as NB6Plus4WN:

E.g. http://www.netcommwireless.com/product/adsl/nb6plus4wn

Yeah thought that didn't quite look right on the config page.

Interesting. Adding the DNS to the OS works fine. Doesn't work if its pointing to the router. Ok, now I'm confused.

gished

276 posts

Ultimate Geek
+1 received by user: 10

#1124613 8-Sep-2014 18:14

jeffnz: just wanted to say thanks to the guys on here helping fix the problem and also keeping us updated as best you can, also understand there is no requirement for you to even comment on here.

Personally if was a minor inconvenience, annoying but not enough to make me want to change ISP's as I've had great service from Spark and its employee's in the past, mush the same as I think other ISP's on here do. I find it sad and frustrating that some keep pushing the arguments to the point it seems almost a personal crusade or soapbox when a less aggressive approach would probably achieve more but such are the forums I guess.

keep up all the good work you guys we don't all think you are required to make excuses for the companies you work for.

Second that!

cbrpilot

964 posts

Ultimate Geek
+1 received by user: 555

Trusted

Spark NZ

#1124614 8-Sep-2014 18:16

gished, if you can get back to me on the modem model, we may be able to progess with you a more permanent fix for the issues you are having. Feel free to PM me if you'd like to take this offline.
Alternatively if you can't find the model number, if you PM me your phone number we can see what your modem is identifying itself as from our end.

My views are my own, and may not necessarily represent those of my employer.

Geekzone

Want to support Geekzone and browse the site without the ads? Subscribe to Geekzone now (monthly, annual and lifetime options).

Tel69

Tel69
261 posts

Ultimate Geek
+1 received by user: 4

Trusted

Lifetime subscriber

#1124622 8-Sep-2014 18:37

Tel69: The modem at the place I'm staying at is one of the old Technicolor TG582n's
Default settings, no password for the Administrator account and this.
{Administrator}=>dns server config
domain   : telecom
timeout : 15s
suppress : 0
state    : enabled
trace    : disabled
syslog   : disabled
spoofing : enabled
spoof ip : 198.18.1.0

Ummm, the modems running it's own DNS server with spoofing enabled (Oh and no logging)??

Am I reading that right?
(Also is that DNS server running an Open Recursive DNS, http://support.zen.co.uk/kb/Knowledgebase/Broadband-Securing-your-Technicolor-TG582n-against-Open-Recursive-DNS indicates it might be)

OK so turning off the DNS server on the modem kills internet on a wireless level at least (I can't currently check wired, the modem is out in the garage, what a stupid place to put it).
I stopped the spoofing and things seem to have sped up quite a bit.

Don't want to play with it majorly however, it's not my internet connection or modem.

wasabi2k

2102 posts

Uber Geek
+1 received by user: 860

#1124627 8-Sep-2014 18:50

michaelmurfy:
wasabi2k: Post on reddit, linked to an article claiming this was all due to people looking at the nude celeb leaks - wut?

God the articles are terrible:
http://www.stuff.co.nz/business/industries/10468669/Naked-pics-link-to-internet-problems - just what? How are the two issues related?
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11320100 - "A frenzy over fake or leaked nude celebrity photos possibly sparked this weekend's disastrous internet meltdown." - No, it didn't

Good lord - how can such tripe be published?

It was people getting Malware by searching up such things that sparked a DDoS on Sparks DNS servers, so is correct.

not so much - http://www.theregister.co.uk/2014/09/08/nude_celeb_pics_wrongly_blamed_for_ddos_at_new_zealands_largest_isp/

freitasm

BDFL - Memuneh
80661 posts

Uber Geek
+1 received by user: 41083

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#1124628 8-Sep-2014 18:51

I've just noticed something and I'd like to know from Talkiet (or someone at Telecom) if this DNS work could be related? Or is this the description of the fix?

Latest Update:

Technical remediation activity was completed at 19:00 and customer browsing experience has returned to BAU levels. Customer facing teams confirm current contact volumes are in-line with expectations. Next update post 22:00 audio.

Location:

Nationwide

Event Description:

Degradation of Internet connectivity

Customer & Business Impact:

Mobile and Broadband customers are experiencing intermittent loss of browsing and email services.

Home, Mobile & Business Contact centre volumes are returning to BAU levels.

Detail at 20:30
- 36 calls waiting, longest wait 8 minutes

Start Date & Time:

05/09/2014 @ 20:00

Resolution Date & Time:

Duration:

Application(s) Impacted:

DNS - Mobile Data & Broadband Browsing

Summary of Technical Fix:

Technical remedial activity to date:

05/09/2014 PM (night)
- Blacklisted two domain addresses (URL's) that were identified as causing issues.

06/09/2014
- At c. 12:00 – DNS Servers switched to BIND application to increase server capacity.
- Between 16:00 & 19:00 – Blocked inbound DNS UDP Port 53 to all BNG's (18) resulting in a 100 MBPS traffic reduction.
- Blacklisted additional three domain addresses (URL's) that were identified as causing issues. Total of five domain addresses have been blacklisted.
- DNS health checks on servers and network links show performance is now in line with normal expectations.
- Work stream underway to check design elements.

Activities planned for overnight:
- GGI continuing to identify and block offending source IP address/es if identified
- Web Apps – TWAS continuing to monitor and manually add domains to the blocked list if identified

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

surfisup1000

5288 posts

Uber Geek
+1 received by user: 2159

#1124631 8-Sep-2014 18:55

My mother just phoned to say her internet on the ipad is not working since the weekend. For some reason, her ipad skype was working though.

Today she called spark and they did various tests and said her connection is just fine.

Then, she called apple and they couldn't help either.

So, I'm the last on the list so she called me now. I heard something about DNS problems with telecom but i didn't think it was that because news is that it is fixed and telecom said her connection was just fine.

But, i told her to change the DNS on the ipad to be google, (8.8.8.8) instead of the router (192.168.1.1).

Woohoo, it worked.

but, i don't get this. Telecom and apple both say it is not their problem -- i don't really want her to use the google dns server as i read somewhere that it was slower than telecoms.

Or, is there no choice? Confused.

1 | ... | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | ... | 21