Anyone experiencing problems on BigPipe 21-Oct-2015?

Forums › BigPipe › Anyone experiencing problems on BigPipe 21-Oct-2015?

1 | 2

BigPipeNZ

1142 posts

Uber Geek
+1 received by user: 748

Trusted

BigPipe

Reply # 1410827 22-Oct-2015 09:00

Hey everyone,

Picked this up last night and the network team are onto it.
Looks like there are (were) two simultaneous issue going on.

Something with our network affecting some customers, and also unblock-us DNS went down at the same time.

according to this link (https://support.unblock-us.com/customer/en/portal/articles/2168719-open-issues-with-australian-and-new-zealand-dns ) the unblock-us issue is now fixed so if anyone is still having issues after a modem restart and changing to Bigpipe DNS please get in contact with our support team if you haven't already with as much detail as you can about the issue (i.e. post tracerts etc and whatever you have already tried)

bigpipe.co.nz
https://www.facebook.com/BigPipeNZ
https://twitter.com/BigPipeNZ

timbosan

1423 posts

Uber Geek
+1 received by user: 137

Subscriber

Reply # 1410828 22-Oct-2015 09:02

BigPipeNZ: Hey everyone,

Picked this up last night and the network team are onto it.
Looks like there are (were) two simultaneous issue going on.

Something with our network affecting some customers, and also unblock-us DNS went down at the same time.

according to this link (https://support.unblock-us.com/customer/en/portal/articles/2168719-open-issues-with-australian-and-new-zealand-dns ) the unblock-us issue is now fixed so if anyone is still having issues after a modem restart and changing to Bigpipe DNS please get in contact with our support team if you haven't already with as much detail as you can about the issue (i.e. post tracerts etc and whatever you have already tried)

Thanks for the update! As I posted I cannot email so cannot reply to the ticket I raised. I am using BigPipe DNS and was having problems at 8am this morning.

BigPipeNZ

1142 posts

Uber Geek
+1 received by user: 748

Trusted

BigPipe

Reply # 1410829 22-Oct-2015 09:03

timbosan: If @BigPipe read this - I cannot reply to your emails about the service desk incident I raised, as my emails are not sending.

So:

1) I can read geekzone but not post
2) I can receive emails but not send
3) Most sites load the home page, but often get no further

This is the same as yesterday

are you unable to send emails from your phone too via 3G/4G? sounds like that could be unrelated if so.

If you DM me your email address I'll make sure the team are aware you cannot contact via email.

As an alternative, you can also contact us via Facebook PM if that is working for you. That goes direct to our support team just like an email.

cheers

bigpipe.co.nz
https://www.facebook.com/BigPipeNZ
https://twitter.com/BigPipeNZ

timbosan

1423 posts

Uber Geek
+1 received by user: 137

Subscriber

Reply # 1410831 22-Oct-2015 09:06

BigPipeNZ:
timbosan: If @BigPipe read this - I cannot reply to your emails about the service desk incident I raised, as my emails are not sending.

So:

1) I can read geekzone but not post
2) I can receive emails but not send
3) Most sites load the home page, but often get no further

This is the same as yesterday

are you unable to send emails from your phone too via 3G/4G? sounds like that could be unrelated if so.

If you DM me your email address I'll make sure the team are aware you cannot contact via email.

As an alternative, you can also contact us via Facebook PM if that is working for you. That goes direct to our support team just like an email.

cheers

I run an Exchange server at home, and any emails I try to send come up as "delayed delivery". There is no way for me to put the exchange server onto 3G. I will DM you

Shindig

617 posts

Ultimate Geek
+1 received by user: 45

Trusted

Reply # 1410838 22-Oct-2015 09:12

Noticed an issue as well..... came good though and a good excuse to reboot the router for once.

The little things make the biggest difference.

Rudster

88 posts

Master Geek
+1 received by user: 17

Reply # 1410862 22-Oct-2015 09:31

Unable to test again right now as in the city but apparently it is still the same story as last night.
This is supposed to be 200/200 fibre

cr250bromo

221 posts

Master Geek
+1 received by user: 81

Reply # 1410987 22-Oct-2015 11:22

Not sure if this is related, but my flatmate noticed very slow FTP uploads to his website, when I ssh'ed in from work my connection kept freezing.

It looked possibly like an MTU problem to me, as a stopgap measure I changed ip tcp adjust-mss 1452 to 1400 (cisco ios). Probably much too low but good enough to get it working again. When I get home I'll test further to confirm what the max is. But it seems to have sorted my flatmates FTP upload problem for now.

On 100/20 UFB in Grey Lynn.

BigPipeNZ

1142 posts

Uber Geek
+1 received by user: 748

Trusted

BigPipe

Reply # 1411172 22-Oct-2015 14:54

3 people support this post

Hey guys,

Thanks to everyone who has submitted tickets, It's all very helpful for our network team to help identify the issue. And apologies for the issues. Definitely not what we like to see.

It seems there were two, unrelated issues here (and having two issues at the same time makes diagnosis that much harder unfortunately)

1) unblock-us DNS went down. This cause ALL browsing when using unblock-US to fail. Nothing to do with Bigpipe, and should be resolved by now according to unblock-us. If you had problems with ALL browsing, and use unblock-us, check that first.

2) an unrelated issue that appears to be confined to a small number of UFB customers in Auckland (small enough that overall traffic trend looks totally normal - so probably less than 5% impacted).
(anyone else having issues who isn't on UFB or isn't in Auckland, please let us know via our support form - as it could be a separate issue)

Our network team have identified what they believe is the issue and will be putting a fix through Friday morning at 3-4am. (i.e. tonight)

This will cause all Auckland customers (including ADSL/VDSL) to briefly drop off the network and reconnect either once or twice. between 3.30-4.30. Most modems should reconnect automatically, but if they do not reconnect after about 10-20 mins (or if still disconnected when you wake up in the morning) then a reboot of the modem should sort it out.

We'll be sending out an email to those customers who may be affected, but thought it worth posting here too.

thanks again for the help. Geekzoners rock

bigpipe.co.nz
https://www.facebook.com/BigPipeNZ
https://twitter.com/BigPipeNZ

Wacky

55 posts

Master Geek
+1 received by user: 6

Reply # 1411176 22-Oct-2015 14:58

Thank you. Looking forward to this fix.

BigPipeNZ: Hey guys,

Thanks to everyone who has submitted tickets, It's all very helpful for our network team to help identify the issue. And apologies for the issues. Definitely not what we like to see.

It seems there were two, unrelated issues here (and having two issues at the same time makes diagnosis that much harder unfortunately)

1) unblock-us DNS went down. This cause ALL browsing when using unblock-US to fail. Nothing to do with Bigpipe, and should be resolved by now according to unblock-us. If you had problems with ALL browsing, and use unblock-us, check that first.

2) an unrelated issue that appears to be confined to a small number of UFB customers in Auckland (small enough that overall traffic trend looks totally normal - so probably less than 5% impacted).
(anyone else having issues who isn't on UFB or isn't in Auckland, please let us know via our support form - as it could be a separate issue)

Our network team have identified what they believe is the issue and will be putting a fix through Friday morning at 3-4am. (i.e. tonight)

This will cause all Auckland customers (including ADSL/VDSL) to briefly drop off the network and reconnect either once or twice. between 3.30-4.30. Most modems should reconnect automatically, but if they do not reconnect after about 10-20 mins (or if still disconnected when you wake up in the morning) then a reboot of the modem should sort it out.

We'll be sending out an email to those customers who may be affected, but thought it worth posting here too.

thanks again for the help. Geekzoners rock

graemeh

2075 posts

Uber Geek
+1 received by user: 228

Subscriber

Reply # 1411179 22-Oct-2015 15:12

4 people support this post

BigPipeNZ: We'll be sending out an email to those customers who may be affected, but thought it worth posting here too.

What a refreshing change to have an ISP notify its customers before an outage.

BigPipe are teaching the other ISPs how they should be operating.

Rudster

88 posts

Master Geek
+1 received by user: 17

Reply # 1411483 23-Oct-2015 07:36

Seems to be worse this morning.
Tracing route to speedtest.net [216.146.46.11]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms router.asus.com [192.168.1.1]
2 2 ms 2 ms 2 ms 210.54.39.129
3 * * * Request timed out.
4 2 ms 1 ms 3 ms g7-2-8-201.akbr6.global-gateway.net.nz [122.56.118.65]
5 2 ms 2 ms 2 ms ae5-2.akbr7.global-gateway.net.nz [210.55.202.213]
6 2 ms 2 ms 2 ms ae1-10.tkbr12.global-gateway.net.nz [202.50.232.37]
7 133 ms 133 ms 153 ms xe-0-0-7-3.sjbr3.global-gateway.net.nz [202.50.232.18]
8 166 ms 161 ms 161 ms ae0.pabr5.global-gateway.net.nz [203.96.120.74]
9 162 ms 162 ms 162 ms ix-8-2-5-0.tcore1.PDI-Palo-Alto.as6453.net [66.198.127.49]
10 133 ms 133 ms 133 ms if-3-2.tcore1.LVW-Los-Angeles.as6453.net [66.198.127.26]
11 163 ms 163 ms 163 ms 66.110.59.22
12 133 ms 132 ms 132 ms redirector2.dynect.net [216.146.46.11]

I know this is to a server in USA but it took speedtestnet 7 minutes to load.

Trace complete.

Seems any server outside of Auckland has low upload bandwidth and high latency. Traffic being directed down south?

timbosan

1423 posts

Uber Geek
+1 received by user: 137

Subscriber

Reply # 1411502 23-Oct-2015 08:39

All seems to be working for me. A quick test this morning:

Emails are sending
Sites like Trademe and live.com are working
Youtube works

Didn't even have to reboot my gateway! Thanks BigPipe!

BTW - any indication of what the fault was? I am assuming that with Chorus involved and the small number of affected users it was a hardware fault?

insane

2276 posts

Uber Geek
+1 received by user: 364

Trusted

Subscriber

Reply # 1411511 23-Oct-2015 09:13

I'm guessing the same problem which seems to affect every ISP... Handover congestion or incorrectly configured buffer sizes

BigPipeNZ

1142 posts

Uber Geek
+1 received by user: 748

Trusted

BigPipe

Reply # 1411535 23-Oct-2015 09:53

Rudster: Seems to be worse this morning.
Tracing route to speedtest.net [216.146.46.11]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms router.asus.com [192.168.1.1]
2 2 ms 2 ms 2 ms 210.54.39.129
3 * * * Request timed out.
4 2 ms 1 ms 3 ms g7-2-8-201.akbr6.global-gateway.net.nz [122.56.118.65]
5 2 ms 2 ms 2 ms ae5-2.akbr7.global-gateway.net.nz [210.55.202.213]
6 2 ms 2 ms 2 ms ae1-10.tkbr12.global-gateway.net.nz [202.50.232.37]
7 133 ms 133 ms 153 ms xe-0-0-7-3.sjbr3.global-gateway.net.nz [202.50.232.18]
8 166 ms 161 ms 161 ms ae0.pabr5.global-gateway.net.nz [203.96.120.74]
9 162 ms 162 ms 162 ms ix-8-2-5-0.tcore1.PDI-Palo-Alto.as6453.net [66.198.127.49]
10 133 ms 133 ms 133 ms if-3-2.tcore1.LVW-Los-Angeles.as6453.net [66.198.127.26]
11 163 ms 163 ms 163 ms 66.110.59.22
12 133 ms 132 ms 132 ms redirector2.dynect.net [216.146.46.11]

I know this is to a server in USA but it took speedtestnet 7 minutes to load.

Trace complete.

Seems any server outside of Auckland has low upload bandwidth and high latency. Traffic being directed down south?

all other reports so far seem to indicate the fix has worked.

Sounds like you are still having an issue though. Could I ask you to please let our support team know about this.
It could be the same issue, or something else totally unrelated. cheers

bigpipe.co.nz
https://www.facebook.com/BigPipeNZ
https://twitter.com/BigPipeNZ

BigPipeNZ

1142 posts

Uber Geek
+1 received by user: 748

Trusted

BigPipe

Reply # 1411536 23-Oct-2015 09:56

One person supports this post

timbosan: All seems to be working for me. A quick test this morning:

Emails are sending
Sites like Trademe and live.com are working
Youtube works

Didn't even have to reboot my gateway! Thanks BigPipe!

BTW - any indication of what the fault was? I am assuming that with Chorus involved and the small number of affected users it was a hardware fault?

without going into too many technical details (mainly because I don't understand them) it seems to be related to MTU. What's odd is that we haven't made any changes to our network in a few weeks, so it is really odd that this just started a couple of days ago.

I'm thinking it's probably something to do with Back to The Future. Seems likely that the flux capacitor on the delorean caused some interference with UFB when it arrived yesterday.

bigpipe.co.nz
https://www.facebook.com/BigPipeNZ
https://twitter.com/BigPipeNZ

1 | 2