Bigpipe - Help resolving ticket.

Forums › Spark New Zealand (including Skinny and BigPipe) › Bigpipe - Help resolving ticket.

1 | 2 | 3 | 4 | 5

1421 posts

Uber Geek
+1 received by user: 314
Inactive user

#2045721 28-Jun-2018 18:29

Linux:

@nunz @BigPipeNZ Please keep us updated once the issue is resolved so we actually know what the issue was

John

Hi @Linux

The connection has corrected itself. it seemed better (but did fail at least once) after it was 'reset' by BP. I've no idea what was wrong as they haven't replied to requests for info.

They blamed the modem as it showed < 0.1% (3 out of over 5000 pings in a 20 hour session ) missed on the internal network ping checks. Compared to 70% missing on pings to external network sites - i doubt it. Also as it is now fixed and I've changed nothing on the internal networks it makes no sense.

The internal ping was across a wireless network connection that then routes through two devices to the gateway. I wanted worst possible results if it was an internal issue. 3 pings missed out of 5000 on a busy wireless network with competing channels is acceptable. It could just as possible have been the machine that missed as the network as it was chewing through large data backups across the network at the time.

Still a mystery. thanks for your interest in it.

nunz

1421 posts

Uber Geek
+1 received by user: 314
Inactive user

#2047395 2-Jul-2018 12:15

nunz:

Linux:

@nunz @BigPipeNZ Please keep us updated once the issue is resolved so we actually know what the issue was

John

Hi @Linux

The connection has corrected itself. it seemed better (but did fail at least once) after it was 'reset' by BP. I've no idea what was wrong as they haven't replied to requests for info.

They blamed the modem as it showed < 0.1% (3 out of over 5000 pings in a 20 hour session ) missed on the internal network ping checks. Compared to 70% missing on pings to external network sites - i doubt it. Also as it is now fixed and I've changed nothing on the internal networks it makes no sense.

The internal ping was across a wireless network connection that then routes through two devices to the gateway. I wanted worst possible results if it was an internal issue. 3 pings missed out of 5000 on a busy wireless network with competing channels is acceptable. It could just as possible have been the machine that missed as the network as it was chewing through large data backups across the network at the time.

Still a mystery. thanks for your interest in it.

My bad - still ongoing. Drop outs - now - not complete failures.

Horrifc traceroutes outside of modem.

Last four all taken in 5 minute window of each other.

C:\Users\shaun>tracert 8.8.8.8

Tracing route to google-public-dns-a.google.com [8.8.8.8]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 5 ms 5 ms 5 ms 125-236-192-9.adsl.xtra.co.nz [125.236.192.9]
3 * * * Request timed out.
4 19 ms 19 ms 19 ms ae8-10.akbr6.global-gateway.net.nz [122.56.116.5
]
5 24 ms 24 ms 25 ms ae7-2.akbr7.global-gateway.net.nz [122.56.119.53
]
6 45 ms 45 ms 42 ms xe7-0-2.sgbr3.global-gateway.net.nz [202.50.232.
10]
7 43 ms 43 ms 43 ms ae2-10.sgbr4.global-gateway.net.nz [202.50.232.2
46]
8 43 ms 42 ms 42 ms 72.14.217.100
9 44 ms 44 ms 44 ms 108.170.247.81
10 43 ms 43 ms 43 ms 66.249.94.239
11 42 ms 42 ms 42 ms google-public-dns-a.google.com [8.8.8.8]

Trace complete.

C:\Users\shaun>

C:\Users\shaun>tracert mail.digiweb.net.nz

Tracing route to mail.digiweb.net.nz [202.174.115.225]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 * 2626 ms 1903 ms 125-236-192-9.adsl.xtra.co.nz [125.236.192.9]
3 * 21 ms 21 ms mdr-ip24-dom.msc.global-gateway.net.nz [122.56.1
16.10]
4 19 ms 19 ms 19 ms ae8-20.akcr11.global-gateway.net.nz [122.56.116.
9]
5 19 ms 19 ms 19 ms ae10-44.tkcr5.global-gateway.net.nz [122.56.127.
210]
6 44 ms 44 ms 44 ms vocus-dom.tkcr5.global-gateway.net.nz [122.56.11
8.114]
7 43 ms 43 ms 43 ms bundle-11.cor01.alb01.akl.vocus.net.nz [114.31.2
02.48]
8 46 ms 43 ms 43 ms ten-2-2-0.bdr01.akl05.akl.VOCUS.net.nz [114.31.2
02.43]
9 56 ms 55 ms 56 ms ip-67.87.45.175.VOCUS.net.au [175.45.87.67]
10 56 ms 56 ms 58 ms chc1-brj-01.ublr.net [202.174.114.37]
11 60 ms 66 ms 66 ms 202.174.119.1
12 60 ms 61 ms 62 ms mail.digiweb.net.nz [202.174.115.225]

Trace complete.

C:\Users\shaun>

C:\Users\shaun>tracert mail.digiweb.net.nz

Tracing route to mail.digiweb.net.nz [202.174.115.225]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 1374 ms 1100 ms 896 ms 125-236-192-9.adsl.xtra.co.nz [125.236.192.9]
3 * * * Request timed out.
4 19 ms 19 ms 19 ms ae8-20.akcr11.global-gateway.net.nz [122.56.116.
9]
5 19 ms 19 ms 19 ms ae10-44.tkcr5.global-gateway.net.nz [122.56.127.
210]
6 45 ms 44 ms 44 ms vocus-dom.tkcr5.global-gateway.net.nz [122.56.11
8.114]
7 43 ms 43 ms 43 ms bundle-11.cor01.alb01.akl.vocus.net.nz [114.31.2
02.48]
8 43 ms 43 ms 43 ms ten-2-2-0.bdr01.akl05.akl.VOCUS.net.nz [114.31.2
02.43]
9 56 ms 55 ms 55 ms ip-67.87.45.175.VOCUS.net.au [175.45.87.67]
10 56 ms 56 ms 55 ms chc1-brj-01.ublr.net [202.174.114.37]
11 57 ms 57 ms 56 ms 202.174.119.1
12 58 ms 57 ms 57 ms mail.digiweb.net.nz [202.174.115.225]

C:\Users\shaun>tracert mail.digiweb.net.nz

Tracing route to mail.digiweb.net.nz [202.174.115.225]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 * * * Request timed out.
3 * * * Request timed out.
4 19 ms 18 ms 19 ms ae8-20.akcr11.global-gateway.net.nz [122.56.116.
9]
5 20 ms 19 ms 19 ms ae10-44.tkcr5.global-gateway.net.nz [122.56.127.
210]
6 44 ms 44 ms 44 ms vocus-dom.tkcr5.global-gateway.net.nz [122.56.11
8.114]
7 43 ms 43 ms 43 ms bundle-11.cor01.alb01.akl.vocus.net.nz [114.31.2
02.48]
8 44 ms 43 ms 43 ms ten-2-2-0.bdr01.akl05.akl.VOCUS.net.nz [114.31.2
02.43]
9 56 ms 56 ms 56 ms ip-67.87.45.175.VOCUS.net.au [175.45.87.67]

nunz

1421 posts

Uber Geek
+1 received by user: 314
Inactive user

#2047402 2-Jul-2018 12:23

BP support telling me that the issue is my modem.

Wont answer my question : How does changing my modem fix an issue outside of my network? (second step of tracert is outside the modem surely).

hio77

'That VDSL Cat'
13036 posts

Uber Geek
+1 received by user: 3896

ID Verified

Trusted

Lizard Networks

Subscriber

#2047404 2-Jul-2018 12:24

I'm actually seeing no issue with those traces?

we deprioritize ICMP to our borders. What your seeing is exactly that, past the first two hops the connection is showing near perfect. Seeing about sub 1ms jitter there?

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

nunz

1421 posts

Uber Geek
+1 received by user: 314
Inactive user

#2047408 2-Jul-2018 12:30

Seriously - you consider 2600 plus MS acceptable? I have networks across wireless over 60km long that eat those stats for breakfast. Satellite is faster. That's a bollocks response from their server.

hio77

'That VDSL Cat'
13036 posts

Uber Geek
+1 received by user: 3896

ID Verified

Trusted

Lizard Networks

Subscriber

#2047413 2-Jul-2018 12:35

nunz:

Seriously - you consider 2600 plus MS acceptable? I have networks across wireless over 60km long that eat those stats for breakfast. Satellite is faster. That's a bollocks response from their server.

You are not quite understanding.

There is nothing wrong with the network, this is a typical configuration.

If you look at the hops following it, the network itself isn't delayed.

If a issue with the network was legitimately causing 2000+ms delays in packets, you would see this all the way through, not just on that single line.

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

AliExpress

Shop now on AliExpress (affiliate link).

nunz

1421 posts

Uber Geek
+1 received by user: 314
Inactive user

#2047423 2-Jul-2018 12:38

hio77:

I'm actually seeing no issue with those traces?

we deprioritize ICMP to our borders. What your seeing is exactly that, past the first two hops the connection is showing near perfect. Seeing about sub 1ms jitter there?

@hio77 I am being told to get a new modem. What in the TR's above (or that fact the total disconnections for hours is now sparodic interupptions to packet data ), would indicate the need for a new modem?

The only issue I can see is the shocking first step outside the modem. TR is a one of the most basic of tools and is absolutely vital to ensure we can figure out issues. Ping is one of the most basic uptime checks people run. De-priotising those leads to false warnings for uptime monitors, and makes me wonder what else is depriotised and not working - leading to the issues we face.

If I got stats like that from a server I would look at scrapping it as it was failing.

Linux

12188 posts

Uber Geek
+1 received by user: 8480

Trusted

Lifetime subscriber

#2047428 2-Jul-2018 12:47

@nunz I would be checking out the connections where the modem plugs into and other joins

John

hio77

'That VDSL Cat'
13036 posts

Uber Geek
+1 received by user: 3896

ID Verified

Trusted

Lizard Networks

Subscriber

#2047429 2-Jul-2018 12:49

nunz:

hio77 I am being told to get a new modem. What in the TR's above (or that fact the total disconnections for hours is now sparodic interupptions to packet data ), would indicate the need for a new modem?

The only issue I can see is the shocking first step outside the modem. TR is a one of the most basic of tools and is absolutely vital to ensure we can figure out issues. Ping is one of the most basic uptime checks people run. De-priotising those leads to false warnings for uptime monitors, and makes me wonder what else is depriotised and not working - leading to the issues we face.

OK. so your actual issue is, your being advised to replace the modem.

With these disconnections you are experiencing is there any loss of DSL?

Sounds to me like there is not. which would make a modem being a potential likely cause.

If we got with the assumption it's not a modem fault we then get to the next stages..

Is there spikes of upstream bandwidth? particularly if you have a low upstream connection, this will cause issues that feel like micro studding. delays in packets etc.

There are many valid reasons for deprotization of ICMP, just as there are for no responding at all.

I'm not going to get into that argument any further, as you indicate you administrate networks i'd expect you should know the pros and cons here already. If not, I'd recommend some research :)

How it looks on tools such as pingplotter is a unfortunate side-affect however, even pingplotter supports alternative methods such as unix style (udp) pings that will preform differently.

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

cbrpilot

964 posts

Ultimate Geek
+1 received by user: 555

Trusted

Spark NZ

#2047442 2-Jul-2018 13:07

Hi nunz,

The de-prioritisation of ICMP only relates to ICMP response from our router itself - i.e only the packets that elicit a response from our router itself. If it is transiting ICMP then no impact. This is a standard router protection mechanism.

My views are my own, and may not necessarily represent those of my employer.

nunz

1421 posts

Uber Geek
+1 received by user: 314
Inactive user

#2051061 7-Jul-2018 12:40

Linux:

@nunz I would be checking out the connections where the modem plugs into and other joins

John

The modem lives in a locked cabinet that has not been accessed in months (until I turned it off and on the other day to reset the connection).

There is Zero chance of any movement between cables and joins - so if there was an issue I would expect it to be consistent. I've eyeballed lights on routers, switches, plugs etc - and there is nothing amiss. most of it is hard wired into wires that go into the ceiling via a patch panel or a direct cable back to the demarcation outside the building.

When the initial trouble started we were told they could see DSL but not authentication and were told it was our modem settings. We were told to scrap all the settings, reset it up again (using exactly the same settings) and move forward. We did none of the things in this list and it fixed itself.

With zero changes at my end and lots of changes seen at BPs radius or what ever other system they use for authentication, then I would expect the issue to be their end.If we have DSL but cant authenticate - then packets wont traverse their systems and our internet wont work. That's not a physical layer one issue but further up the stack and less likely to be modem caused. It is also very unlikely to self correct - so when they reset THEIR end of things and it got better - then I'm less likely to look at the modem as a cause.

Similarly they told us we had to change settings as they were dickering around with IPV6 - which we don't use. It was again a piece of useless advice off a list of useless advice that didn't fit our experience. Their support person was giving us their polite version of Faff off and stop bothering us as we cant be bothered paying attention to your issue. It is a typical lack of attention and I don't care attitude that is seen so regularly in 'support'(sic) systems.

All this leads me to believe the fault lays at their end. The modem has been replaced previously with similar issues experienced and no resolution found. BP have had to be pushed hard to do basic checks, have failed to return information on those checks and when they changed something (reset the connection at their end) the connection improved.

To replace a modem is over $300 (wholesale) in physical hardware charges along with 2 - 4 hours work resetting it back up with ports, VPN, certificates , static IPS, etc etc etc. Dont even ask about the routing from other connected routers, peering for wireless if required and the calls from users whose wireless doesn't work all of a sudden. $800 - $1000 in costs to do that is a not unreasonable figure.

All I expect is someone half way competent to actually pay some attention to the issue at hand and get back to us as promised. I also don't expect to be put through a list of useless advice where the support person has zero idea what they are doing and are throwing make work at us hoping to get lucky with a fix. IT is more scientific than that.

It is noticeable that the system we ran from end to end (end point to gateways to radius / dhcp / authneticated routers) until it backboned into the wholesale internet had almost zero issues like this. It's when we deal with the 'professional' isps tha we suddenly have issues.

HP

Shop now for HP laptops and other devices (affiliate link).

Linux

12188 posts

Uber Geek
+1 received by user: 8480

Trusted

Lifetime subscriber

#2051065 7-Jul-2018 12:47

How do you know moisture has not got into one of the connections somewhere like at the ETP?

John

hio77

'That VDSL Cat'
13036 posts

Uber Geek
+1 received by user: 3896

ID Verified

Trusted

Lizard Networks

Subscriber

#2051066 7-Jul-2018 12:50

nunz:

To replace a modem is over $300 (wholesale) in physical hardware charges along with 2 - 4 hours work resetting it back up with ports, VPN, certificates , static IPS, etc etc etc. Dont even ask about the routing from other connected routers, peering for wireless if required and the calls from users whose wireless doesn't work all of a sudden. $800 - $1000 in costs to do that is a not unreasonable figure.

All I expect is someone half way competent to actually pay some attention to the issue at hand and get back to us as promised. I also don't expect to be put through a list of useless advice where the support person has zero idea what they are doing and are throwing make work at us hoping to get lucky with a fix. IT is more scientific than that.

It is noticeable that the system we ran from end to end (end point to gateways to radius / dhcp / authneticated routers) until it backboned into the wholesale internet had almost zero issues like this. It's when we deal with the 'professional' isps tha we suddenly have issues.

fun fact... devices CAN fail, not to say they have in this case. but typically when they do fail it's a slow fail - things randomly just stop working as they should.

as a competent IT person, i would expect you have a spare modem handy.

you plugin the spare modem, see how things go. If another modem shows the same symptoms, then it's likely external.

It's very easy to work with someone and go hey, fair enough your not detecting any drops on the hardware side, I'll change devices and can you run a 24 hour test to cross reference with while we do it?)

The fact that you instead opt to rant on a public forum about incompetency but can't follow this basic troubleshooting step is rather arrogant.

Based on your description around users with connected routers, have you considered internally there is something a miss?

a good one i've seen is there is a particular modem out there, that while works solidly it also happens to bridge PPPoE sessions by default.

When a user goes oh, i'll fix this. i'll setup a broadband connection on their PC and accidentally created a PPPoE dialer, this caused disconnections.

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

Linux

12188 posts

Uber Geek
+1 received by user: 8480

Trusted

Lifetime subscriber

#2051072 7-Jul-2018 13:04

If the fault was on the @BigPipeNZ side I would expect to see far more customers complaining

John

robjg63

4161 posts

Uber Geek
+1 received by user: 1427

Subscriber

#2051075 7-Jul-2018 13:13

hio77:

fun fact... devices CAN fail, not to say they have in this case. but typically when they do fail it's a slow fail - things randomly just stop working as they should.

you plugin the spare modem, see how things go. If another modem shows the same symptoms, then it's likely external.

Yep - that's how they expire - slowly and erratically. Have seen this quite a few times - would be much easier if they just died - but that would be too easy.

Nothing is impossible for the man who doesn't have to do it himself - A. H. Weiler

1 | 2 | 3 | 4 | 5