Bigpipe - Help resolving ticket.

Forums › Spark New Zealand (including Skinny and BigPipe) › Bigpipe - Help resolving ticket.

1 | 2 | 3 | 4 | 5

1421 posts

Uber Geek
Inactive user

#2051080 7-Jul-2018 13:25

hio77:

nunz:

hio77 I am being told to get a new modem. What in the TR's above (or that fact the total disconnections for hours is now sparodic interupptions to packet data ), would indicate the need for a new modem?

The only issue I can see is the shocking first step outside the modem. TR is a one of the most basic of tools and is absolutely vital to ensure we can figure out issues. Ping is one of the most basic uptime checks people run. De-priotising those leads to false warnings for uptime monitors, and makes me wonder what else is depriotised and not working - leading to the issues we face.

OK. so your actual issue is, your being advised to replace the modem.

With these disconnections you are experiencing is there any loss of DSL?

Sounds to me like there is not. which would make a modem being a potential likely cause.

If we got with the assumption it's not a modem fault we then get to the next stages..

Is there spikes of upstream bandwidth? particularly if you have a low upstream connection, this will cause issues that feel like micro studding. delays in packets etc.

There are many valid reasons for deprotization of ICMP, just as there are for no responding at all.

I'm not going to get into that argument any further, as you indicate you administrate networks i'd expect you should know the pros and cons here already. If not, I'd recommend some research :)

How it looks on tools such as pingplotter is a unfortunate side-affect however, even pingplotter supports alternative methods such as unix style (udp) pings that will preform differently.

Most data is downsteam (imap / activesync) to PCs. The VDSL reports great connection speeds when it is running. We aren't on the end of a line by any means. i think our cabinet is 1 - 1.5 km away. We do know that cabinet / area network has given other bigger organizations in the area horrific issues. People closer than us have struggled to make 5Mbps and they are on enterprise level support and have been screwed around for months. They have had outages and and issues for quite a while.

Look at the image below - taken today from the crappy old PC on the long end of the network and via two switches / routers. That's awesome VDSL connectivity. No changes at our end, 1ms jitter.

We set up iperf server at both ends and ran connections - it works great - when it works. 0ms to 20ms between two BP connected systems when it works. When it doesn't there is no response.

--re icmp

I'm old school - started on the internet when all there was for people like me was a single phone line out of Whangarei- and it took 24 hours to get an email to USA and back - as all connections were dial up modem to model / peer to peer via Telix and some of the NZ BBS systems. Tracking failures meant the old tools had to work. Expecting some of the sockets connects like today didn't work.

As it progressed to dial up modems via ISPs then we used ping, telnet etc to track issues. Wrapping monitoring in TCP / IP means that there are lower level checks that don't happen. all socket / UDP / TCP packets do is tell you if it gets there or not. ICMP allows more detailed responses - it is much more informational.

People like google, gateway-nz and other servers respond to icmp traffic - and they are much more vulnerable to smurf etc. Why allow? They know the value of diagnostics.

---

so far here is what I know for sure:

BP claim we weren't authenticating and had to change all our settings. We didn't but when they reset the connection at their end we got connectivity back - for a time.

We did have DSL.

We have had no information on the results of their checks except when I phoned them back and queried why no news. The reason was it didn't fail for 24 hours. nothing more detailed than that. We have had no detailed info from BP re the issues at the time it was bad and no tests or support to track the cause. As such it is a mystery and all we can do is take wild guesses as we cannot do much in the way of diagnositcs if BP wont engage at a meaningful level.

We cant even find out if the modem tried to authenticate when there were issues. No one grepped logs or bothered to to do more than throw wild guess ideas that will cost a bomb at us.

The only things that have helped are things BP did to their network / at the exchange. I'm still guessing it is their issue.

nunz

1421 posts

Uber Geek
Inactive user

#2051096 7-Jul-2018 14:17

hio77:

nunz:

To replace a modem is over $300 (wholesale) in physical hardware charges along with 2 - 4 hours work resetting it back up with ports, VPN, certificates , static IPS, etc etc etc. Dont even ask about the routing from other connected routers, peering for wireless if required and the calls from users whose wireless doesn't work all of a sudden. $800 - $1000 in costs to do that is a not unreasonable figure.

All I expect is someone half way competent to actually pay some attention to the issue at hand and get back to us as promised. I also don't expect to be put through a list of useless advice where the support person has zero idea what they are doing and are throwing make work at us hoping to get lucky with a fix. IT is more scientific than that.

It is noticeable that the system we ran from end to end (end point to gateways to radius / dhcp / authneticated routers) until it backboned into the wholesale internet had almost zero issues like this. It's when we deal with the 'professional' isps tha we suddenly have issues.

fun fact... devices CAN fail, not to say they have in this case. but typically when they do fail it's a slow fail - things randomly just stop working as they should.

as a competent IT person, i would expect you have a spare modem handy.

you plugin the spare modem, see how things go. If another modem shows the same symptoms, then it's likely external.

It's very easy to work with someone and go hey, fair enough your not detecting any drops on the hardware side, I'll change devices and can you run a 24 hour test to cross reference with while we do it?)

The fact that you instead opt to rant on a public forum about incompetency but can't follow this basic troubleshooting step is rather arrogant.

Based on your description around users with connected routers, have you considered internally there is something a miss?

a good one i've seen is there is a particular modem out there, that while works solidly it also happens to bridge PPPoE sessions by default.

When a user goes oh, i'll fix this. i'll setup a broadband connection on their PC and accidentally created a PPPoE dialer, this caused disconnections.

> as a competent IT person, i would expect you have a spare modem handy.

and I do - but I also have to justify why I've disrupted users and forced them to re setup connections then un-disrupt them and reset them up to old settings, then charge for my time.

Also your assumption is that BP would actually do something competent like look at logs - which they flat out failed to do. It is a non starter as a fix unless BP are prepared to actually do something.

Also it didn't start as a rant but the more I spend time dealing with crap like the above the more it is likely to go that way.

How is getting peeved about not getting through to support and asking for help a rant?

But onwards with your comment ...

On my first call to BP the person was so incompetent that I wouldn't follow what i was told. They were making wild guesses not following any process. when I talked to someone who said we had DSL but no authentication they wouldn't answer if that was because we either:

1 - Weren't making authentication requests (our fault if this was the case)

2 - were starting the process but failing (would need more diagnosis)

3 - Started the process and gave bad user name / passwords (our fault if this was the case)

4 - Did all the above correctly but still failed. (BPS fault.)

That's basic information and when the techs wont grep logs to give basic diagnositc info but keep giving suggestions that are plain stupid (reset router and use these settings. Hey those settings are what we are already using. Do it any way. ) then following their advice doesn't make sense. They threw three wild guesses at us and gave no information helpful to diagnosing this issue.

They don't listen to what they are told. E.g. Move the router to another jack. There is no jack, its hard wired into a cable from the Demarc. No jacks in the building.

Then kept telling us to try other jacks. Telling us jacks wirings go wiring etc. Obviously not listening.

To get things working i bridged the router to another persons wireless / internet connection so we had a wireless connection in the middle. We had no issues (other than less performance). Again - that seems to take the router out of the issue. - other than DSL nic failing.

Again resetting at the exchange / ISP improved things. Again that points away from the modem.

My competence tells me that all things point away from the router and towards something external. My history of the area tells me there are some horrible connections and issues around and possibly we were hitting them.

If it is arrogant to write off a dumb ass list spouting person who wont listen then I am arrogant. if it is arrogant to feel it is reasonable to exclude certain things based on tests we have run vs following wild guesses and assumptions from an ignorant (e.g. not bothering to check logs) tech then yes i am arrogant.

If it is arrogant to not cost my client hundreds of dollars on a wild guess from a tech who isn't listening then yes i am arrogant.

We provided information from over 5000 tests showing our internal network was rock solid. The same tests showed huge issues as soon as we tried to exit through the gateway.

Is it arrogant to look at that and go - well the modem seems to be in good shape from a physical layer level - it is routing signals internally beautifully. Could it be the NIC that does DSL causing issues? Maybe but performance wise at every other level it seems fine.

When a modem starts to fail - yes it can be sporadic - but it is very unlikely to have good DSL signal and bad authentication then self correct. Authentication is at a software protocol layer - not a hardware layer. DSL is more hardware. software corrupts and stays corrupt. It is very very very unlikely to go from a corrupt / failed state back to okay then back again multiple times. It also doesn't improve when the ISP resets their end of the connection.

Is it arrogant to have made an educated guess that the modem is probably stable from the above information? If so then all educated assumptions are arrogant.

What is arrogant is not taking time to listen to a client, and then work with them to resolve an issue. it is arrogant to not provide one drop of meaningful info when asked - e.g. whats the authentication server saying about our failed authentication?

But hey - you are SPARK. You know best. After two or more days of crap connection we have had a miracle and our modem is working beautifully again. Huzzaaahh!! The gods of I.T. have smiled upon us and reversed the state of a failing modem - or maybe it just wasn't failing!

> When a user goes oh, i'll fix this. i'll setup a broadband connection on their PC and accidentally created a PPPoE dialer, this caused disconnections.

1 - My users aren't that technically competent.

2 - If the damn BP person had done their job and looked at the authentication logs then we might do more than guess at whether this was a possibility or not.

If I am ranting then it is about this one thing: Everything you / BP are offering are wild guesses and supposition. I expect tests, logs and competence. Oddly I am the only one who has done reasonable levels of testing yet it seems this issue is my fault. Hmm - something wrong with that idea.

Oh yeah - when you say two hour response i don't expect to be waiting 8 plus hours even after chasing up.

RunningMan

8961 posts

Uber Geek

#2051117 7-Jul-2018 15:05

When troubleshooting an intermittent xDSL connection, swapping out the modem is a pretty fundamental and easy thing to do. It seems like a reasonable request from Bigpipe to isolate it from the equation. You mentioned you have a spare modem - downtime should only be pretty short to do the swap.

nunz

1421 posts

Uber Geek
Inactive user

#2051121 7-Jul-2018 15:15

Linux:

How do you know moisture has not got into one of the connections somewhere like at the ETP?

John

We have considered it - and it might be - hellishly bad rain in Chch recently. However the reset of the connection at their end improved the connection and that would seem to exclude damp or other physical causes.

We also have had no crackling or other indications of bad lines in the telephones and if it is hitting dsl copper then it is likely to hit phone.

Also DSL seems to be rock solid (if what BP meant by you have DSL shows no DSL droppage) . It was above that layer according to BP as no authentication. Again that seems to rule out physical layer issues.

Nothing seems to seriously point at the modem or physical cabling.

nunz

1421 posts

Uber Geek
Inactive user

#2051123 7-Jul-2018 15:27

RunningMan:

When troubleshooting an intermittent xDSL connection, swapping out the modem is a pretty fundamental and easy thing to do. It seems like a reasonable request from Bigpipe to isolate it from the equation. You mentioned you have a spare modem - downtime should only be pretty short to do the swap.

Each client has their own setup with VPN, Wireless, Certs, routing rules, firewalls, Static DHCP, and sometimes mac level rules to ensure internet script kiddies have less fun with our gateway. It's not a short swap out like a home network is.

And again (see the above post) nothing seems to point at a physical issue. If BP had actually looked at the authentication logs and given us info such as :

1 - Are we trying to authenticate? if so whats happening.

Then we would have something to go on and might have had to look at the modem. However at present changes at their end improve our connection and now us doing nothing returns perfect connectivity.

Swapping out a modem makes sense in some cases - but a crap shoot costing 2 - 4 hours of time and interruptions is hard to sell to a client, especially when diagnostics and logic seem to point else where.

There are two ways to fix an issue:

1 - Logically go through diagnostics, working up the OSI Layers until a problem is found - then fix it based on fact and logic.

2 - Throw stuff in the air and hope something sticks. This has been BPs approach. Even a monkey can eventually write poetry on the ceiling given enough time and poo to throw.

But I'm sapiens, not monkey. I'm paid for brains not luck.

The whole point of this post was "Hey BP - Need Help. Supposed to get it inside 2 hours and now after 8 hours plus still nothing!! Reaching out to resolve this. "

Honestly, long term I don't care if they resolve this or not. When the client instructs me to move suppliers it makes more paid work for me and costs me nothing. Not so BP.

RunningMan

8961 posts

Uber Geek

#2051138 7-Jul-2018 16:17

nunz: [snip]

Each client has their own setup with VPN, Wireless, Certs, routing rules, firewalls, Static DHCP, and sometimes mac level rules to ensure internet script kiddies have less fun with our gateway. It's not a short swap out like a home network is.

I would have thought the opposite - your average home user isn't likely to have the backup/exported config ready to drop into the replacement hardware as a contingency.

You don't say what hardware you are using, but a benefit in discrete modems, routers rather than all in one is that these sorts of swap outs are easier.

Linux

11430 posts

Uber Geek

Trusted

Lifetime subscriber

#2051173 7-Jul-2018 16:49

Then change ISP I'm sure they won't miss you as a customer

John

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

nunz

1421 posts

Uber Geek
Inactive user

#2054788 12-Jul-2018 09:50

Linux: Then change ISP I'm sure they won't miss you as a customer

John

...and again - think about improving your people skills.

Linux

11430 posts

Uber Geek

Trusted

Lifetime subscriber

#2054797 12-Jul-2018 09:58

nunz:

Linux: Then change ISP I'm sure they won't miss you as a customer

John

...and again - think about improving your people skills.

Those that live in glass houses should not throw stones!

John

nunz

1421 posts

Uber Geek
Inactive user

#2054800 12-Jul-2018 10:02

RunningMan:

nunz: [snip]

Each client has their own setup with VPN, Wireless, Certs, routing rules, firewalls, Static DHCP, and sometimes mac level rules to ensure internet script kiddies have less fun with our gateway. It's not a short swap out like a home network is.

I would have thought the opposite - your average home user isn't likely to have the backup/exported config ready to drop into the replacement hardware as a contingency.

You don't say what hardware you are using, but a benefit in discrete modems, routers rather than all in one is that these sorts of swap outs are easier.

I have backups of configs but here are the two issues we face with that.

1 - Backups of configs are missing settings - we have run into it before and notified the manufacturers who dont get back and dont care. when we restore we have to go through carefully checking each setting is working - a process that can be almost as long as setting them up from scratch.

2 - The spare modem / routers we have are not the same as the ones onsite. Why? Manufacturers keep changing models and short of buying 5 - 10 of the same device and waiting for the in place ones to die, then we cant keep a spare of every modem we are using onsite.

Re separation of function:

Generally our systems have a router -> firewall -> switch ~> wireless and other routers type scenario. In some places that's not feasible. This is one such place where we have three different networks in play, and putting dhcp and port forwarding on the firewall made the most sense. The router we used is one we would not buy again. We had failure rates of 40% over a year and switched manufacturer. Even spending money on supposedly premium brands was not a good return on investment.

While it may sound arrogant to not switch out the router as part of the testing - all diagnostics pointed to outside the router as an issue. It turns out my stand was correct as the network is running well again - and we changed nothing inside it. BP poking around on their networks resolved our issues. Why and how i have no idea - but we have a stable and fast connection again.

Swapping a router is often our first step - along with isolation tests etc. But again - everything in this case pointed else where - so subjecting the client to extra cost made no sense. If BP had worked with us, told us what the authentication logs said we would have had a cleaner resolution process. if BP had actually come back within 2 hours and not (after being chased 8 hours later) finally started responding the next day, then we would have had a cleaner process.

BP used to be a great ISP to refer people too. since being assimilated back into spark that's changed. We will start looking for a responsive company again.

nunz

1421 posts

Uber Geek
Inactive user

#2054802 12-Jul-2018 10:04

Linux:

nunz:

Linux: Then change ISP I'm sure they won't miss you as a customer

John

...and again - think about improving your people skills.

Those that live in glass houses should not throw stones!

John

john - Short of changing a router I had excluded based on reasonably solid diagnosis process (and which diagnosis proved correct) - what should I have done differently?

Linux

11430 posts

Uber Geek

Trusted

Lifetime subscriber

#2054825 12-Jul-2018 10:19

I would suggest pick an ISP that has a 24/7 NZ based help desk you can call and maybe get a managed service contract with SLA's

So in future when something goes wrong you can speak to someone direct

John

hio77

12999 posts

Uber Geek

ID Verified

Trusted

Lizard Networks

#2054857 12-Jul-2018 10:36

reading between the lines, it sounds like your using a bigpipe connection to resell your own network?

as such, you should have a plan that supports your usecase.

a residential connection is not designed for this.

If your network is so tightly designed around the fact that you can't switch out a piece of equipment, Your set out for failure - What happens when it eventually does die...

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

evnafets

537 posts

Ultimate Geek

Lifetime subscriber

#2054877 12-Jul-2018 11:05

Having been following this thread, for a while I can pick nunz's next response.

He has been there and done that with business plans and SLA's and paid money for support - which he doesn't rate that highly.

Bigpipe has in the past been very responsive and competent hence he was happy with them. As noticed in this thread (and others), the Bigpipe move from Spark Ventures back into the Spark 'mothership' has led to a decrease in the quality of support given.

Agreed that in this thread he appears to come across as a demanding git at times. But I can also see that is generally out of frustration.

He DOES have a process to replace the modem if necessary - but at this point (according to him) there is no evidence pointing to the need for that, while there is a wealth of evidence pointing to outside problems. No response from the support to verify / validate that evidence to give him a REASON to check out the modem.

I think what we all really want is https://xkcd.com/806/

nunz

1421 posts

Uber Geek
Inactive user

#2056454 15-Jul-2018 14:07

hio77:

reading between the lines, it sounds like your using a bigpipe connection to resell your own network?

as such, you should have a plan that supports your usecase.

a residential connection is not designed for this.

If your network is so tightly designed around the fact that you can't switch out a piece of equipment, Your set out for failure - What happens when it eventually does die...

1 - I dont resell networks. The client pays for their own network.

2 - I pay thousands / tens thousands per year for leased network / cloud equipment. I pay for quality. Having previously done this before with ISPs have found it not worth the cost as support always comes back with: Best efforts. The TCom , fixed speed, guaranteed lines failed a client, disrupting their NZ wide lan services. TCom took days to get onto it and most of the answers were : business support is 9-5, Mon to Fri. They didn't care the client was paying $1500 - $2500 per site per month for a 2Mbps dedicated line. It was their equivalent to this: https://www.sparkdigital.co.nz/solutions/connectivity/internet-service/ Support turned out to be ex-geni inhouse staff and even getting someone on the line became an issue. The 0800 225598 line had better hours of support and responsiveness - despite what we were meant to be receiving.

Similarly with clear / Vodafone - days of bitching on the phone before they sent a technician out who went - sorry, you were right. Lets fix that. Again, not a standard business or residential connection.

Signed contracts with vodafone when working with another company. Big connections as the company was a small isp with wireless roll out. Systems failed, Again no support. Tried to escalate and got the normal stalling routine. Politely asked for support at GeekZone and had higher up manager threaten the CEO of the company.

The best support was with Whoosh wireless - the cheapest isp around. Had a dedicated line, good tech support and responsive. Best ISP post earthquake working inner city. Money does not equate to quality and when SLAs fail the big companies are often prepared to drag it out as they know the costs of taking legally are too high for most smaller players.

Similarly with high end phone packages costing $2500 per month or more. Support came back with - we dont support that any more (just upgraded systems internally less than 6 months after selling it to us) so you need to rebuy your phones (less than 6 months after purchasing) and resign up for a full term.

Every time we have paid ISPs for quality it has been a shocking waste of time and money.

3 - All reasonable networks are tied around not being able to switch out a piece of equipment easily. VPNs look at MAC and certs, routing rules are applied at switches / routers, Secured wireless backbones often use MAC address to lock out spurious signals. There will always be a piece of equipment that is a break point unless you can pay for two of everything working in parallel including networks back to a different ISP on a different backbone.

The issue here was not the modem. Without any changes to it the network is back to working well after the ISP did things at their end.

My assertion that it would be a waste of money to switch out the router was correct. My diagnosis proved to be correct. I could get the systems switched out in a reasonable time (defined as what the client can stand in downtime) but I wont do it when a technician is making wild guesses based on no evidence to show there is a modem issue (and evidence showing the fault most likely lays else where). I have a fiscal responsibility to my client.

1 | 2 | 3 | 4 | 5