Devoli Outage 20-08-19

Forums › New Zealand Broadband › Devoli Outage 20-08-19

chevrolux

4962 posts

Uber Geek
+1 received by user: 2638
Inactive user

#255599 20-Aug-2019 09:37

Morning team,

Anyone been caught up in this cluster f this morning?

Apparently they did an upgrade at Mayoral Drive last night, the update has introduced a bunch of bugs and now they are rolling back.

How does a company like this not actually test this sort of stuff before upgrading it?!

1 | 2

11094 posts

Uber Geek
+1 received by user: 6071

Trusted

Lifetime subscriber

#2302257 20-Aug-2019 10:17

Hmm... my home connection was down this morning, and I still can't remote into it so it appears to still be down. I wonder whether my ISP (Mynxnet) uses Devoli upstream, or whether it's an unrelated coincidence.

I need to shop around for a better deal one of these days, but that's neither here nor there...

Edit: Looks like it could be a Yes; my IP address is in the range listed here!

Beccara

1473 posts

Uber Geek
+1 received by user: 517

ID Verified

#2302264 20-Aug-2019 10:19

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year

Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated

AgentK

34 posts

Geek
+1 received by user: 11

#2302276 20-Aug-2019 10:37

Mynxnet is the residential ISP business of Devoli, yes. Vibecom baught Mynxnet a few years ago off a friend of mine who originally started it.

And yes - I agree, Devoli's planned maintenance at MDR these days nearly always seem to result in something breaking or having to be rolled back.

chevrolux

4962 posts

Uber Geek
+1 received by user: 2638
Inactive user

#2302282 20-Aug-2019 10:49

Beccara:

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year

Yep for us, 3rd issue in as many weeks.

All related to doing network upgrades that haven't worked. First was a router at MDR gonig down and being swapped one Friday evening, last week they made a routing change that sent SIP traffic headed to Auckland off on a wild goose chase to AU at around midday on a business day, and now this today which has probably had the biggest impact for us at is has affected a ton of their NGA handovers.

Supposedly they have bought a bunch on brand new Juniper gear, but it's full of bugs and they keep having to do these rollbacks or switching across to Cisco. Christ at this rate I could go throw them some Mikrotik gear and it would be more reliable!!

Beccara

1473 posts

Uber Geek
+1 received by user: 517

ID Verified

#2302286 20-Aug-2019 10:53

Yeah cisco house is going to have some diehards in there moving to Juniper. Almost all major ISP's run Juniper so more likely to be PEBCAC issues than actual bugs. Run away from them, If you need stability stick to the majors and for SIP always run 2 carriers with trunk down failover capacity

MichaelNZ

1594 posts

Uber Geek
+1 received by user: 485

Trusted

Net Trust Ltd

#2305533 24-Aug-2019 22:58

chevrolux:

Christ at this rate I could go throw them some Mikrotik gear and it would be more reliable!!

Unfortunately, Mikrotik, as great as it is, has a low ceiling with full table BGP. In context - it does work and we are doing it - but anymore than about 1.5 - 2 million routes is beyond its capacity.

Hopefully ROS7 and it's much anticipated multi-threaded BGP will sort this out.

WFH Linux Systems and Networks Engineer in the Internet industry | Specialising in Mikrotik | APNIC member | Open to job offers | ZL2NET

Samsung

Shop now on Samsung phones, tablets, TVs and more (affiliate link).

sparkz25

750 posts

Ultimate Geek
+1 received by user: 284
Inactive user

#2305547 24-Aug-2019 23:59

chevrolux:

Morning team,

Anyone been caught up in this cluster f this morning?

Apparently they did an upgrade at Mayoral Drive last night, the update has introduced a bunch of bugs and now they are rolling back.

How does a company like this not actually test this sort of stuff before upgrading it?!

Yes we were caught up in this, and as the report mentions all was well until 0725am, Then it all went to Sh1t. The report is below for the event that day.

https://www.dropbox.com/s/rttm154mtm3dbix/%20Devoli%20RFO%20200819%20-%20final%20.pdf?dl=0

Beccara:

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year

Who are you using for your wholesale and how have you found them?

Beccara

1473 posts

Uber Geek
+1 received by user: 517

ID Verified

#2305594 25-Aug-2019 09:06

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.

noroad

1025 posts

Uber Geek
+1 received by user: 675

Trusted

#2305596 25-Aug-2019 09:09

Who are you using for your wholesale and how have you found them?

wholesale suggestion... http://feenix.co.nz

chevrolux

4962 posts

Uber Geek
+1 received by user: 2638
Inactive user

#2305597 25-Aug-2019 09:12

Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

dt

1152 posts

Uber Geek
+1 received by user: 371
Inactive user

#2305635 25-Aug-2019 11:33

Our company was momentarily effect by this, luckily our service provider swung us over to backup provider but our internet break out was down for at least an hour before that happened and when you multiply that by 800 users it was quite a disruption to our business.

chevrolux: Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

Totally agree with this, what the hell.. It was an upgrade from the night before as well.. AFAIK they didn't even know until companies started coming online the following morning.

They clearly have a poor processes in place and has been the straw that broke the camels back for our network provider as they have already kicked off the process of looking for another primary wholesaler.

HP

Shop now for HP laptops and other devices (affiliate link).

sparkz25

750 posts

Ultimate Geek
+1 received by user: 284
Inactive user

#2305678 25-Aug-2019 11:46

chevrolux: Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

That report is publicly available if you know where to look.

http://devoli.status.io

Its on there if you have a good look.

sparkz25

750 posts

Ultimate Geek
+1 received by user: 284
Inactive user

#2305680 25-Aug-2019 11:51

Beccara:

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.

I thought that Lightwire used Vocus as their backhaul?

And they are owned by IT Partners as well.

https://bgp.he.net/AS45267#_graph4

MichaelNZ

1594 posts

Uber Geek
+1 received by user: 485

Trusted

Net Trust Ltd

#2305681 25-Aug-2019 11:52

chevrolux:
But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

1. Test environment with some expensive pieces of kit sitting there idle most of the time.

2. Cheapest internet access

Both positions exist on Geekzone and both can't be true.

WFH Linux Systems and Networks Engineer in the Internet industry | Specialising in Mikrotik | APNIC member | Open to job offers | ZL2NET

Beccara

1473 posts

Uber Geek
+1 received by user: 517

ID Verified

#2305688 25-Aug-2019 12:12

sparkz25:

Beccara:

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.

I thought that Lightwire used Vocus as their backhaul?

And they are owned by IT Partners as well.

https://bgp.he.net/AS45267#_graph4

Brendan from DTS is handling their business side and running a good setup, Chorus wavelengths around the country etc for tails and agg. Using Vocus for International but they also run their own kit in AKL/SYD so there's 1/3-2/3rds of your traffic anyway. Ownership I dont think has changed, they work with IT Partners but not owned by them

1 | 2