Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




4499 posts

Uber Geek

Trusted

#255599 20-Aug-2019 09:37
Send private message

Morning team,

 

Anyone been caught up in this cluster f this morning?

 

Apparently they did an upgrade at Mayoral Drive last night, the update has introduced a bunch of bugs and now they are rolling back.

 

How does a company like this not actually test this sort of stuff before upgrading it?!


View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
7015 posts

Uber Geek

Trusted
Lifetime subscriber

  #2302257 20-Aug-2019 10:17
Send private message

Hmm... my home connection was down this morning, and I still can't remote into it so it appears to still be down. I wonder whether my ISP (Mynxnet) uses Devoli upstream, or whether it's an unrelated coincidence.

 

 

I need to shop around for a better deal one of these days, but that's neither here nor there...

 

 

Edit: Looks like it could be a Yes; my IP address is in the range listed here!

1276 posts

Uber Geek


  #2302264 20-Aug-2019 10:19
Send private message

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

 
 
 
 


20 posts

Geek


  #2302276 20-Aug-2019 10:37
Send private message

Mynxnet is the residential ISP business of Devoli, yes. Vibecom baught Mynxnet a few years ago off a friend of mine who originally started it.

 

And yes - I agree, Devoli's planned maintenance at MDR these days nearly always seem to result in something breaking or having to be rolled back.




4499 posts

Uber Geek

Trusted

  #2302282 20-Aug-2019 10:49
Send private message

Beccara:

 

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year

 

 

Yep for us, 3rd issue in as many weeks.

 

All related to doing network upgrades that haven't worked. First was a router at MDR gonig down and being swapped one Friday evening, last week they made a routing change that sent SIP traffic headed to Auckland off on a wild goose chase to AU at around midday on a business day, and now this today which has probably had the biggest impact for us at is has affected a ton of their NGA handovers.

 

Supposedly they have bought a bunch on brand new Juniper gear, but it's full of bugs and they keep having to do these rollbacks or switching across to Cisco. Christ at this rate I could go throw them some Mikrotik gear and it would be more reliable!!

 

 


1276 posts

Uber Geek


  #2302286 20-Aug-2019 10:53
Send private message

Yeah cisco house is going to have some diehards in there moving to Juniper. Almost all major ISP's run Juniper so more likely to be PEBCAC issues than actual bugs. Run away from them, If you need stability stick to the majors and for SIP always run 2 carriers with trunk down failover capacity





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

Linux Systems Admin
1173 posts

Uber Geek

Trusted
Integrity Tech Solutions
Subscriber

  #2305533 24-Aug-2019 22:58
Send private message

chevrolux:

 

Christ at this rate I could go throw them some Mikrotik gear and it would be more reliable!!

 

 

Unfortunately, Mikrotik, as great as it is, has a low ceiling with full table BGP. In context - it does work and we are doing it - but anymore than about 1.5 - 2 million routes is beyond its capacity.

 

Hopefully ROS7 and it's much anticipated multi-threaded BGP will sort this out.





Integrity Tech Solutions @ Norsewood, New Zealand


535 posts

Ultimate Geek


  #2305547 24-Aug-2019 23:59
Send private message

chevrolux:

 

Morning team,

 

Anyone been caught up in this cluster f this morning?

 

Apparently they did an upgrade at Mayoral Drive last night, the update has introduced a bunch of bugs and now they are rolling back.

 

How does a company like this not actually test this sort of stuff before upgrading it?!

 

 

Yes we were caught up in this, and as the report mentions all was well until 0725am, Then it all went to Sh1t. The report is below for the event that day.

 

https://www.dropbox.com/s/rttm154mtm3dbix/%20Devoli%20RFO%20200819%20-%20final%20.pdf?dl=0

 

Beccara:

 

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year

 

 

Who are you using for your wholesale and how have you found them?


 
 
 
 


1276 posts

Uber Geek


  #2305594 25-Aug-2019 09:06
Send private message

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

593 posts

Ultimate Geek

Trusted

  #2305596 25-Aug-2019 09:09
Send private message

 

Who are you using for your wholesale and how have you found them?

 

 

 

 

wholesale suggestion... http://feenix.co.nz

 

 




4499 posts

Uber Geek

Trusted

  #2305597 25-Aug-2019 09:12
Send private message

Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

dt

616 posts

Ultimate Geek


  #2305635 25-Aug-2019 11:33
Send private message

Our company was momentarily effect by this, luckily our service provider swung us over to backup provider but our internet break out was down for at least an hour before that happened and when you multiply that by 800 users it was quite a disruption to our business. 

 

 

 

chevrolux: Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

 

 

 

Totally agree with this, what the hell.. It was an upgrade from the night before as well.. AFAIK they didn't even know until companies started coming online the following morning.

 

They clearly have a poor processes in place and has been the straw that broke the camels back for our network provider as they have already kicked off the process of looking for another primary wholesaler. 


535 posts

Ultimate Geek


  #2305678 25-Aug-2019 11:46
Send private message

chevrolux: Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

 

That report is publicly available if you know where to look.

 

http://devoli.status.io

 

Its on there if you have a good look.


535 posts

Ultimate Geek


  #2305680 25-Aug-2019 11:51
Send private message

Beccara:

 

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.

 

 

I thought that Lightwire used Vocus as their backhaul?

 

And they are owned by IT Partners as well.

 

 

 

https://bgp.he.net/AS45267#_graph4


Linux Systems Admin
1173 posts

Uber Geek

Trusted
Integrity Tech Solutions
Subscriber

  #2305681 25-Aug-2019 11:52
Send private message

chevrolux:
But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

 

1. Test environment with some expensive pieces of kit sitting there idle most of the time.

 

2. Cheapest internet access

 

Both positions exist on Geekzone and both can't be true.





Integrity Tech Solutions @ Norsewood, New Zealand


1276 posts

Uber Geek


  #2305688 25-Aug-2019 12:12
Send private message

sparkz25:

 

Beccara:

 

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.

 

 

I thought that Lightwire used Vocus as their backhaul?

 

And they are owned by IT Partners as well.

 

 

 

https://bgp.he.net/AS45267#_graph4

 

 

 

 

Brendan from DTS is handling their business side and running a good setup, Chorus wavelengths around the country etc for tails and agg. Using Vocus for International but they also run their own kit in AKL/SYD so there's 1/3-2/3rds of your traffic anyway. Ownership I dont think has changed, they work with IT Partners but not owned by them





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic



Twitter and LinkedIn »



Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

OPPO Find X2 Lite brings flagship features to mid-range 5G smartphone
Posted 29-May-2020 12:52


Sony introduces the digital camera ZV-1 for content creators
Posted 27-May-2020 12:47


Samsung Announces 2020 QLED TV Range
Posted 20-May-2020 16:29


D-Link A/NZ launches AI-Powered body temperature measuring system
Posted 20-May-2020 16:22


NortonLifeLock Online Banking Protection now available for New Zealand banks
Posted 20-May-2020 16:14


SD Express delivers new gigabyte speeds for SD memory cards
Posted 20-May-2020 15:00


D-Link A/NZ launches Nuclias cloud managed network solution hosted in Australia
Posted 11-May-2020 17:53


Logitech introduces new video streaming solution for home studios
Posted 11-May-2020 17:48


Next generation Volvo cars to be powered by Luminar LiDAR technology
Posted 7-May-2020 13:56


D-Link A/NZ launches Wi-Fi Certified EasyMesh system
Posted 7-May-2020 13:51


Spark teams up with Microsoft to bring Xbox All Access to New Zealand
Posted 7-May-2020 13:01


Microsoft plans to establish its first datacenter region in New Zealand
Posted 6-May-2020 11:35


Genesis School-gen has joined forces with Mind Lab Kids
Posted 1-May-2020 12:53


Malwarebytes expands into privacy with fast, frictionless VPN
Posted 30-Apr-2020 16:06


Kordia to donate TV airtime on Channel 200 to community groups
Posted 30-Apr-2020 16:00



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.