Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




4212 posts

Uber Geek


# 255599 20-Aug-2019 09:37
Send private message quote this post

Morning team,

 

Anyone been caught up in this cluster f this morning?

 

Apparently they did an upgrade at Mayoral Drive last night, the update has introduced a bunch of bugs and now they are rolling back.

 

How does a company like this not actually test this sort of stuff before upgrading it?!


View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
6735 posts

Uber Geek

Trusted
Lifetime subscriber

  # 2302257 20-Aug-2019 10:17
Send private message quote this post

Hmm... my home connection was down this morning, and I still can't remote into it so it appears to still be down. I wonder whether my ISP (Mynxnet) uses Devoli upstream, or whether it's an unrelated coincidence.

 

 

I need to shop around for a better deal one of these days, but that's neither here nor there...

 

 

Edit: Looks like it could be a Yes; my IP address is in the range listed here!

1129 posts

Uber Geek


  # 2302264 20-Aug-2019 10:19
4 people support this post
Send private message quote this post

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

 
 
 
 


20 posts

Geek


  # 2302276 20-Aug-2019 10:37
Send private message quote this post

Mynxnet is the residential ISP business of Devoli, yes. Vibecom baught Mynxnet a few years ago off a friend of mine who originally started it.

 

And yes - I agree, Devoli's planned maintenance at MDR these days nearly always seem to result in something breaking or having to be rolled back.




4212 posts

Uber Geek


  # 2302282 20-Aug-2019 10:49
2 people support this post
Send private message quote this post

Beccara:

 

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year

 

 

Yep for us, 3rd issue in as many weeks.

 

All related to doing network upgrades that haven't worked. First was a router at MDR gonig down and being swapped one Friday evening, last week they made a routing change that sent SIP traffic headed to Auckland off on a wild goose chase to AU at around midday on a business day, and now this today which has probably had the biggest impact for us at is has affected a ton of their NGA handovers.

 

Supposedly they have bought a bunch on brand new Juniper gear, but it's full of bugs and they keep having to do these rollbacks or switching across to Cisco. Christ at this rate I could go throw them some Mikrotik gear and it would be more reliable!!

 

 


1129 posts

Uber Geek


  # 2302286 20-Aug-2019 10:53
One person supports this post
Send private message quote this post

Yeah cisco house is going to have some diehards in there moving to Juniper. Almost all major ISP's run Juniper so more likely to be PEBCAC issues than actual bugs. Run away from them, If you need stability stick to the majors and for SIP always run 2 carriers with trunk down failover capacity





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

Linux Systems Admin
1123 posts

Uber Geek

Trusted
Integrity Tech Solutions
Subscriber

  # 2305533 24-Aug-2019 22:58
One person supports this post
Send private message quote this post

chevrolux:

 

Christ at this rate I could go throw them some Mikrotik gear and it would be more reliable!!

 

 

Unfortunately, Mikrotik, as great as it is, has a low ceiling with full table BGP. In context - it does work and we are doing it - but anymore than about 1.5 - 2 million routes is beyond its capacity.

 

Hopefully ROS7 and it's much anticipated multi-threaded BGP will sort this out.





Integrity Tech Solutions @ Norsewood, New Zealand


420 posts

Ultimate Geek


  # 2305547 24-Aug-2019 23:59
Send private message quote this post

chevrolux:

 

Morning team,

 

Anyone been caught up in this cluster f this morning?

 

Apparently they did an upgrade at Mayoral Drive last night, the update has introduced a bunch of bugs and now they are rolling back.

 

How does a company like this not actually test this sort of stuff before upgrading it?!

 

 

Yes we were caught up in this, and as the report mentions all was well until 0725am, Then it all went to Sh1t. The report is below for the event that day.

 

https://www.dropbox.com/s/rttm154mtm3dbix/%20Devoli%20RFO%20200819%20-%20final%20.pdf?dl=0

 

Beccara:

 

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year

 

 

Who are you using for your wholesale and how have you found them?


 
 
 
 


1129 posts

Uber Geek


  # 2305594 25-Aug-2019 09:06
Send private message quote this post

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

526 posts

Ultimate Geek

Trusted

  # 2305596 25-Aug-2019 09:09
Send private message quote this post

 

Who are you using for your wholesale and how have you found them?

 

 

 

 

wholesale suggestion... http://feenix.co.nz

 

 




4212 posts

Uber Geek


  # 2305597 25-Aug-2019 09:12
One person supports this post
Send private message quote this post

Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

dt

525 posts

Ultimate Geek


  # 2305635 25-Aug-2019 11:33
One person supports this post
Send private message quote this post

Our company was momentarily effect by this, luckily our service provider swung us over to backup provider but our internet break out was down for at least an hour before that happened and when you multiply that by 800 users it was quite a disruption to our business. 

 

 

 

chevrolux: Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

 

 

 

Totally agree with this, what the hell.. It was an upgrade from the night before as well.. AFAIK they didn't even know until companies started coming online the following morning.

 

They clearly have a poor processes in place and has been the straw that broke the camels back for our network provider as they have already kicked off the process of looking for another primary wholesaler. 


420 posts

Ultimate Geek


  # 2305678 25-Aug-2019 11:46
Send private message quote this post

chevrolux: Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

 

That report is publicly available if you know where to look.

 

http://devoli.status.io

 

Its on there if you have a good look.


420 posts

Ultimate Geek


  # 2305680 25-Aug-2019 11:51
Send private message quote this post

Beccara:

 

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.

 

 

I thought that Lightwire used Vocus as their backhaul?

 

And they are owned by IT Partners as well.

 

 

 

https://bgp.he.net/AS45267#_graph4


Linux Systems Admin
1123 posts

Uber Geek

Trusted
Integrity Tech Solutions
Subscriber

  # 2305681 25-Aug-2019 11:52
Send private message quote this post

chevrolux:
But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

 

1. Test environment with some expensive pieces of kit sitting there idle most of the time.

 

2. Cheapest internet access

 

Both positions exist on Geekzone and both can't be true.





Integrity Tech Solutions @ Norsewood, New Zealand


1129 posts

Uber Geek


  # 2305688 25-Aug-2019 12:12
Send private message quote this post

sparkz25:

 

Beccara:

 

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.

 

 

I thought that Lightwire used Vocus as their backhaul?

 

And they are owned by IT Partners as well.

 

 

 

https://bgp.he.net/AS45267#_graph4

 

 

 

 

Brendan from DTS is handling their business side and running a good setup, Chorus wavelengths around the country etc for tails and agg. Using Vocus for International but they also run their own kit in AKL/SYD so there's 1/3-2/3rds of your traffic anyway. Ownership I dont think has changed, they work with IT Partners but not owned by them





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic



Switch your broadband provider now - compare prices


Twitter and LinkedIn »



Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

OPPO Reno2 Series lands, alongside hybrid noise-cancelling Wireless Headphones
Posted 24-Oct-2019 15:32


Waikato Data Scientists awarded $13 million from the Government
Posted 24-Oct-2019 15:27


D-Link launches Wave 2 Unified Access Points
Posted 24-Oct-2019 15:07


LG Electronics begins distributing the G8X THINQ
Posted 24-Oct-2019 10:58


Arlo unveils its first video doorbell
Posted 21-Oct-2019 08:27


New Zealand students shortlisted for James Dyson Award
Posted 21-Oct-2019 08:18


Norton LifeLock Launches Norton 360
Posted 21-Oct-2019 08:11


Microsoft New Zealand Partner Awards results
Posted 18-Oct-2019 10:18


Logitech introduces new Made for Google keyboard and mouse devices
Posted 16-Oct-2019 13:36


MATTR launches to accelerate decentralised identity
Posted 16-Oct-2019 10:28


Vodafone X-Squad powers up for customers
Posted 16-Oct-2019 08:15


D Link ANZ launches EXO Smart Mesh Wi Fi Routers with McAfee protection
Posted 15-Oct-2019 11:31


Major Japanese retailer partners with smart New Zealand technology IMAGR
Posted 14-Oct-2019 10:29


Ola pioneers one-time passcode feature to fight rideshare fraud
Posted 14-Oct-2019 10:24


Spark Sport new home of NZC matches from 2020
Posted 10-Oct-2019 09:59



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.