Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


chevrolux

4962 posts

Uber Geek
Inactive user


#255599 20-Aug-2019 09:37
Send private message

Morning team,

 

Anyone been caught up in this cluster f this morning?

 

Apparently they did an upgrade at Mayoral Drive last night, the update has introduced a bunch of bugs and now they are rolling back.

 

How does a company like this not actually test this sort of stuff before upgrading it?!


View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
Behodar
10501 posts

Uber Geek

Trusted
Lifetime subscriber

  #2302257 20-Aug-2019 10:17
Send private message

Hmm... my home connection was down this morning, and I still can't remote into it so it appears to still be down. I wonder whether my ISP (Mynxnet) uses Devoli upstream, or whether it's an unrelated coincidence.

 

 

I need to shop around for a better deal one of these days, but that's neither here nor there...

 

 

Edit: Looks like it could be a Yes; my IP address is in the range listed here!



Beccara
1469 posts

Uber Geek

ID Verified

  #2302264 20-Aug-2019 10:19
Send private message

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

AgentK
34 posts

Geek


  #2302276 20-Aug-2019 10:37
Send private message

Mynxnet is the residential ISP business of Devoli, yes. Vibecom baught Mynxnet a few years ago off a friend of mine who originally started it.

 

And yes - I agree, Devoli's planned maintenance at MDR these days nearly always seem to result in something breaking or having to be rolled back.




chevrolux

4962 posts

Uber Geek
Inactive user


  #2302282 20-Aug-2019 10:49
Send private message

Beccara:

 

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year

 

 

Yep for us, 3rd issue in as many weeks.

 

All related to doing network upgrades that haven't worked. First was a router at MDR gonig down and being swapped one Friday evening, last week they made a routing change that sent SIP traffic headed to Auckland off on a wild goose chase to AU at around midday on a business day, and now this today which has probably had the biggest impact for us at is has affected a ton of their NGA handovers.

 

Supposedly they have bought a bunch on brand new Juniper gear, but it's full of bugs and they keep having to do these rollbacks or switching across to Cisco. Christ at this rate I could go throw them some Mikrotik gear and it would be more reliable!!

 

 


Beccara
1469 posts

Uber Geek

ID Verified

  #2302286 20-Aug-2019 10:53
Send private message

Yeah cisco house is going to have some diehards in there moving to Juniper. Almost all major ISP's run Juniper so more likely to be PEBCAC issues than actual bugs. Run away from them, If you need stability stick to the majors and for SIP always run 2 carriers with trunk down failover capacity





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

MichaelNZ
1385 posts

Uber Geek

Trusted
Integrity Tech Solutions

  #2305533 24-Aug-2019 22:58
Send private message

chevrolux:

 

Christ at this rate I could go throw them some Mikrotik gear and it would be more reliable!!

 

 

Unfortunately, Mikrotik, as great as it is, has a low ceiling with full table BGP. In context - it does work and we are doing it - but anymore than about 1.5 - 2 million routes is beyond its capacity.

 

Hopefully ROS7 and it's much anticipated multi-threaded BGP will sort this out.





WFH Linux Systems and Networks Engineer in the Internet industry | Specialising in Mikrotik | APNIC member | Open to job offers


sparkz25
750 posts

Ultimate Geek
Inactive user


  #2305547 24-Aug-2019 23:59
Send private message

chevrolux:

 

Morning team,

 

Anyone been caught up in this cluster f this morning?

 

Apparently they did an upgrade at Mayoral Drive last night, the update has introduced a bunch of bugs and now they are rolling back.

 

How does a company like this not actually test this sort of stuff before upgrading it?!

 

 

Yes we were caught up in this, and as the report mentions all was well until 0725am, Then it all went to Sh1t. The report is below for the event that day.

 

https://www.dropbox.com/s/rttm154mtm3dbix/%20Devoli%20RFO%20200819%20-%20final%20.pdf?dl=0

 

Beccara:

 

I am super super glad I never decided to use them for wholesale. They have shown time and time again they can't manage their network and their higher ups were gloating years back they were just waiting for a buyout. This will be the 3rd major incident for them I've been aware of this year

 

 

Who are you using for your wholesale and how have you found them?


 
 
 

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.
Beccara
1469 posts

Uber Geek

ID Verified

  #2305594 25-Aug-2019 09:06
Send private message

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

noroad
949 posts

Ultimate Geek

Trusted

  #2305596 25-Aug-2019 09:09
Send private message

 

Who are you using for your wholesale and how have you found them?

 

 

 

 

wholesale suggestion... http://feenix.co.nz

 

 


chevrolux

4962 posts

Uber Geek
Inactive user


  #2305597 25-Aug-2019 09:12
Send private message

Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

dt

dt
1152 posts

Uber Geek
Inactive user


  #2305635 25-Aug-2019 11:33
Send private message

Our company was momentarily effect by this, luckily our service provider swung us over to backup provider but our internet break out was down for at least an hour before that happened and when you multiply that by 800 users it was quite a disruption to our business. 

 

 

 

chevrolux: Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

 

 

 

Totally agree with this, what the hell.. It was an upgrade from the night before as well.. AFAIK they didn't even know until companies started coming online the following morning.

 

They clearly have a poor processes in place and has been the straw that broke the camels back for our network provider as they have already kicked off the process of looking for another primary wholesaler. 


sparkz25
750 posts

Ultimate Geek
Inactive user


  #2305678 25-Aug-2019 11:46
Send private message

chevrolux: Not sure that report should be posted on a public forum?...

But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

 

That report is publicly available if you know where to look.

 

http://devoli.status.io

 

Its on there if you have a good look.


sparkz25
750 posts

Ultimate Geek
Inactive user


  #2305680 25-Aug-2019 11:51
Send private message

Beccara:

 

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.

 

 

I thought that Lightwire used Vocus as their backhaul?

 

And they are owned by IT Partners as well.

 

 

 

https://bgp.he.net/AS45267#_graph4


MichaelNZ
1385 posts

Uber Geek

Trusted
Integrity Tech Solutions

  #2305681 25-Aug-2019 11:52
Send private message

chevrolux:
But yea, the thing that really annoyed me there was that they are still blaming the whole thing on juniper software bug. Surely, for such a key network element, you have an environment to test thoroughly before putting in to production!!

 

1. Test environment with some expensive pieces of kit sitting there idle most of the time.

 

2. Cheapest internet access

 

Both positions exist on Geekzone and both can't be true.





WFH Linux Systems and Networks Engineer in the Internet industry | Specialising in Mikrotik | APNIC member | Open to job offers


Beccara
1469 posts

Uber Geek

ID Verified

  #2305688 25-Aug-2019 12:12
Send private message

sparkz25:

 

Beccara:

 

Vocus but since they've been acting like a cat wanting to go outside/inside with the whole sale/splitup saga we're moving to Lightwire Business.

 

 

I thought that Lightwire used Vocus as their backhaul?

 

And they are owned by IT Partners as well.

 

 

 

https://bgp.he.net/AS45267#_graph4

 

 

 

 

Brendan from DTS is handling their business side and running a good setup, Chorus wavelengths around the country etc for tails and agg. Using Vocus for International but they also run their own kit in AKL/SYD so there's 1/3-2/3rds of your traffic anyway. Ownership I dont think has changed, they work with IT Partners but not owned by them





Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated 

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic





News and reviews »

Air New Zealand Starts AI adoption with OpenAI
Posted 24-Jul-2025 16:00


eero Pro 7 Review
Posted 23-Jul-2025 12:07


BeeStation Plus Review
Posted 21-Jul-2025 14:21


eero Unveils New Wi-Fi 7 Products in New Zealand
Posted 21-Jul-2025 00:01


WiZ Introduces HDMI Sync Box and other Light Devices
Posted 20-Jul-2025 17:32


RedShield Enhances DDoS and Bot Attack Protection
Posted 20-Jul-2025 17:26


Seagate Ships 30TB Drives
Posted 17-Jul-2025 11:24


Oclean AirPump A10 Water Flosser Review
Posted 13-Jul-2025 11:05


Samsung Galaxy Z Fold7: Raising the Bar for Smartphones
Posted 10-Jul-2025 02:01


Samsung Galaxy Z Flip7 Brings New Edge-To-Edge FlexWindow
Posted 10-Jul-2025 02:01


Epson Launches New AM-C550Z WorkForce Enterprise printer
Posted 9-Jul-2025 18:22


Samsung Releases Smart Monitor M9
Posted 9-Jul-2025 17:46


Nearly Half of Older Kiwis Still Write their Passwords on Paper
Posted 9-Jul-2025 08:42


D-Link 4G+ Cat6 Wi-Fi 6 DWR-933M Mobile Hotspot Review
Posted 1-Jul-2025 11:34


Oppo A5 Series Launches With New Levels of Durability
Posted 30-Jun-2025 10:15









Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.