Vodafone Upstream (HE) major outage?

Forums › One New Zealand (including Vodafone, WxC, Farmside) › Vodafone Upstream (HE) major outage?

Lias

5589 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#242490 30-Oct-2018 19:51

I can't connect to a _lot_ of sites right now, and anything attempting to route via Hurricane appears to be the cause.

3 3 ms 3 ms 3 ms 10.200.12.17
4 137 ms 140 ms 138 ms vodafone-nz-ltd.10gigabitethernet6-11.core1.lax2.he.net [216.218.236.102]
5 138 ms 139 ms 139 ms 10ge6-11.core1.lax2.he.net [216.218.236.101]
6 139 ms 139 ms 138 ms 100ge2-2.core1.lax1.he.net [72.52.92.121]
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.

I'm a geek, a gamer, a dad, a Quic user, and an IT Professional. I have a full rack home lab, size 15 feet, an epic beard and Asperger's. I'm a bit of a Cypherpunk, who believes information wants to be free and the Net interprets censorship as damage and routes around it. If you use my Quic signup you can also use the code R570394EKGIZ8 for free setup.

mdf

3512 posts

Uber Geek

Trusted

#2117085 30-Oct-2018 19:59

I'm getting something similar with some sites, e.g.:

1 <1 ms <1 ms <1 ms 10.22.22.254
2 9 ms 35 ms 9 ms 203-109-234-1.dsl.dyn.ihug.co.nz [203.109.234.1]
3 10 ms 10 ms 10 ms 10.200.12.17
4 142 ms 177 ms 144 ms vodafone-nz-ltd.10gigabitethernet6-11.core1.lax2.he.net [216.218.236.102]
5 144 ms 142 ms 143 ms 10ge6-11.core1.lax2.he.net [216.218.236.101]
6 143 ms * 143 ms 100ge2-2.core1.lax1.he.net [72.52.92.121]
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.

Vodafone FibreX in Wellington

Lias

5589 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#2117092 30-Oct-2018 20:09

Something weird definitely appears to be going on within Hurricane, am now seeing some very intermittent connectivity beyond 72.52.92.121

e.g.

4 135 ms 139 ms 135 ms vodafone-nz-ltd.10gigabitethernet6-11.core1.lax2.he.net [216.218.236.102]
5 138 ms 138 ms 139 ms 10ge6-11.core1.lax2.he.net [216.218.236.101]
6 137 ms 135 ms 139 ms 100ge2-2.core1.lax1.he.net [72.52.92.121]
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * 302 ms 300 ms 100ge16-1.core1.ams1.he.net [184.105.213.65]
13 322 ms 323 ms 322 ms 100ge8-2.core1.sto1.he.net [184.105.65.126]
14 325 ms 323 ms 323 ms 100ge4-1.core1.tll1.he.net [184.104.192.82]
15 348 ms 347 ms 347 ms rg-b04.telia.lv [195.208.209.188]
16 * * * Request timed out.

Then a few minutes later

6 142 ms 138 ms 138 ms 100ge2-2.core1.lax1.he.net [72.52.92.121]
7 252 ms 223 ms 228 ms 100ge12-1.core1.ash1.he.net [184.105.80.201]
8 225 ms 223 ms 223 ms 100ge5-1.core2.ash1.he.net [72.52.92.226]
9 227 ms 227 ms 226 ms 100ge8-1.core1.nyc5.he.net [184.105.81.149]
10 286 ms 283 ms 286 ms 100ge8-2.core1.dub1.he.net [184.105.65.246]
11 289 ms 291 ms 290 ms 100ge3-2.core1.man1.he.net [72.52.92.197]
12 298 ms 299 ms 299 ms 100ge16-1.core1.ams1.he.net [184.105.213.65]
13 319 ms * * 100ge8-2.core1.sto1.he.net [184.105.65.126]
14 321 ms 322 ms 324 ms 100ge4-1.core1.tll1.he.net [184.104.192.82]
15 * * * Request timed out.
16 * * * Request timed out.

Lias

5589 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#2117093 30-Oct-2018 20:15

@joelfreak it'll probably be over by the time you see this, but can ya come let us know what was going on? :-)

Joelfreak

6 posts

Wannabe Geek

Trusted

Hurricane Electric

#2117109 30-Oct-2018 20:50

This was a window for router maintenance in the USA. You are seeing reconvergence of the network. This probably looked like this for... Under a minute?

Lias

5589 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#2117159 31-Oct-2018 00:44

Joelfreak: This was a window for router maintenance in the USA. You are seeing reconvergence of the network. This probably looked like this for... Under a minute?

Cheers @joelfreak, but I'd say much longer, around 30-35 minutes?

I'd spent 5-10 minutes troubleshooting and checking things myself before I posted at 19:51 NZ time, started to get some intermittent traffic but definitely not normal service around 20:05-20:10ish, but I'd say it was another 10+ minutes after that everything stopped being intermittent and resumed normal service.

PS: Any chance of you guys making a public status page? I had a look for one but couldn't find one.

Joelfreak

6 posts

Wannabe Geek

Trusted

Hurricane Electric

#2117163 31-Oct-2018 05:55

Actually, in this case there was a small outage on some backbone circuits as well which is probably why you saw what you saw. This wasn't anything major, and customers are notified of any outages affecting routers they directly connect to.