Weird packet loss 23/01/20

Forums › One NZ (including Farmside) › Weird packet loss 23/01/20

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#265487 23-Jan-2020 22:48

Noticed some trouble connecting to some servers tonight, did an mtr, getting some packet loss from a telstraglobal.net node (which the response time when it does respond indicates is in NZ), but what is weird is that an mtr to one of my amazon IP's gets returns from that same node without issue.

Anybody else getting connection trouble tonight?

Bad:

Good:

Bad (No DNS):

Good (No DNS):

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

1 | 2

2239 posts

Uber Geek
+1 received by user: 594

#2405587 23-Jan-2020 23:08

In that case the problem is likely to be on the return path.

Not on Vodafone but I'm also getting loss pinging the problematic IP

--- 72.34.38.3 ping statistics ---
52 packets transmitted, 39 packets received, 25% packet loss
round-trip min/avg/max = 181.372/182.144/182.913 ms

On pathping/traceroute the loss begins from 67.222.102.106 for me.

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#2405601 23-Jan-2020 23:50

yitz:

In that case the problem is likely to be on the return path

I thought about that

Internet routing is well above my pay grade, but I didn't think that the return path from the final destination should affect ICMP echos from points earlier in the route.

In other words, in a route A B C if C has a bad route getting to B then B should still be able to return their own echo to A, just responses from C would be affected.

But that appears to be what I see in those mtr traces, the high packet loss begins early in the trace and yet it's entirely dependant on the target IP address way later in the trace.

Like there is some sort of filtering going on at that first point of bad packet loss based on the destination IP.

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#2405606 24-Jan-2020 00:25

Somebody else has tweeted generally about bad international traffic tonight

https://twitter.com/katZicael/status/1220246096850472960

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#2405660 24-Jan-2020 09:44

Hmm, this is a gif flipping between two mtr traces to two IPs on the same network, indeed both of these ips are the same physical server.

Hop 6 (and the last one) changes only, and packet loss is seen from hop-7 onwards.

It seems reproducible, if hop 6 is .37 it has the losses **with this destination**, if hop 6 is not it doesn't.

"mtr --no-dns 72.34.48.2" always produces the bad hop 6

"mtr --no-dns 72.34.48.17" never produces the bad hop 6

But

"mtr --no-dns 54.241.37.107" has .37 hop 6 and is OK, BUT the hop 7 is different to the above

So... I'm going to point to 134.159.174.37 ( unknown.telstraglobal.net ) having issues routing to or back from 202.84.223.110 ( i-0-4-0-1.hptw-core02.telstraglobal.net. )?

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

yitz

2239 posts

Uber Geek
+1 received by user: 594

#2405762 24-Jan-2020 12:20

OK you might be right it does seem Vodafone have a bad link somewhere probably closer to Auckland than too far offshore.

--- 202.78.155.1 ping statistics ---
101 packets transmitted, 87 packets received, 13% packet loss
round-trip min/avg/max = 189.200/211.970/306.129 ms

Interesting that you seem to be on a specific part of their network that is non-peered (explains higher round trip times), pinging from same location within NZ to another part of the Vodafone NZ network:

--- 27.252.203.254 ping statistics ---
101 packets transmitted, 101 packets received, 0% packet loss
round-trip min/avg/max = 24.914/25.192/26.280 ms

I'm still get small amounts of loss to those 72.34.48.x, seems to connect fine so perhaps the destination host/network just doesn't handle pings well, otherwise I'm not having any trouble with the Internet connection I am using.

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#2406194 24-Jan-2020 18:42

I have a fault reference for this now FWIW if any vodafone staff are following and want to see more details...

INC000002723708

"allow 3 business days for it to be investigated" is the last I heard.

In the mean time, I've spun up an OpenVPN instance in AWS to route my own traffic through. Sigh. Hopefully none of my clients are affected, nobody seems to be shouting at me asking why their sites are so slow and flakey, yet.

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

Geekzone

Want to support Geekzone and browse the site without the ads? Subscribe to Geekzone now (monthly, annual and lifetime options).

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#2406784 25-Jan-2020 16:47

hackaday.com is also being affected by this, as a more "mainstream" website which has nothing to do with the other servers mentioned

So at least that should confirm that it's an issue in vodafone/telstraglobal network.

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#2406822 25-Jan-2020 19:39

Another by random chance of a thread in TM forums found affected: miami.cbslocal.com - as with the others, 6 hops from me and packet loss goes to 90-100%, server effectively unreachable.

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#2406981 26-Jan-2020 11:56

php.net inaccessible with the same massive packet loss from hop 6/7

This is spreading like a cancer.

$ mtr -rc 40 --no-dns `dig +short php.net`
Start: Sun Jan 26 11:54:27 2020
HOST: mortimer Loss% Snt Last Avg Best Wrst StDev
1.|-- 192.168.10.1 0.0% 40 0.5 0.4 0.4 0.5 0.0
2.|-- 202.78.155.1 0.0% 40 12.0 20.1 10.1 53.7 9.2
3.|-- 203.98.50.2 37.5% 40 52.5 26.2 22.3 52.5 6.0
4.|-- 203.98.50.251 0.0% 40 24.1 25.4 21.0 56.8 5.3
5.|-- 203.109.130.41 0.0% 40 25.1 25.6 21.7 41.2 3.3
6.|-- 134.159.174.41 0.0% 40 41.6 42.6 36.9 73.4 6.9
7.|-- 202.84.223.42 95.0% 40 31.4 33.8 31.4 36.3 3.5
8.|-- ??? 100.0 40 0.0 0.0 0.0 0.0 0.0
9.|-- ??? 100.0 40 0.0 0.0 0.0 0.0 0.0
10.|-- ??? 100.0 40 0.0 0.0 0.0 0.0 0.0
11.|-- ??? 100.0 40 0.0 0.0 0.0 0.0 0.0
12.|-- ??? 100.0 40 0.0 0.0 0.0 0.0 0.0
13.|-- 195.66.224.238 97.5% 40 297.5 297.5 297.5 297.5 0.0
14.|-- ??? 100.0 14 0.0 0.0 0.0 0.0 0.0

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

Linux

12181 posts

Uber Geek
+1 received by user: 8475

Trusted

Lifetime subscriber

#2407039 26-Jan-2020 16:07

Seems to be lots of raging on the VodafoneNZ community as well due to quite a few issues @JasonParis

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#2407667 27-Jan-2020 17:15

Somebody else on twitter with almost certainly the same problem, unable to access their wordpress site, confirmed at least for me it suffers the same near-complete packet loss across that hop 6 to 7.

https://twitter.com/AJPonderBooks/status/1221572697244590080

Good to know it's not just me at least.

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

HP

Shop now for HP laptops and other devices (affiliate link).

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#2407852 27-Jan-2020 22:31

Looks like somebody just fixed it in the last hour or so.

Traceroutes are clean with no 100% packet loss any more. Hooray!

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#2407867 27-Jan-2020 23:52

Spoke a bit too soon, hackaday.com and php.net at least still suffering the same fate.

$ mtr -rc 60 --no-dns `dig +short hackaday.com`
Start: Mon Jan 27 23:50:23 2020
HOST: mortimer Loss% Snt Last Avg Best Wrst StDev
1.|-- 192.168.10.1 5.0% 60 0.4 0.5 0.2 1.1 0.0
2.|-- 202.78.155.1 0.0% 60 41.6 43.6 11.3 90.7 17.8
3.|-- 203.98.50.2 0.0% 60 47.0 49.3 23.2 95.3 17.5
4.|-- 203.98.50.251 0.0% 60 52.3 51.4 26.2 88.9 14.6
5.|-- 203.109.130.41 0.0% 60 54.9 48.8 24.6 99.1 14.8
6.|-- 134.159.174.41 0.0% 60 67.3 63.0 36.4 101.9 14.0
7.|-- ??? 100.0 60 0.0 0.0 0.0 0.0 0.0
8.|-- ??? 100.0 60 0.0 0.0 0.0 0.0 0.0
9.|-- ??? 100.0 60 0.0 0.0 0.0 0.0 0.0
10.|-- 202.84.141.25 98.3% 60 183.5 183.5 183.5 183.5 0.0
11.|-- ??? 100.0 60 0.0 0.0 0.0 0.0 0.0
12.|-- ??? 100.0 60 0.0 0.0 0.0 0.0 0.0
13.|-- 213.248.72.106 98.3% 60 204.6 204.6 204.6 204.6 0.0
14.|-- ??? 100.0 23 0.0 0.0 0.0 0.0 0.0

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

Jase2985

13730 posts

Uber Geek
+1 received by user: 6202

ID Verified

Lifetime subscriber

#2407872 28-Jan-2020 05:19

you know some servers de-prioritize ICMP packet when they are busy, hence trace routes and pings tests arent always the best way of testing.

sleemanj

1514 posts

Uber Geek
+1 received by user: 315

#2407884 28-Jan-2020 07:31

Back to square one, all the affected sites are affected again

Jase, that is true but not the case here, look carefully at the diagnoses.

The target servers (websites) are effectively unreachable during these periods, not just icmp, the point at which packets vanish is *link* dependant between a few nodes, not node dependant, the point at which packets disappear is consistently 6 -7 hops out from me at the same place in telstraglobal network. And the sites are myriad and unrelated.

---
James Sleeman
I sell lots of stuff for electronic enthusiasts...

1 | 2