Trustpower UFB CGNAT to Spark UFB in same town (strange routes)

Forums › New Zealand Broadband › Trustpower UFB CGNAT to Spark UFB in same town (strange routes)

kiwirock

685 posts

Ultimate Geek

#230626 5-Mar-2018 20:30

I'm just curious if anyone else on Trustpower that does a trace route to a Spark IP go through Sydney?

Same result if I just do a trace to 125.236.192.9 which is my next hop from my Spark UFB connection to Spark.

And tracing back the other way back to the CGNAT at Trustpower the last reply back I get is from a Vodafone router:

Tracing route to 150.107.173.92 over a maximum of 30 hops

1 1 ms 1 ms 1 ms [172.24.12.254]
2 2 ms 1 ms 1 ms [172.24.24.24]
3 3 ms 3 ms 15 ms 125-236-192-9.adsl.xtra.co.nz [125.236.192.9]
4 * * * Request timed out.
5 28 ms 27 ms 27 ms ae8-20.akcr11.global-gateway.net.nz [122.56.116.9]
6 26 ms 27 ms 27 ms 203.98.18.65
7 27 ms 27 ms 28 ms 203.97.59.122
8 * * * Request timed out.
9 * * * Request timed out. 10 * * * and so on....

If I trace from Spark to HOP 7 on the Trustpower to Spark trace I get:

1 1 ms 1 ms 1 ms [172.24.12.254]
2 2 ms 2 ms 2 ms [172.24.24.24]
3 16 ms 5 ms 7 ms 125-236-192-9.adsl.xtra.co.nz [125.236.192.9]
4 * 26 ms 27 ms mdr-ip24-int.msc.global-gateway.net.nz [122.56.116.6]
5 33 ms 29 ms 28 ms ae8-10.akbr6.global-gateway.net.nz [122.56.116.5]
6 27 ms 27 ms 27 ms ae7-2.akbr7.global-gateway.net.nz [122.56.119.53]
7 53 ms 50 ms 52 ms xe7-0-2.sgbr3.global-gateway.net.nz [202.50.232.10]
8 51 ms 52 ms 51 ms ae2-10.sgbr4.global-gateway.net.nz [202.50.232.246]
9 55 ms 53 ms 51 ms pacnet.sgbr4.global-gateway.net.nz [202.50.237.26]
10 50 ms 51 ms 52 ms ge-0-0-6.gw2.syd5.10026.telstraglobal.net [202.147.14.205]
11 52 ms 52 ms 51 ms gi2-0-0-900.gw1.syd2.10026.telstraglobal.net [202.147.55.90]
12 55 ms 50 ms 51 ms as4648.nsw.ix.asn.au [218.100.52.136]

That's an awfully long way for a packet to get 2.5KM's across town in one direction. Granted, it gets there none the less.

Sounddude

I fix stuff!
1928 posts

Uber Geek

Trusted

2degrees

Lifetime subscriber

#1968480 5-Mar-2018 20:33

Spark change ISP's to pass traffic from other ISP's.

Example

Spark charge the UFB customer, as well as trust power for the trustpower customer to talk to the spark customer over the internet.

Trustpower probably got sick of paying and have routed the traffic over a free path via AU.

(Vocus are looking at doing the same in the near future).

sbiddle

30853 posts

Uber Geek

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#1968483 5-Mar-2018 20:40

Spark's peering (or lack of it as some may see it as) hasn't changed for a number of years. Rather than even interconnecting this traffic at a single point (say Auckland) which still requires payment they're presumably choosing to interconnect at presumably the cheapest point which would seem to be Sydney. This may seem strange to people who don't understand the way the Internet works, but it certainly isn't the first and won't be he last time things like this happen.

yitz

2082 posts

Uber Geek

#1968489 5-Mar-2018 20:54

It seems multiple ISPs have chosen to stop purchasing domestic transit routes into Spark and Vodafone in the last month or two.

Has Spark Local Peering also officially ended?

Sounddude

I fix stuff!
1928 posts

Uber Geek

Trusted

2degrees

Lifetime subscriber

#1968490 5-Mar-2018 20:56

yitz: . Has Spark Local Peering also officially ended?

Being turned off 1st April.

kyhwana2

2566 posts

Uber Geek

#1968495 5-Mar-2018 21:03

Spark doesn't peer locally with any other ISPs and they havn't really done so in the past, so it's not surprising you're seeing your packets go via an IX in Sydney.

(Blame Spark for most of these issues)

kiwirock

685 posts

Ultimate Geek

#1968496 5-Mar-2018 21:03

sbiddle:

Spark's peering (or lack of it as some may see it as) hasn't changed for a number of years. Rather than even interconnecting this traffic at a single point (say Auckland) which still requires payment they're presumably choosing to interconnect at presumably the cheapest point which would seem to be Sydney. This may seem strange to people who don't understand the way the Internet works, but it certainly isn't the first and won't be he last time things like this happen.

Yeah the nature of $ and c, and interconnections everywhere.

It just surprises me, that it's cheaper to go via an International link. There's more equipment and infrastructure along the way.

Obviously it costs to much for infrastructure here compared to the larger pipes at discount offshore. Or someone's over-inflating what they are worth?

kiwirock

685 posts

Ultimate Geek

#1968498 5-Mar-2018 21:07

I thought some of this might have settled down over the years gone by since the first major spat between Telecom and Telstra bean counters years ago. I guess Telecom is still Telecom, or Spark if you will.

How on earth does it get cheaper though to introduce a middle man and more cabling.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

kiwirock

685 posts

Ultimate Geek

#1968499 5-Mar-2018 21:11

Sounddude:

yitz: . Has Spark Local Peering also officially ended?

Being turned off 1st April.

I see.

Sounddude

I fix stuff!
1928 posts

Uber Geek

Trusted

2degrees

Lifetime subscriber

#1968500 5-Mar-2018 21:11

Its almost 10 times cheaper to go via AU then it is across the city to sparks network.

Its crazy.

michaelmurfy

meow
13265 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1968553 5-Mar-2018 21:55

Spark really need to get their act together here eh @bartender...

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

nathanward

5 posts

Wannabe Geek

#1968615 5-Mar-2018 23:47

kiwirock:

I'm just curious if anyone else on Trustpower that does a trace route to a Spark IP go through Sydney?

Same result if I just do a trace to 125.236.192.9 which is my next hop from my Spark UFB connection to Spark.

And tracing back the other way back to the CGNAT at Trustpower the last reply back I get is from a Vodafone router:

It's not CGN specific, thanks for posting about it. 25ms isn't really a huge problem for most applications, but still, not really ideal. (you should have just had a 25ms hit to your actual host rather than some of the intermediary 50ms hits, I think, because of the way traceroute works in MPLS networks etc.)

It looks like Spark turned up some peering on Megaport and NSWIX that I don't think was there when we last did a round of traffic engineering optimisation, and we were now preferring those routes for various reasons to do with moving things around, etc. etc. Networks are complicated :)

Should be fixed now - I can't actually ping the host you gave over either path, but it seems to be working better anyway. Let me know if not, and post some more traceroutes. Copy/pasted text is best if you can, screenshots of traceroutes make me cry :'(

nathanward

5 posts

Wannabe Geek

#1968618 5-Mar-2018 23:55

Lots of conjecture and opinions in here in other posts about Spark local peering and all sorts - really, it just comes down to a sub-optimal BGP policy, that is now ever so slightly closer to optimal.

The local peering they're turning off something they offered so you could pick up traffic for their customers locally in regions around NZ, and only customers from that region. I'm aware of one, maybe two networks who took advantage of it, everyone else stuck with paid domestic transit, which is very close to free if you have any negotiating bones in your body. Means traffic doesn't follow a theoretical ideal of staying "local", but for most networks that isn't a problem either. The in-country path between Spark and Trustpower doesn't go through those local peering links, it may have in the past, but I don't believe it has for a number of years.

Anyway OP thanks for the heads up, and I hope that's fixed for you :)

kiwirock

685 posts

Ultimate Geek

#1968625 6-Mar-2018 00:44

Thanks Nathan. I was more curious why it was going off shore.

However, grateful none the less for the change, and perhaps a few less bits and watts for the trouble.

Also got around to looking up the syntax for a Mikrotik terminal traceroute output to include dns on the Trustpower end:

[admin@MikroTik] /tool> traceroute *XXXXXX* use-dns yes (very orginal ;) )
# ADDRESS LOSS SENT LAST AVG BEST WORST
1 100.65.0.3 0% 2 25.8ms 25.8 25.8 25.8
2 10.55.85.101 0% 1 28.2ms 28.2 28.2 28.2
3 ip-103-241-56-4.kinect.net.nz 0% 1 26.4ms 26.4 26.4 26.4
4 10.55.85.49 0% 1 28.4ms 28.4 28.4 28.4
5 10.55.85.182 0% 1 27.8ms 27.8 27.8 27.8
6 10.55.85.38 0% 1 31.9ms 31.9 31.9 31.9
7 10.55.85.45 0% 1 27.7ms 27.7 27.7 27.7
8 203.97.59.122 0% 1 27.5ms 27.5 27.5 27.5
9 203.97.59.121 0% 1 28.1ms 28.1 28.1 28.1
10 203.98.18.69 0% 1 28ms 28 28 28
11 203.98.18.70 0% 1 28.4ms 28.4 28.4 28.4
12 ae9-44.akcr11.global-gateway.... 0% 1 31.7ms 31.7 31.7 31.7
13 mdr-ip24-dom.msc.global-gatew... 0% 1 32.1ms 32.1 32.1 32.1
14 222-154-224-9.adsl.xtra.co.nz 0% 1 53ms 53 53 53
15 222-XXX-236-XXX.adsl.xtra.co.nz 0% 1 54.7ms 54.7 54.7 54.7

Yuck, next time I'll figure out how to paste the pretty formatting.

A straight ping is the same. I have seen instances where trace routes are long than pings though.

yitz

2082 posts

Uber Geek

#1968626 6-Mar-2018 00:48

Nice, perhaps other smaller ISPs are seeing the same then.

What a pain having to deal with separate international and domestic BGP tables, Spark and Vodafone should re-engineer networks to be simpler and at the same time completely peer at the peering exchange of the day.

nathanward

5 posts

Wannabe Geek

#1968629 6-Mar-2018 01:01

yitz: Nice, perhaps other smaller ISPs are seeing the same then.

What a pain having to deal with separate international and domestic BGP tables, Spark and Vodafone should re-engineer networks to be simpler and at the same time completely peer at the peering exchange of the day.

Networks support business goals, not the other way around. They’ll do what’s right for their businesses. NZ isn’t magical or unique in this sense anymore, all countries have different networking environments with their own challenges because of the markets and so on. It’s why routers have so many knobs :)