Telecom Equipment Upgrade Affects Wholesale Customers

Forums › New Zealand Broadband › Telecom Equipment Upgrade Affects Wholesale Customers

1 | 2

Fraktul

836 posts

Ultimate Geek

Trusted

#87069 18-Sep-2007 18:57

KiwiOverseas66:
Fraktul:
It is not unheard of for a customer to be provisioned from the outset tunneled to the incorrect ISP. First time we came across this quite some time back there was a bit of head scratching going on.

whoa! that's a neat trick. If the link was useable, I wonder how long you could use it before anyone noticed?

Well you still need the correct login details so that rather puts a dampener on things.

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#87073 18-Sep-2007 19:09

Depending on how it is setup, the tunnel could be authenticated against the correct ISP's RADIUS Server, but then tunneled to the wrong ISP.

This would then show up as not being able to SEND e-mail, as they are outside of the correct IP range for the ISP, but depending on the ISP they would still be able to POP mail off the ISP server (from a different network.)

I've heard of this happening before.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

PenultimateHop

637 posts

Ultimate Geek
+1 received by user: 2

Trusted

#87074 18-Sep-2007 19:14

exportgoldman:

Depending on how it is setup, the tunnel could be authenticated against the correct ISP's RADIUS Server, but then tunneled to the wrong ISP.

No - this is not how UBS works, and it can't happen.

There is no "tunnel authentication" with an ISP RADIUS server - the tunnel is brought up by Telecom equipment, prior to any authentication happening at the ISP end. If the subscriber is tunneled to the wrong ISP, that's it - and unless your credentials happen to match on the incorrect ISP, you aren't going to get authenticated.

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#87078 18-Sep-2007 19:20

PenultimateHop:
exportgoldman:

Depending on how it is setup, the tunnel could be authenticated against the correct ISP's RADIUS Server, but then tunneled to the wrong ISP.
No - this is not how UBS works, and it can't happen.

There is no "tunnel authentication" with an ISP RADIUS server - the tunnel is brought up by Telecom equipment, prior to any authentication happening at the ISP end. If the subscriber is tunneled to the wrong ISP, that's it - and unless your credentials happen to match on the incorrect ISP, you aren't going to get authenticated.

Happened to quite a few people in Christchurch, an IT company wrote about it (I think here on geekzone) and they mentioned the same symptoms about e-mail etc.

It fits all the symptoms. Do you KNOW this cannot ever happen, or are you guessing? :-) Because you seem pretty sure that this isn't a possible failure mode.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

PenultimateHop

637 posts

Ultimate Geek
+1 received by user: 2

Trusted

#87079 18-Sep-2007 19:23

KiwiOverseas66:
sniff, sniff, sniff......that would be a rat I smell! rscole86 is correct I think. It sounds like what the CSR was referring to was authentication relay. Authentication would be done by Slingshot, but the information would be relayed via a telecom DSLAM at the exchange. This is how they operate their corporate remote access services. Its possible that a change to the telecom device might "erase" the relay account and stop the connection from being authenticated by slingshot - but it would be the whole connection that would stop working. Initially Oyajipunk's slingshot access had been authenticated and was working, as was his mail service - it was just smtp access the was initially inoperable - and this would have nothing to do with Telecom since once the connection is up and running their part is over (mail is just another service on the connection authenticated and run entirely by slingshot). It was the following day the whole connection stopped running requiring a rebuild of the account. So I wonder if what really happened was...

This is not quite how UBS, or Remote Office functionality for OneOffice VPNs.

When a subscriber for UBS is provisioned, the line identity of the subscriber (which is a tuple of the subscriber atm vp/vc and BRAS identifier) is used by Telecom's RADIUS proxies as a lookup key. This lookup key returns the QoS profile (256K, 2M, 7M) and the tunnel termination details.

Once the tunnel session is established, the ISP's LNS takes over PPP, which includes credential authentication via their own RADIUS (or TACACS or Diameter) platform.

I suspect it's probably somewhat of a red herring that mail stopped working prior to the line misconfiguration, assuming it was that, during the ASAM/ISAM migration.

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#87080 18-Sep-2007 19:28

http://www.geekzone.co.nz/forums.asp?ForumId=49&TopicId=10800&page_no=2

Heres the article which shows very detailed information from the IT company in christchurch with traceroutes showing their connection is being routed via the wrong ISP.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

Apple TV

Stream your favourite shows now on Apple TV (affiliate link).

PenultimateHop

637 posts

Ultimate Geek
+1 received by user: 2

Trusted

#87081 18-Sep-2007 19:30

exportgoldman: Happened to quite a few people in Christchurch, an IT company wrote about it (I think here on geekzone) and they mentioned the same symptoms about e-mail etc.

It fits all the symptoms. Do you KNOW this cannot ever happen, or are you guessing? :-) Because you seem pretty sure that this isn't a possible failure mode.

On Unbundled Bitstream Service (UBS) it is not possible. On WBS or the other legacy FastIP Direct products, it [i]was[/i] possible for a similar situation to occur, but it had nothing to do with tunnels.

When, say, an Xnet subscriber is tunneled to Orcon by mistake, Xnet has no visibility of it. Only Orcon will see the subscriber authentication attempts and will reject them - unless by some coincidence the credentials happen to match, or Orcon have a captive portal for subscribers using the wrong credentials, etc.

For the problem you describe to occur on WBS/FastIP Direct, it was generally a case of:

1. The ISP not returning a Framed-IP-Address attribute for the subscriber which resulted in the Telecom BRAS choosing a random named pool to assign an IP address out of; OR
2. The ISPs dynamic pool (for, e.g. Jetstart) was full, and the BRAS erroneously chose another named pool to assign an IP address out of.

Either of these scenarios would have resulted in the subscriber getting an IP address that may not be permitted to relay through their ISP's SMTP server, as it was not a part of their allocated FIP IP range.

Note that WBS and FastIP Direct are entirely different from UBS and the failure scenarios for the two are quite different.

PenultimateHop

637 posts

Ultimate Geek
+1 received by user: 2

Trusted

#87082 18-Sep-2007 19:32

exportgoldman:

http://www.geekzone.co.nz/forums.asp?ForumId=49&TopicId=10800&page_no=2

Heres the article which shows very detailed information from the IT company in christchurch with traceroutes showing their connection is being routed via the wrong ISP.

That post (after a quick skim) describes a fault on a WBS/FastIP service, which is very different from UBS. The failure modes are not even remotely comparable.

KiwiOverseas66

173 posts

Master Geek
Inactive user

#87086 18-Sep-2007 20:17

PenultimateHop:
This is not quite how UBS, or Remote Office functionality for OneOffice VPNs.

When a subscriber for UBS is provisioned, the line identity of the subscriber (which is a tuple of the subscriber atm vp/vc and BRAS identifier) is used by Telecom's RADIUS proxies as a lookup key. This lookup key returns the QoS profile (256K, 2M, 7M) and the tunnel termination details.

Once the tunnel session is established, the ISP's LNS takes over PPP, which includes credential authentication via their own RADIUS (or TACACS or Diameter) platform.

I suspect it's probably somewhat of a red herring that mail stopped working prior to the line misconfiguration, assuming it was that, during the ASAM/ISAM migration.

ah...gotcha. Makes sense - if I understand you correctly still identified/ managed initially by telecom radius - but without the need for masses of ISP provided details. Very clever. Presumably though the remote service still allows for radius relay? I seem to remember Telecom providing some very specific direction on what kind of server could relay to their radius (merit AAA, steel belted, that sort of thing). Sorry - very off topic.

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#87095 18-Sep-2007 21:22

PenultimateHop:
exportgoldman:

http://www.geekzone.co.nz/forums.asp?ForumId=49&TopicId=10800&page_no=2

Heres the article which shows very detailed information from the IT company in christchurch with traceroutes showing their connection is being routed via the wrong ISP.

That post (after a quick skim) describes a fault on a WBS/FastIP service, which is very different from UBS. The failure modes are not even remotely comparable.

You know what your talking about. I believe you :-) Very good explanation

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

Oyajipunk

28 posts

Geek

#87227 19-Sep-2007 19:22

Well, I'm glad we got that sorted.

Geekzone support

Support Geekzone with one-off or recurring donations Donate via PressPatron.

KiwiOverseas66

173 posts

Master Geek
Inactive user

#87256 19-Sep-2007 21:00

Oyajipunk: Well, I'm glad we got that sorted.

well I for one do apologies for going off topic on this one (especially since your problem had already been solved), but its always good to get the inside word from someone (penultimatehop) who clearly knows the kit, the network, and the services. I would guess he/ she works for either Telecom or Alcatel in either network deployment or the IP config team - hence its a rare opportunity to get the inside word from someone who actually works with this stuff (as opposed to the usual journo or CSR fudd). Thanks for bringing up the subject Oyajipunk - and thanks Penultimatehop for providing some real information. Much appreciated

PenultimateHop

637 posts

Ultimate Geek
+1 received by user: 2

Trusted

#87289 20-Sep-2007 02:35

KiwiOverseas66: ah...gotcha. Makes sense - if I understand you correctly still identified/ managed initially by telecom radius - but without the need for masses of ISP provided details. Very clever. Presumably though the remote service still allows for radius relay? I seem to remember Telecom providing some very specific direction on what kind of server could relay to their radius (merit AAA, steel belted, that sort of thing). Sorry - very off topic.

UBS doesn't need any sort of RADIUS relaying. There are two very discrete transactions that happen:

1. Telecom BRAS (L2TP LAC) to Telecom's wholesale RADIUS environment. This looks up the atm vp/vc (or, in the case of ISAM - forgot to mention this in the previous post - the ethernet c-tag and s-tag) and BRAS ID from the RADIUS Access-Request. The result of this in the subscriber database includes the QoS profile and wholesaler details. An Access-Response is created and sent to the BRAS, which authorizes the subscriber, and starts the L2TP tunnel.

2. [If necessary, PPP proxy authentication is handled here] The ISP L2TP LNS re-starts PPP authentication, or uses PPP proxy auth from the BRAS. The LNS will use RADIUS (or TACACS or whatever the ISP's choice of AAA environment is) to authorise the subscriber based on credentials the ISP requires for the product set. Typically this is User-name and User-password...

In the other wholesale products, such as the IPNet dial platform or WBS/FastIP, then there is use of RADIUS proxy authentication through the Wholesale RADIUS platform to the ISP. Telecom may have in a very old handbook described the RADIUS servers recommended, but anything which complies with RFC2138 RADIUS will work, or preferred RFC2865/2866 RADIUS.

No problem providing real information. It helps people stop chasing their tails.

The UBS User Guide available at http://www.telecom.co.nz/binarys/ubs_user_guide_october_2006_comp.pdf might provide some interesting reading, in particular Appendix XI on Page 29; Page 32 describes the flow of establishing a PPP session over UBS.

KiwiOverseas66

173 posts

Master Geek
Inactive user

#87301 20-Sep-2007 08:01

PenultimateHop: UBS doesn't need any sort of RADIUS relaying

Yep...got that point. You'll see in my post that I was actually asking about Telecom's remote access service (use to be branded IPremote - then One Office remote - then Office Anywhere). Some of the larger corporate customers had it - and liked it too! I've been out of the country for about a year now and its all changed fairly dramatically.

In the other wholesale products, such as the IPNet dial platform or WBS/FastIP, then there is use of RADIUS proxy authentication through the Wholesale RADIUS platform to the ISP. Telecom may have in a very old handbook described the RADIUS servers recommended, but anything which complies with RFC2138 RADIUS will work, or preferred RFC2865/2866 RADIUS.

hey...I'm not that old

. Actually it was only 3-4 years ago and I still have the user guide (I guess prior to whatever group you're currently associated with)? It was after microsoft bundled a freebie radius app with one of its server packages that problems occurred. Turned out the app was missing some key definition files.

No problem providing real information. It helps people stop chasing their tails.

ah but that's half the fun!

The UBS User Guide available at http://www.telecom.co.nz/binarys/ubs_user_guide_october_2006_comp.pdf might provide some interesting reading, in particular Appendix XI on Page 29; Page 32 describes the flow of establishing a PPP session over UBS.

great stuff - cheers!

Up4It

11 posts

Geek

#87890 24-Sep-2007 13:32

I had an unexplained outage at home on Actrix ADSL the other week, and I notice we've got a notice from telecom of planned work here at work in the first week fo next month as part of the ASAM to ISAM migration.

1 | 2