[TelstraClear] TelstraClear cable down 2012-03-23

Forums › One NZ (including Farmside) › [TelstraClear] TelstraClear cable down 2012-03-23

1 | 2 | 3 | 4 | 5

BarTender

3629 posts

Uber Geek
+1 received by user: 2572

ID Verified

Trusted

Lifetime subscriber

#600377 26-Mar-2012 16:46

DonGould: For a company the size of TCL to accept that the TBox is crap, what impact do you think that might have? If you were working for BT and building a business case to buy a new TV box for the BT netowrk, or working for Orange and looking at the TBox v's the one that Cisco just purchased last week, then you read that TBox thread, what's the impact?

I think we're often to introspective when looking at these things and don't consider the global implications.

While 200,000 customers is big here, it's just about nothing in Australia and IS nothing in the UK.

But if you sign off on a product for a customer base of 25 million and there was a TBox thread saying the product is crap then would you choose the Cisco option just because people who buy IBM don't get fired?

My perspective is either

A) TCL haven't spent much on the Digisoft box or Support and only have one developer 20% of their time dedicated to fixing any issues. So it's TCL's fault since they haven't spent enough money on the initial contract or on support.

or

B) Digisoft or Verimatrix CA aren't up to snuff in the Telco grade market. So it's TCLs fault for not doing enough diligence selecting Digisoft/Verimatrix, or they are extremely unresponsive. TCL is very small player in the world with less than 80k total digital TV subs from what I heard a while ago so it's hard to justify a high end expensive box. So if digisoft are a small player one would think they would put all their time and efforts to get it right and use it as a test case to show to the world.

To me it speaks volumes about both TCL and Digisoft. But I don't want to hijack this thread as there is already a long thread talking about this as everything I am saying is purely speculation.

Much as I hate to say it. I've had a MySky for 1 year now. How many times have I rebooted it? Never in 12 months.

DonGould

3892 posts

Uber Geek
+1 received by user: 164

#600388 26-Mar-2012 17:12

BarTender: My perspective is either

....

Much as I hate to say it. I've had a MySky for 1 year now. How many times have I rebooted it? Never in 12 months.

Ya you raise some valid point, but I do wonder if you didn't quite follow mine?

The proxy problem was caused by an issue in the proxy boxes (bluecoat?) The BRAS problem is a problem with the Juniper ERX platform?

While Telstra nor Digi aren't big video companies, Telstra is a reasonable sized Juniper customer aren't they?

But what about Gary or DV? Are they planning on working for Telstra for ever? If on of them is seen to be bagging the bluecoat product and it transpires that problem was simply because on of the Telstra techs just didn't read the doco properly and configure the Juniper box feeding in to it and then they go and apply for a job with Juniper or Bluecoat, then what?

Would you be pleased to hire the guy who bags your crap all day as the guy to talk up your product now?

What chance do you think I've getting a job at Telstra? I've been here for months bashing about the crap way the co is running.

What chance do you think I've for of getting a job at Orcon, Slingshot, Telecom, etc... sure I don't think I've been baging them out to hard, or at all... but they're reading. I've already had one co tell me they feel I'm a bit to hot to touch.

See how this translates to open transparency?

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

freitasm

BDFL - Memuneh
80652 posts

Uber Geek
+1 received by user: 41045

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#600389 26-Mar-2012 17:15

I've received an update from TelstraClear on a couple of questions I put for them:

My questions:

Are you able to clarify if

a) There’s a process in place to notify/alarm when the proxy starts interfering with customer usage?
b) There’s a process to get a customer opt-out of the proxy?

TelstraClear's answers:

We apologise for the delay in informing customers of this issue and are looking at what processes can be improved to identify problems more quickly and so notify customers and, of course, have the issue resolved. DNS and other data errors caused by transparent proxy servers are practically impossible to monitor as the fault is not a hardware issue. Such data errors can also take time to identify and rectify. As noted, we are looking at whether any improvements can be made.

A transparent proxy is used to improve the customer’s browsing experience, as is shown from the results of Epitiro and TrueNet testing. It is not possible for customers to opt out of using the transparent proxy.

Thanks!

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#601490 28-Mar-2012 19:35

BarTender:

In TCLs defence (which I know I don't often do) it's a bit difficult to activly monitor a cache farm.

But at the same time when it was clear it was an issue I would have assumed they bypassed the cache and quickly rectified the current issue and investigated what's gone wrong. Then swap back once it's sorted... But what would I know :)

I don't think it would be too hard to actively monitor a cache farm. Setup a bunch of queries to go through the different nodes of the cache farm and monitor responses, check and alert on unusual responses.

If your game you could even script it to take out the cache farm automatically on failure, but then again that would introduce more failure modes and it would be fun to watch a cache farm flap.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#601508 28-Mar-2012 19:45

DonGould:

I do agree about the business problems you're suggesting though. Do you know how to fix a TCL proxy? I know I don't... I don't even know what they're using. I seem to recall mention of a 'blue coat' or was that Telecom or was it Slingshot, or was it Orcon, I just can't remember who's using what.

From a 'business case' point of view, these suckers are cool... they transfer spend from SCX to a little to the proxy vendor and some to the guys who support them, of which I'm sure that's a very small community in .nz given how few ISP's we have and even less proxy server farms.

As for someone's suggestion that they're just removed from the network when they go wrong... how are they actually in the network in the first place?

Do they work like a VCR? You know, the antenna cable goes in one side and comes out the other, so it's between the antenna on the roof and the TV set?

Is it as simple as the tech just unplugging the cables and just putting a patch lead in to bridge it out?

You would usually have your proxy farm in a certain network and route all international traffic through that route, when the proxy farm failed you would change your routing tables to route international traffic around the proxy farm. This should just be a few lines in a console etc. No physically unplugging things.

a ISP could put their proxy's around the country, but it would cost more as you would need at least 2 for redundancy and the (and correct me if I'm wrong) main cost the ISP's are trying to save by using proxy servers is the international traffic, so a central cluster in your main data centre is fine.

I'm not sure about the BlueCoat's, but a lot of proxys allow you to run at Layer 2 or 3, 3 is IP so just a next hop in the network, Layer two is interesting as the proxy is totally transparent to any TCP/IP device, it just sniffs ethernet packets, decodes them and caches the traffic.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#601509 28-Mar-2012 19:48

DoomlordVekk: There were 2 distinct issues at play here. One was layer 3 connectivity related in Wellington and the other was cache related.
The unfortunate thing was the cache problem reared its head about the same time as the cable issue in Welly.
Both problems produce similar sypmtoms.

So I guess the lesson here isn't to fixate on the more obvious of problems as the answer to the larger issue.

Occam's Razor would suggest that if you were having a BRAS issue, in lieu of contrary information, that would be the cause of the errors that customers are reporting. When that issue resolves itself and the web connectivity problem remains, to a lessor and somewhat murkier extent, then you have to go looking for another answer.

I wouldn't also discount option 3 which could be a proxy failure stemming from the first failure (the cable network outage), perhaps so many clients coming back online flooded the server, or part of the cable outage caused the proxy's to fail as they had a dependency etc.

Whatever it was, it seems TelstraClear are having lots of problems with their cable network in the last year.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

AliExpress

Shop now on AliExpress (affiliate link).

blakamin

4431 posts

Uber Geek
+1 received by user: 1306
Inactive user

#601511 28-Mar-2012 19:51

DonGould:

For a company the size of TCL to accept that the TBox is crap,

I think we're often to introspective when looking at these things and don't consider the global implications.

While 200,000 customers is big here, it's just about nothing in Australia and IS nothing in the UK.

You know that Telstra in Aussie use the TBox too, yeah??

DonGould

3892 posts

Uber Geek
+1 received by user: 164

#601522 28-Mar-2012 19:57

exportgoldman: I don't think it would be too hard to actively monitor a cache farm. Setup a bunch of queries to go through the different nodes of the cache farm and monitor responses, check and alert on unusual responses.

If your game you could even script it to take out the cache farm automatically on failure, but then again that would introduce more failure modes and it would be fun to watch a cache farm flap.

This makes the assumption that the cache farm doesn't log when ever it starts sending 302 errors as well.

For all we know, it just isn't set up and being monitored properly (well in the way we elitist keyboard worriers think it should be done).

The next generation of routers is going to sort this problem out though.

Smarter gear at the edge.

The routers I've got sitting on my desk have a proxy server and layer 7 filtering built in. I can script all sorts of stuff to happen.

If the cache farm starts sending 302s then just jump to a different provider for those addresses, re-request for the user, report to me about the fault, keep checking for life on the LCR and restore if LCR goes live again.

TSticks and mobile data is going to get so dirt cheap that you'll just have a back up stick on every router.

You won't even need to keep the thing in credit, and you also won't pay stupid casual data rates when you use it. The router will just know that it's out of date and order up a Gig for you.

Edit: In the mean time... as I've got 100 routers all sending me emails/txts to day that my providers proxy is throwing 302 errors I'll be on the phone to the providers NOC and posting on NOG to give everyone the heads up.

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

DonGould

3892 posts

Uber Geek
+1 received by user: 164

#601533 28-Mar-2012 20:10

blakamin: You know that Telstra in Aussie use the TBox too, yeah??

Yes. I was discussing it with them recently. They are two different boxes, but I assume that the .nz work is with some vision because iirc our version is differently powered.

They're moving much more traffic to theirs than we are to ours as well as I understand it.

They have some very cool content on theirs and I asked why the company is not leveraging its size to bring us that content as well.

D

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#601535 28-Mar-2012 20:11

exportgoldman:
DoomlordVekk: There were 2 distinct issues at play here. One was layer 3 connectivity related in Wellington and the other was cache related.
The unfortunate thing was the cache problem reared its head about the same time as the cable issue in Welly.
Both problems produce similar sypmtoms.

So I guess the lesson here isn't to fixate on the more obvious of problems as the answer to the larger issue.

Occam's Razor would suggest that if you were having a BRAS issue, in lieu of contrary information, that would be the cause of the errors that customers are reporting. When that issue resolves itself and the web connectivity problem remains, to a lessor and somewhat murkier extent, then you have to go looking for another answer.

I wouldn't also discount option 3 which could be a proxy failure stemming from the first failure (the cable network outage), perhaps so many clients coming back online flooded the server, or part of the cable outage caused the proxy's to fail as they had a dependency etc.

Whatever it was, it seems TelstraClear are having lots of problems with their cable network in the last year.

haha Sorry DoomlordVekk I just realised (thanks Don) who you are and what you do at TelstraClear. I will bow to your much more accurate knowledge of what happened. :)

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

BarTender

3629 posts

Uber Geek
+1 received by user: 2572

ID Verified

Trusted

Lifetime subscriber

#601542 28-Mar-2012 20:28

exportgoldman: I don't think it would be too hard to actively monitor a cache farm. Setup a bunch of queries to go through the different nodes of the cache farm and monitor responses, check and alert on unusual responses.

If your game you could even script it to take out the cache farm automatically on failure, but then again that would introduce more failure modes and it would be fun to watch a cache farm flap.

It's more challenging than you would think with lots of different components in the total solution. Starts with a Layer 7 router at the core which is statefully inspecting all HTTP traffic be it on port 80 or otherwise to ensure it's a http request which would then route only the applicable traffic then via a load balancer sitting in front of the proxy cache farm. Which in turn would route back via another load balancer to the Layer 7 and out to the internet with their source address spoofed. Then the return traffic would be inspected and routed back via the load balancer and then to the farm and back to the Layer 7 and to the client. Then the Cache farm needs to ensure consistency between all nodes and maintain the most recent version of the cached items across all nodes. There are lots of different components in the end to end solution and to monitor only certain bits is really quite difficult and if only one node is playing up makes things more challenging. Especially when you're counting packets per second as a 5 or larger digit number. To monitor all the bits really isn't as easy on the enterprise grade level.
Or maybe I am just giving them too much credit. ;)

This isn't necessarily how it's setup at TCL however I have seen a similar configuration on another T1 telco so it wouldn't surprise me if it was setup this way.

blakamin:
DonGould:
For a company the size of TCL to accept that the TBox is crap,
I think we're often to introspective when looking at these things and don't consider the global implications.
While 200,000 customers is big here, it's just about nothing in Australia and IS nothing in the UK.

You know that Telstra in Aussie use the TBox too, yeah??

The Telstra AU TBox is completely different hardware running DVB-T and streaming Foxtel TV over ADSL. Not a DVB-C with a unheard of CA for crypting the DVB-C ECM feed used in the TelstraClear TBox. The only thing that is similar is the use of the name.

1 | 2 | 3 | 4 | 5