[TelstraClear] TelstraClear cable down 2012-03-23

Forums › One New Zealand (including Vodafone, WxC, Farmside) › [TelstraClear] TelstraClear cable down 2012-03-23

1 | 2 | 3 | 4 | 5

20574 posts

Uber Geek

Trusted

Lifetime subscriber

#599326 23-Mar-2012 23:02

A vpn would work around this sort of problem. I didn't get around to trying astrill yet.

DoomlordVekk

129 posts

Master Geek

Trusted

#599348 24-Mar-2012 00:53

Proxy's really seem like a stupid idea to me. The amount of money wasted on the servers and the resources needed to keep them on track, it just seems like a pointless exercise.

Better to put the money into just buying more SCX capacity aren't they?

D

Regrettably DG, those that track what the extra bandwidth would have actually cost per GB saved/served by the caches will tell you a very compelling business story otherwise.

Yes, a cache can run amock occasionally, as can any piece of kit that goes round.
Most of the time, you're not aware of them and they enhance the general browsing experience, for most people, mostly...
If it can go round, it can invariably go wrong.

"Customers don’t expect you to be perfect. They do expect you to fix things when they go wrong." Donald Porter – British Airways

The views expressed here are my own and are not reflective of other organisms or organisations.

StevieT

702 posts

Ultimate Geek

#599358 24-Mar-2012 07:05

timmmay: If internet access is really that important to someone then they should have two connections, maybe cable with a backup of ADSL or wireless. No service is completely reliable.

Not particularly cost-effective, especially if it sits idle and you're paying for it.

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#599363 24-Mar-2012 08:08

I have no problems with an ISP having a cache/transparent proxy, providing it is "transparent".

The problem is when the thing doesn't set alarm bells ringing somewhere when it goes bad, impacting customers nationwide from four to six hours.

For those who absolutely hate proxies, imagine how much more bandwidth would be necessary and how much slower things would be.

timmmay

20574 posts

Uber Geek

Trusted

Lifetime subscriber

#599364 24-Mar-2012 08:13

StevieT:
timmmay: If internet access is really that important to someone then they should have two connections, maybe cable with a backup of ADSL or wireless. No service is completely reliable.

Not particularly cost-effective, especially if it sits idle and you're paying for it.

No, if it's really important to your work or life then it's probably $50 well spent.

freitasm: I have no problems with an ISP having a cache/transparent proxy, providing it is "transparent".

The problem is when the thing doesn't set alarm bells ringing somewhere when it goes bad, impacting customers nationwide from four to six hours.

For those who absolutely hate proxies, imagine how much more bandwidth would be necessary and how much slower things would be.

If anyone doesn't like transparent proxies use a VPN.

BarTender

3606 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#599376 24-Mar-2012 09:27

freitasm: I have no problems with an ISP having a cache/transparent proxy, providing it is "transparent".

The problem is when the thing doesn't set alarm bells ringing somewhere when it goes bad, impacting customers nationwide from four to six hours.

In TCLs defence (which I know I don't often do) it's a bit difficult to activly monitor a cache farm.

But at the same time when it was clear it was an issue I would have assumed they bypassed the cache and quickly rectified the current issue and investigated what's gone wrong. Then swap back once it's sorted... But what would I know :)

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#599382 24-Mar-2012 09:45

Sure, but "alarms" also involve other inputs - in this case there were reports on Geekzone, Twitter, Facebook, and I am sure some people even endured the 40 - 45 minutes wait to report problems.

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

DoomlordVekk

129 posts

Master Geek

Trusted

#599554 24-Mar-2012 19:07

Mauricio, lets be honest here, a a reasonable percentage of the time, 'Alarms' from Geekzone users are more about venting angst and frustration, than about explaing an issue in a level of detail that is useful for troubleshooting.

Social media is one thing for disseminating ideas between quasi-interesting groups of loosely affiliated people, it's a completely unuseful tool for imparting detailed thinking between affected parties.

Too often, repeating inaccurate or even worse, fundamentally wrong info via a retweet or reposting someone' grump renders the tool just 'more noise on the wire'. Yes GZ is good for a number of things but concise, impartial and targeted info parsing into a technical organisation for 'assistance' isn't one of those uses sorry.

Offence is not an intention as a result of this post, just lets keep it real.

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#599557 24-Mar-2012 19:22

In this context "alarms" is not something definite like a NOC alarm wired directly to a box. But if you start seeing a lot of people, in different locations (not in the same node), all complaining about "http err 302" or "connection reset by peer" then it's an indication something is going on...

It's a lot different than some people saying "my dsl connection drops every now and then..."

This is the same argument that TelstraClear help desk seems to use. When someone calls them and say "I have cable service and it's down in Churton Park, is it just me or more people?" the help desk says "we don't track this so I don't know"... Well this is poor, because if suddenly you have a lot of people calling to complain of something then it would be fair to raise some flags.

I disagree with your sentiment.

DoomlordVekk

129 posts

Master Geek

Trusted

#599564 24-Mar-2012 19:55

I do agree with your last statement, an upsurge in call volume to the front-of-house teams should and often does ring bells.
Likewise, good monitoring is also key to making sure you have a good handle on whats going on in your network.
All too often however, the more complex aspects of software give rise to complex failure modes.
Yes, box is still forwarding traffic, fans functional, all lights blinking.
Hmmm, you say some requests are failing. Which ones?
Right, take that cache out of the farm, it's doing strange stuff.

Humans are analog, like moving coil meters, they take time to react. Esp. one hand letting the other hand know what it's up to or that "crap, thats hot! Don't touch it!"

DonGould

3892 posts

Uber Geek

#599586 24-Mar-2012 21:04

I agree with aspects of what both DV and MF are saying... but the question is what can ISPs do to provide a better experience for users?

Personally I think Geoff Huston outlined the answer over a decade ago with a paper he wrote about the smarts moving to the network edge. That process isn't complete yet. The power of edge technology is still a generation away in my view.

Fortunately for net users, as Bevan Slattery commented half a generation ago, an internet generation is only 18 months.

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

insane

3236 posts

Uber Geek

ID Verified

Trusted

#599623 24-Mar-2012 22:35

DonGould: I agree with aspects of what both DV and MF are saying... but the question is what can ISPs do to provide a better experience for users?

In our industry it's part of the game that things break, and you can't always engineer around that, so hopefully their NOC/SOC teams will only get questioned around their monitoring systems, assuming they were not negligent.

From reading this thread it sounds like the problem was that it took a while for information to reach someone who could actually do something about it, for that the Business as a whole needs to address that process.

DonGould

3892 posts

Uber Geek

#599655 25-Mar-2012 00:45

insane:

In our industry it's part of the game that things break, and you can't always engineer around that, so hopefully their NOC/SOC teams will only get questioned around their monitoring systems, assuming they were not negligent.

From reading this thread it sounds like the problem was that it took a while for information to reach someone who could actually do something about it, for that the Business as a whole needs to address that process.

Your comment almost seems at odds with what DV said. DV's comments suggested to me that no one actually knew it was a problem, just a few GZ users here were reporting issues, but the NOC hadn't confirmed the proxy was sending 301's.

I do agree about the business problems you're suggesting though. Do you know how to fix a TCL proxy? I know I don't... I don't even know what they're using. I seem to recall mention of a 'blue coat' or was that Telecom or was it Slingshot, or was it Orcon, I just can't remember who's using what.

From a 'business case' point of view, these suckers are cool... they transfer spend from SCX to a little to the proxy vendor and some to the guys who support them, of which I'm sure that's a very small community in .nz given how few ISP's we have and even less proxy server farms.

As for someone's suggestion that they're just removed from the network when they go wrong... how are they actually in the network in the first place?

Do they work like a VCR? You know, the antenna cable goes in one side and comes out the other, so it's between the antenna on the roof and the TV set?

Is it as simple as the tech just unplugging the cables and just putting a patch lead in to bridge it out?

I've read mention of 'proxy farms'... how does that work? Is there one for each part of the country? People were talking about problems in Jvil to start with, then petone then Christchurch... did the problem migrate to the different proxy's in the farm?

All of a sudden, as I write this, I start to really understand DV's comment above about users not actually being any help at all. I like to think I've got a few clues, more than some and less than others, but I confess, I know so little about TCL's set up that I'm sure most of what I've written above will be just reading like a joke to the TCL tech's who might be reading this - perhaps that's my point?

D

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

DoomlordVekk

129 posts

Master Geek

Trusted

#599667 25-Mar-2012 08:19

There were 2 distinct issues at play here. One was layer 3 connectivity related in Wellington and the other was cache related.
The unfortunate thing was the cache problem reared its head about the same time as the cable issue in Welly.
Both problems produce similar sypmtoms.

So I guess the lesson here isn't to fixate on the more obvious of problems as the answer to the larger issue.

Occam's Razor would suggest that if you were having a BRAS issue, in lieu of contrary information, that would be the cause of the errors that customers are reporting. When that issue resolves itself and the web connectivity problem remains, to a lessor and somewhat murkier extent, then you have to go looking for another answer.

StevieT

702 posts

Ultimate Geek

#599784 25-Mar-2012 16:54

I honestly know next-to-nothing about what happens - when I have issues - on TelstraClear's side. Is there a networking book that might be of some use to me?

If perhaps someone from TC enlightens us on how their system works, perhaps we might be able to suggest various things that would signal alarm bells (for them) when something does go wrong?

1 | 2 | 3 | 4 | 5