[TelstraClear] TelstraClear cable down 2012-03-23

Forums › One New Zealand (including Vodafone, WxC, Farmside) › [TelstraClear] TelstraClear cable down 2012-03-23

1 | 2 | 3 | 4 | 5

702 posts

Ultimate Geek

#599786 25-Mar-2012 16:58

DoomlordVekk: Mauricio, lets be honest here, a a reasonable percentage of the time, 'Alarms' from Geekzone users are more about venting angst and frustration, than about explaing an issue in a level of detail that is useful for troubleshooting.

Social media is one thing for disseminating ideas between quasi-interesting groups of loosely affiliated people, it's a completely unuseful tool for imparting detailed thinking between affected parties.

Too often, repeating inaccurate or even worse, fundamentally wrong info via a retweet or reposting someone' grump renders the tool just 'more noise on the wire'. Yes GZ is good for a number of things but concise, impartial and targeted info parsing into a technical organisation for 'assistance' isn't one of those uses sorry.

Offence is not an intention as a result of this post, just lets keep it real.

I don't know how much more specific -I- can be when I have issues. I can say x, y, and z don't load, and I get "..." message, I have this brand of router. The lights on my modem/router seem all in accord with normality.

In the future, what other, technical, details should I include?

insane

3236 posts

Uber Geek

ID Verified

Trusted

#599841 25-Mar-2012 20:14

StevieT:

In the future, what other, technical, details should I include?

Given that you're purchasing a commodity service you should simply have to call them and describe what you were doing, and what error you noticed. The person on the other end of the phone should prompt you for more information based on what you're describing as needed.

If you choose to raise your issue via email then annotated screen shots may be seen as a good start.

StevieT: I honestly know next-to-nothing about what happens - when I have issues - on TelstraClear's side. Is there a networking book that might be of some use to me?

If perhaps someone from TC enlightens us on how their system works, perhaps we might be able to suggest various things that would signal alarm bells (for them) when something does go wrong?

Sadly no 'networking for dummies' book is going to help you understand how their network works better than their Network Engineers. I will give you a head start though and suggest it's all a series of tubes.

DonGould

3892 posts

Uber Geek

#599959 25-Mar-2012 23:38

StevieT: The lights on my modem/router seem all in accord with normality.

In the future, what other, technical, details should I include?

Stevie, in the case of the two faults DV's talking about the lights aren't going to tell you anything at all.

The BRAS fault is just crap they'll have to sort, to be honest I didn't even bother to reread exactly what DV even wrote past it being a BRAS issue.

You seem to be a guy who doesn't know that much about this stuff yet, but is quite keen to know more just for fun, the same way anyone likes to know stuff... :) ... in this case you should look up BRAS in wikipedia and have a read about what that thing actually is. There's stuff all you can really do to help if a BRAS has an issue, but it is interesting to understand more about how the 1's and 0's get to you.

The proxy problem is quite interesting though. My proxy server was reporting 302 errors to my wife because she had her Firefox set to use our prosy server, which was receiving the fault codes and presenting them to her in an error message rather than just letting the browser do it.

What a 30x error is, who cares, I don't. What is interesting is that web browsers used to tell you this stuff but now seem to just display a blank page with nothing on them and no error code, at least that's what I was getting in FF and Chrome.

This is not new stuff. We've fixed on problem, by taking away the error messages, but haven't actually pushed a solution in yet. The HD phones still light up and the users are still unhappy.

I confess, I see 30x errors so little these days that I didn't even customise our proxy server error pages so I knew what was going on and it wasn't until I saw the GZ thread that I realised the error pages where coming from our proxy and not TCL.

DV, I tend to agree... tell us what the problems are as they unfold and we all take up time speculating that the issue is the fault and "slam TCL's crap systems" without doing basic checks, don't tell us and then we get pissed when we've wasted a hour doing basic checks only to find the company had a collusion of unusual events. I get it, you can't win either way.

Personally I like fault information in the public space in as close to real time as possible because if I'd read the info you posted about the BRAS at the time you were attempting to fix it, then I would have helped translate to users such as Stevie... ...and said 'don't even bother calling the HD..." Personally I think this is what the NOG list should be for, but .nz just doesn't seem to work that way, we're all so precious about this crap.

But I also wonder how much longer we'll see you posting before you get burnt off as well... I know GZ used to have a really good guy posting about TCL stuff and someone commented to me a while back that he got toasted and roasted by the locals so we got relegated back to the PR guys... less than helpful (and yes, for anyone else who's still reading this far down... clearly my comments aren't directed really at DV but at us all as a warning that if we want go intel we need to stop shooting the tech guys! ..and yes, I know I'm as bad as everyone else and need to concentrate on taking my own advice, but feel free to point it out if you must.)

D

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

DoomlordVekk

129 posts

Master Geek

Trusted

#599995 26-Mar-2012 06:57

Thanks DG :)

It's a tough thing sometimes. I mentioned the thread as it started to Gary, expecting he might pop in and let people know something had happened and recovery was under way. I am not a media or PR chappie, so I don't get to talk to the world or GZ about the issues at large.
Not that i have been actively 'discouraged' yet either but I have to walk a careful line.

Yes, I think TelstraClear should talk to its customers as regularly as it can, esp. when a situation pops up that is intrusive. We all hate not having the info a hand when we're trying to determine what our problems are at home.

SteveT, all I can tell you is to use the Customer Help Services. They are paid to do what they can to collect info, do basic troubleshooting (Yes, they will sugesst rebooting cable modems, ADSl Modems and any other CPE they can find ;) )

Do some basic troubleshooting. Almost all systems come with tools like ping, traceroute and dig(nslookup in the windows world).
Can I ping to my local gateway, can I get to the next hop after that etc. Work out where the edge of the problem is.
As a TCL user I find 203.97.33.1 (dns1.clear.net.nz) is always a useful address to remember, as you can check that you can ping and traceroute to it. That tells me that I have layer 3 network connectivity to my ISP.

Once you have determined that the problem isn't a layer 3 reachability issue (through ping and traceroute testing), then look at name resolution.

One of the most intrusive issues is DNS playing up.
In a web browser, it can almost look like a network connectivity problem and it is. Most of the time we forget that a computer system doesn't understand www.TradeMe.co.nz as a IP address to go to. DNS has to give your pc 202.162.73.2 before it can start handshaking to deliver you the page.

By the time you've done your layer 3 testing and DNS checking, if the problem still exists, then you have to sit down and do some compartmentalisation and comparison testing.

Is it all webpages in NZ that have the problem? Is it all webpages outside of NZ with the problem?
All ISP's live ( and die) based on what their upstream(s) do for them. The hop-by-hop routing paradigm will have it no other way.

So, do as much as you can, so you know, in your owh head, where and what the problem is. It takes some time but generally it makes you feel a lot better if you can work out that TCL's upstream Reach has a routing issue. Now getting THAT fixed isn't easy, even for us but it makes us happier in our own heads that we've identified the problem, realise it's out of our hands and to take the dog for a walk or go play catch with the kids in the back yard.

"Customers don’t expect you to be perfect. They do expect you to fix things when they go wrong." Donald Porter – British Airways

The views expressed here are my own and are not reflective of other organisms or organisations.

StevieT

702 posts

Ultimate Geek

#599996 26-Mar-2012 07:08

DoomlordVekk:

SteveT, all I can tell you is to use the Customer Help Services. They are paid to do what they can to collect info, do basic troubleshooting (Yes, they will sugesst rebooting cable modems, ADSl Modems and any other CPE they can find ;) )

I guess the thing that makes me more inclined not to is the wait times, especially on the phone.

DoomlordVekk

129 posts

Master Geek

Trusted

#600011 26-Mar-2012 08:31

Perseverance is a quality valued in most species.

If it's any consolation, and it's probably not, I've written a personal email to the head of our Customer Help team, expressing my concern at being on the end of a phone for 25 min and then getting bounced out of the system due to overloading

When something happens, the front of house teams are generally going to take a hammering.

One of the ongoing challenges in any large, decentralised organisation is to get the right info to the people at the sharp end, helping the customers.

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#600017 26-Mar-2012 08:39

DonGould: The proxy problem is quite interesting though. My proxy server was reporting 302 errors to my wife because she had her Firefox set to use our prosy server, which was receiving the fault codes and presenting them to her in an error message rather than just letting the browser do it.

What a 30x error is, who cares, I don't. What is interesting is that web browsers used to tell you this stuff but now seem to just display a blank page with nothing on them and no error code, at least that's what I was getting in FF and Chrome.

I confess, I see 30x errors so little these days that I didn't even customise our proxy server error pages so I knew what was going on and it wasn't until I saw the GZ thread that I realised the error pages where coming from our proxy and not TCL.

"it wasn't until I saw the GZ thread that I realised the error pages where coming from our proxy and not TCL. "

Wait, what? Haven't you read the discussion up to here? The TelstraClear proxy had problems. The 302 errors were reported north and south, everywhere. One of the problems were not in your proxy, it was on TelstraClear's side.

Did you completely miss this?

DonGould: Personally I like fault information in the public space in as close to real time as possible because if I'd read the info you posted about the BRAS at the time you were attempting to fix it, then I would have helped translate to users such as Stevie... ...and said 'don't even bother calling the HD..."

That's the thing. There was no fault information when the proxy thing started happening, with at least four to five hours before TelstraClear admitted, on Twitter only, that something was going on.

DonGould: Personally I think this is what the NOG list should be for, but .nz just doesn't seem to work that way, we're all so precious about this crap.

Only tech people and some other very few interested subscribe to the NOG list, because as the name says it's then Network Operator Group. "Open information" should be open, the NOG is not - anyone can join, but it's not "interesting".

TelstraClear should have posted at least in their own status page that something was going on.

DonGould: But I also wonder how much longer we'll see you posting before you get burnt off as well... I know GZ used to have a really good guy posting about TCL stuff and someone commented to me a while back that he got toasted and roasted by the locals so we got relegated back to the PR guys...

The TelstraClear account on Geekzone is one of the oldest telco account here. They had created it for communication with the people here. We had probably about two people from TelstraClear who posted here as well, but it seems it's the company policy to have a central account.

If by "the locals" in toasted and roasted you means Geekzone then you are wrong.

DoomlordVekk: So, do as much as you can, so you know, in your owh head, where and what the problem is. It takes some time but generally it makes you feel a lot better if you can work out that TCL's upstream Reach has a routing issue. Now getting THAT fixed isn't easy, even for us but it makes us happier in our own heads that we've identified the problem, realise it's out of our hands and to take the dog for a walk or go play catch with the kids in the back yard.

And that's the thing... When a lot of people start getting 302 and posting here, Twitter, Facebook, then people know it's not only them.

That's the beauty of the thing. If one customer has a problem and no else, then Occam's razor (yes, the same principle) says it's logical this is a problem on his configuration or physical connection. If the customer has a problem and s/he comes to Geekzone and see people from all over the place saying they are experiencing the same thing, then it's with TelstraClear.

To most customers it won't matter if the problem is your upstream provider, or the cache farm. The problem is with TelstraClear and it's TelstraClear who needs to inform customers "we heard you, we are investigating". Or "we heard you, we investigated, and we are working on fixing it."

Silence is your worst enemy, people will keep talking, and getting louder.

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

DoomlordVekk

129 posts

Master Geek

Trusted

#600028 26-Mar-2012 09:03

"When the eagles fall silent the parrots begin to jabber"...??

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#600029 26-Mar-2012 09:06

Except the saying has nothing to do with this... Being "silent" in this context means keeping customers in the dark. Not a good move.

BarTender

3606 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#600030 26-Mar-2012 09:06

freitasm: And that's the thing... When a lot of people start getting 302 and posting here, Twitter, Facebook, then people know it's not only them.

That's the beauty of the thing. If one customer has a problem and no else, then Occam's razor (yes, the same principle) says it's logical this is a problem on his configuration or physical connection. If the customer has a problem and s/he comes to Geekzone and see people from all over the place saying they are experiencing the same thing, then it's with TelstraClear.

To most customers it won't matter if the problem is your upstream provider, or the cache farm. The problem is with TelstraClear and it's TelstraClear who needs to inform customers "we heard you, we are investigating". Or "we heard you, we investigated, and we are working on fixing it."

Silence is your worst enemy, people will keep talking, and getting louder.

I can only agree with you here Mauricio. Just look at how long it took to get an formal acceptance about the TBox issue. And that is now a few weeks away from their 1 year anniversary.

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#600033 26-Mar-2012 09:09

BarTender:
freitasm: Silence is your worst enemy, people will keep talking, and getting louder.

I can only agree with you here Mauricio. Just look at how long it took to get an formal acceptance about the TBox issue. And that is now a few weeks away from their 1 year anniversary.

Companies should go out and read "The Cluetrain Manifesto" and then they will understand what "people will talk" means.

"Markets are conversations" is the single most important thing companies should understand when dealing with customers.

BarTender

3606 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#600036 26-Mar-2012 09:15

freitasm: Companies should go out and read "The Cluetrain Manifesto" and then they will understand what "people will talk" means.

"Markets are conversations" is the single most important thing companies should understand when dealing with customers.

And the internet never forgets. That and "Googling" or perhaps a more generic term of searching for people with common interests / complaints / views helps to bring a much wider (yet perhaps not as physically motivated to get up off their seats and do something about it) audience together.

DonGould

3892 posts

Uber Geek

#600218 26-Mar-2012 13:13

DoomlordVekk: Perseverance is a quality valued in most species.

If it's any consolation, and it's probably not, I've written a personal email to the head of our Customer Help team, expressing my concern at being on the end of a phone for 25 min and then getting bounced out of the system due to overloading

Good luck with that ;)

DoomlordVekk: When something happens, the front of house teams are generally going to take a hammering.

One of the ongoing challenges in any large, decentralised organisation is to get the right info to the people at the sharp end, helping the customers.

To me, this is why Twitter and the NOG list is so useful. I personally also find the GZ IRC channel really useful. I've liked IRC channels for years, but they can be tricky as well. In .au I saw a co who make great use of IRC until competitors started just trolling it to discredit the company which is what the PR guys are always frightened of I know.

I get the whole problem with Twitter and Face book, which are very much consumer level interfaces. If you post "we've got a lot of errors login off card 1c in the erx on the southern bras' then people are just going to drown you with endless questions to explain what the hell a BRAS card is in the WRX and why the errors aren't showing on their web browser and should they use Firefox rather than Thunderbird or Outlook to view them... and should they have a Honda rather than Suburu.

In my mind, this is why you do use IRC, GZ and NOG though. I'm always happy to try and explain things a bit more clearly for others who want to understand but are still getting their head around the difference between layer 7 applications.

D

Edit: fixed some spelling/typos and made the BRAS error example a bit more obvious.

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

DonGould

3892 posts

Uber Geek

#600234 26-Mar-2012 13:25

freitasm: Did you completely miss this?

No I got it, but I didn't explain myself very well sorry.

We run a bunch of proxy servers here for different reasons. My computer wasn't using any of them, my wifes was.

She asked me about a site problem. For me, my computer just showed me a blank page in the web browser.

For her, she got a customizable error message from one of our proxy servers because our proxy server isn't transparent. It didn't just pass a 302 error to her browser. It presented a very valid HTML page with detailed information about the 302 error it had got from Telstra's 'transparent' proxy server.

In days gone by, web browsers used to present you with an error page with the same sort of info. Today it would seem that they don't. At least that was my experience. I recall reading postings on a mailing list a decade ago about this issue and people complaining about IE not presenting the actual error message but a dumbed down version. Now I see the messages have gone from what ever browser I was using.

It I was smart and could be bothered, I could direct our proxy server to send a log message to our monitoring systems to tell me that one of the 'users' was getting 302 errors from our provider.

Having said that, if I was even smarter, then I suspect I could run a layer 7 filter on our boarder router to just switch all traffic for the effected sites to our back up link and send a message to our monitoring system to start a process to keep monitoring the primary link for a RTS..... but as I said earlier, I see so few 302 errors these days that I didn't even recognise the fault for what it was.

D

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

DonGould

3892 posts

Uber Geek

#600351 26-Mar-2012 16:15

BarTender: I can only agree with you here Mauricio. Just look at how long it took to get an formal acceptance about the TBox issue. And that is now a few weeks away from their 1 year anniversary.

BT that TBox thing is really interesting on many different business levels I think.

If I was to come out and say that iPad's are crap then just about no one would care and some of you would laugh. If MF was to say it then there might be some discussion... if the CEO of that company in China that actually makes them was to say they're crap then it's likely to impact the stock value of both Apple and that company.

For a company the size of TCL to accept that the TBox is crap, what impact do you think that might have? If you were working for BT and building a business case to buy a new TV box for the BT netowrk, or working for Orange and looking at the TBox v's the one that Cisco just purchased last week, then you read that TBox thread, what's the impact?

I think we're often to introspective when looking at these things and don't consider the global implications.

While 200,000 customers is big here, it's just about nothing in Australia and IS nothing in the UK.

But if you sign off on a product for a customer base of 25 million and there was a TBox thread saying the product is crap then would you choose the Cisco option just because people who buy IBM don't get fired?

D

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

1 | 2 | 3 | 4 | 5