Office 365 errors with users on Telecom New Zealand

Forums › IT Pro and developers › Office 365 errors with users on Telecom New Zealand

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

635 posts

Ultimate Geek
Inactive user

#741876 8-Jan-2013 19:45

blur:
bigal_nz: And I can open the Singtel website from behind a Vodafone connection.

Some sort of blocking of Xtra by singtel?

From what I understand Vodafone broadband customers who use Office365 are not affected by this issue so I am not surprised that you can open the SingTel website.

That was my point. I was confirming the theory its limited to Xtra customers.

Wise

Send money globally for less with Wise - one free transfer up to NZ$900 (affiliate link).

bigal_nz

635 posts

Ultimate Geek
Inactive user

#741882 8-Jan-2013 19:53

blur: I have received another email from Microsoft's Customer Support team. I have to say that it is not encouraging as they are only NOW asking me to perform some tests. I wonder if they are having trouble tracking down the issue or convincing someone at SingTel that there is in fact a problem? Has anybody else received one similar? This was received at 6:50pm this evening:

Dear Mr. XXXX,

This is Andrew from Microsoft Office 365 Technical Support.

I am writing this e-mail because our Operations Team would like us to gather some information from the affected computers and also computers that aren't experiencing the issue. We would need the following:
1. Run 'tracert mail.office365.com' on an affected PC and capture the results.
2. Use Network Monitor (http://www.microsoft.com/en-us/download/details.aspx?id=4865) to capture a tracing log that demonstrates the issue. Capture approximately 5 mins of log.
3. Perform steps 1 and 2 again on a healthy PC that isn't encountering the issue.
4. Gather MOSDAL (http://support.microsoft.com/kb/960625), ExRCA for Autodiscover (https://www.testexchangeconnectivity.com/), gather a list of affected users, and get the customer's server information (XXXXXXXXXX.outlook.com in OWA's URL).
5. What ISP is the customer using?
6. Can the issue be reproduced on another ISP?

I will follow up with you tomorrow afternoon to see if you were able to gather the information requested from our Operations Team so I can forward that over to them. If you have any questions, please give us a call at Toll-free: 0800 194197, or Local: 09 9127298

Bugger.

Having said that I have spoken to the MS Cloud Manager in NZ. He is aware and states they are working on it together with Xtra.

Perhaps the guy from Xtra watching this thread can confirm?

I sure hope MS sort this out real quick. What happend to redundant servers? Single point of failure????

blur

384 posts

Ultimate Geek

#741886 8-Jan-2013 19:57

Yep, I was considering calling Microsoft this afternoon and asking for our mailboxes to be moved from the Singapore server farm to the US as a workaround.

What concerns me the most about his email to me is that he will call me tomorrow afternoon to check, by which time another entire day will have passed.

I sincerly hope that I am wrong, but I am gutted that this issue just doesn't seem to be getting the priority that is needed.

My HTPC - Case Antec Fusion Remote, MOBO Intel DH67BLB3, CPU Intel Core i5-2400S 2.5 GHz, RAM 8GB DDR3 1333, HDD 120Gig Corsair Force Series 3 SSD system | WD Caviar Black 2TB data, Tuners Black Gold BGT3595 dual DVB-S/S2, dual DVB-T, Video nVIDIA GeForce GT 520, 1024MB, Sound Intel® High Definition Audio (onboard), OS Windows 7 x64

plambrechtsen

1948 posts

Uber Geek
Inactive user

#741892 8-Jan-2013 20:13

Kellysu: Hi plambrechtsen / all

What kind of diagnostics could help? TRACERTS from affected sites to affected pods? We have a number of clients affected and they are all connected via Xtra - its quite frustraiting that a problem like this can take so long to resolve.

A similar issue happened about 6 months ago (July 2012) and the problem was actually resolved by Microsoft - it appeared to be a routing issue, symptoms identical, and the final explanation from MS was vague - but it was one of their edge servers.

What may be helpful for others is to change the DNS servers from the xtra DNS servers to google DNS (8.8.8.8) - this allows users to get on to OWA which is better that nothing, as google will resolve different FE servers.

in the meantime - problem still apparent.

Strangely enough the guy I spoke with at global gateway said the same thing about the outage 6 months ago and the challenges he had finding someone in Microsoft to own the issue.

That's why I'm recommending to everyone to log a case with Microsoft and then use that to escalate to CTS in Telecom.

I'm 99% sure its a problem on Microsoft's end but happy to be corrected if I am wrong.

RalphFromSnap

776 posts

Ultimate Geek

Trusted

Snap Internet

#741895 8-Jan-2013 20:17

Hi All,

Traceroutes can be very helpful in diagnosing faults like this, but what you really need is a traceroute back from Microsoft to your IP Address, Singtel have a looking glass here (unfortunately they don’t have any BGP information) but here is some interesting info which seems to indicate that there is no return path back to Telecom from Singtel for some of their networks (example the network www.telecom.co.nz sits on)

If you put your client’s IP Address into the below tool and do a traceroute this could be helpful for their faults desk.

http://www.stixlg.singtel.com/open/public/lgmain/lgquery/#

Orchard, SG Router traceroute to to www.telecom.co.nz:

traceroute www.telecom.co.nz

Translating "www.telecom.co.nz"...domain server (165.21.100.88)

Type escape sequence to abort.

Tracing the route to www.telecom.co.nz (210.55.11.13)

VRF info: (vrf in name/id, vrf out name/id)

1 203.208.150.37 0 msec 0 msec 0 msec

2 203.208.153.202 0 msec

3 203.208.151.253 32 msec

4 ge-2-0-0-0.nycec-ar1.ix.singtel.com (203.208.151.218) 32 msec

5 203.208.155.179 32 msec 32 msec

6 * * *

Traceroute incomplete due to query timeout

Traceroute to a random XT IP Address, which goes to LA -> San Jose -> NZ (but does get there and www.singtel.com is accessible)

traceroute 222.153.223.1

Type escape sequence to abort.

Tracing the route to 222-153-223-1.mobile.telecom.co.nz (222.153.223.1)

VRF info: (vrf in name/id, vrf out name/id)

1 203.208.150.37 0 msec 0 msec 0 msec

2 203.208.173.185 0 msec 0 msec 0 msec

3 203.208.152.194 4 msec 4 msec 0 msec

4 ae-2.r21.newthk02.hk.bb.gin.ntt.net (129.250.3.10) [AS 2914] [MPLS: Label 301952 Exp 0] 4 msec 0 msec 0 msec

5 p64-5-0-0.r23.tokyjp01.jp.bb.gin.ntt.net (129.250.2.120) [AS 2914] [MPLS: Label 382690 Exp 0] 52 msec 52 msec 52 msec

6 as-2.r05.sydnau01.au.bb.gin.ntt.net (129.250.3.238) [AS 2914] 176 msec 176 msec 176 msec

7 xe-0-1-0.a06.sydnau01.au.ra.gin.ntt.net (202.68.64.21) [AS 2914] 176 msec 172 msec 176 msec

8 202.68.67.158 [AS 2914] 152 msec 144 msec 148 msec

9 ae0-10.sgbr1.global-gateway.net.nz (202.50.232.245) 152 msec 144 msec 148 msec

10 xe3-0-0.akbr4.global-gateway.net.nz (202.50.232.9) [MPLS: Label 361520 Exp 0] 172 msec 168 msec 172 msec

11 ae5-2.akbr5.global-gateway.net.nz (202.50.232.78) 176 msec 172 msec 180 msec

12 * * *

Versus a traceroute to a Snap IP

Orchard, SG Router traceroute to 202.37.101.1:

traceroute 202.37.101.1

Type escape sequence to abort.

Tracing the route to bender.snap.net.nz (202.37.101.1)

1 203.208.155.2 0 msec 4 msec 0 msec

2 203.208.171.169 4 msec 0 msec 4 msec

3 203.208.153.245 0 msec 4 msec 0 msec

4 203.208.166.173 [MPLS: Label 429536 Exp 0] 4 msec 4 msec 0 msec

5 so-5-2-3-0.toknf-cr3.ix.singtel.com (203.208.154.18) 72 msec 72 msec 72 msec

6 i-0-4-0-0.siko02.bi.telstraglobal.net (202.84.219.46) 152 msec 152 msec 149 msec

7 i-9-0-0.siko-core02.bi.telstraglobal.net (202.84.219.45) 144 msec 148 msec 144 msec

8 i-0-6-0-2.sydo-core01.bx.telstraglobal.net (202.84.140.137) 161 msec 168 msec 169 msec

9 i-0-3-2-0.sydo-core02.bi.telstraglobal.net (202.84.220.189) 176 msec 169 msec 176 msec

10 i-2-2-1.hptw-core01.bx.telstraglobal.net (202.84.142.150) 192 msec 196 msec 193 msec

11 unknown.telstraglobal.net (134.159.174.42) 200 msec 175 msec 200 msec

12 xe-0-3-0-1037.mdr.core.snap.net.nz (111.69.27.66) 197 msec 200 msec 201 msec

So it would seem to indicate a problem where Singtel cant see/is not accepting some of Telecom/Global Gateways networks into their routing tables.

Snap

0800 BROADBAND (276 232)
www.snap.net.nz

@SnapInternet on Twitter
Snap Internet on Facebook

Our Social Media Team:
^RO Ricky - Technical Lead
^AT Ashleigh - Retail Marketing Coordinator

plambrechtsen

1948 posts

Uber Geek
Inactive user

#741898 8-Jan-2013 20:24

bigal_nz: Bugger.

Having said that I have spoken to the MS Cloud Manager in NZ. He is aware and states they are working on it together with Xtra.

Perhaps the guy from Xtra watching this thread can confirm?

I sure hope MS sort this out real quick. What happend to redundant servers? Single point of failure????

Who's the MS Cloud manager in NZ?

If you have any email trails or contact of people in NZ or better yet a contact email and number for the Microsoft NOC as I am sure its a routing screwup on MS's end so the global gateway and SingTel and Microsoft NOC guys can sit down and sort it out.

Feel free to email me pl at telecom dot co dot nz and I will see if I can connect the right people.

freitasm

BDFL - Memuneh
79060 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#741903 8-Jan-2013 20:34

I agree with Peter that Microsoft should "own" this problem, or at least show they're putting some effort into it.

Regs

4064 posts

Uber Geek

Trusted

Snowflake

#741908 8-Jan-2013 20:42

i'm now able to connect my Exchange Management Console, connect a remote powershell and kick off on-premises -> cloud mailbox migrations - something I haven't been able to do all yesterday or today. I'm hoping that means that something has been fixed....

Sales Engineer
Snowflake
www.snowflake.com

about.me/nzregs
Twitter: @nzregs

Regs

4064 posts

Uber Geek

Trusted

Snowflake

#741919 8-Jan-2013 20:51

blur: Yep, I was considering calling Microsoft this afternoon and asking for our mailboxes to be moved from the Singapore server farm to the US as a workaround.

What concerns me the most about his email to me is that he will call me tomorrow afternoon to check, by which time another entire day will have passed.

I sincerly hope that I am wrong, but I am gutted that this issue just doesn't seem to be getting the priority that is needed.

you cant just 'move mailboxes from Singapore to US'... its a little more complex than that. The datacentres are separate.

The Microsoft NZ staff are not directly involved with running the global network/routing, nor the office 365 server farms, so all they can really do is try and escalate internally. when they get a number of NZ customers with the same problem, especially from multiple ISPs, they can make a bigger noise and are more likely to be 'heard' when trying to escalate.

if there is a single customer with an issue and Microsoft Ops teams are not sitting on a pile of other service requests - then its going to be hard to convince them that there is a 'major' issue. For all they know, it could be a problem isolated to a change on a customers firewall... or an ISP who messed up their routing tables...

Sales Engineer
Snowflake
www.snowflake.com

about.me/nzregs
Twitter: @nzregs

blur

384 posts

Ultimate Geek

#741926 8-Jan-2013 21:08

Regs:
blur: Yep, I was considering calling Microsoft this afternoon and asking for our mailboxes to be moved from the Singapore server farm to the US as a workaround.

What concerns me the most about his email to me is that he will call me tomorrow afternoon to check, by which time another entire day will have passed.

I sincerly hope that I am wrong, but I am gutted that this issue just doesn't seem to be getting the priority that is needed.

you cant just 'move mailboxes from Singapore to US'... its a little more complex than that. The datacentres are separate.

The Microsoft NZ staff are not directly involved with running the global network/routing, nor the office 365 server farms, so all they can really do is try and escalate internally. when they get a number of NZ customers with the same problem, especially from multiple ISPs, they can make a bigger noise and are more likely to be 'heard' when trying to escalate.

if there is a single customer with an issue and Microsoft Ops teams are not sitting on a pile of other service requests - then its going to be hard to convince them that there is a 'major' issue. For all they know, it could be a problem isolated to a change on a customers firewall... or an ISP who messed up their routing tables...

Talking to the support guys in the US, they explained that it is not uncommon for you to be moved around servers every 3 - 7 days for load balancing reasons. This is why the autodiscover entries in the DNS are so critical.

blur

384 posts

Ultimate Geek

#741927 8-Jan-2013 21:10

freitasm: I agree with Peter that Microsoft should "own" this problem, or at least show they're putting some effort into it.

Here here!

My contract is with Microsoft, and Microsoft's contract is with SingTel.

It falls squarely at Microsoft's feet in my view.

Ryrager

25 posts

Geek

#741928 8-Jan-2013 21:12

Customer that been having issues last 2 days is now working. anyone else seeing an improvement?

blur

384 posts

Ultimate Geek

#741932 8-Jan-2013 21:15

Just tested - seems to be working here too...

Regs

4064 posts

Uber Geek

Trusted

Snowflake

#741957 8-Jan-2013 22:03

blur:
Talking to the support guys in the US, they explained that it is not uncommon for you to be moved around servers every 3 - 7 days for load balancing reasons. This is why the autodiscover entries in the DNS are so critical.

you'll be moved around servers in the Singapore/Hong Kong datacenter, but not from APAC to US, as far as i know...

Sales Engineer
Snowflake
www.snowflake.com

about.me/nzregs
Twitter: @nzregs

kidona

8 posts

Wannabe Geek

#741958 8-Jan-2013 22:04

Testing fine here as well. Both my Office365 mailboxes connected to exchange and synching ok. Hoping it stays this way tomorrow morning.

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9