Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.

Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
613 posts

Ultimate Geek
+1 received by user: 24

Trusted

  Reply # 741876 8-Jan-2013 19:45 Send private message

blur:
bigal_nz: And I can open the Singtel website from behind a Vodafone connection.

Some sort of blocking of Xtra by singtel?


From what I understand Vodafone broadband customers who use Office365 are not affected by this issue so I am not surprised that you can open the SingTel website.


That was my point. I was confirming the theory its limited to Xtra customers.

613 posts

Ultimate Geek
+1 received by user: 24

Trusted

  Reply # 741882 8-Jan-2013 19:53 Send private message

blur: I have received another email from Microsoft's Customer Support team. I have to say that it is not encouraging as they are only NOW asking me to perform some tests. I wonder if they are having trouble tracking down the issue or convincing someone at SingTel that there is in fact a problem? Has anybody else received one similar? This was received at 6:50pm this evening:


Dear Mr. XXXX,

This is Andrew from Microsoft Office 365 Technical Support.

I am writing this e-mail because our Operations Team would like us to gather some information from the affected computers and also computers that aren't experiencing the issue. We would need the following:
1. Run 'tracert mail.office365.com' on an affected PC and capture the results.
2. Use Network Monitor (http://www.microsoft.com/en-us/download/details.aspx?id=4865) to capture a tracing log that demonstrates the issue. Capture approximately 5 mins of log.
3. Perform steps 1 and 2 again on a healthy PC that isn't encountering the issue.
4. Gather MOSDAL (http://support.microsoft.com/kb/960625), ExRCA for Autodiscover (https://www.testexchangeconnectivity.com/), gather a list of affected users, and get the customer's server information (XXXXXXXXXX.outlook.com in OWA's URL).
5. What ISP is the customer using?
6. Can the issue be reproduced on another ISP?

I will follow up with you tomorrow afternoon to see if you were able to gather the information requested from our Operations Team so I can forward that over to them. If you have any questions, please give us a call at Toll-free: 0800 194197, or Local: 09 9127298


Bugger.

Having said that I have spoken to the MS Cloud Manager in NZ. He is aware and states they are working on it together with Xtra.

Perhaps the guy from Xtra watching this thread can confirm?

I sure hope MS sort this out real quick. What happend to redundant servers? Single point of failure????

335 posts

Ultimate Geek


  Reply # 741886 8-Jan-2013 19:57 Send private message

Yep, I was considering calling Microsoft this afternoon and asking for our mailboxes to be moved from the Singapore server farm to the US as a workaround.

What concerns me the most about his email to me is that he will call me tomorrow afternoon to check, by which time another entire day will have passed.

I sincerly hope that I am wrong, but I am gutted that this issue just doesn't seem to be getting the priority that is needed.




My HTPC - Case Antec Fusion Remote, MOBO Intel DH67BLB3, CPU Intel Core i5-2400S 2.5 GHz, RAM 8GB  DDR3 1333, HDD 120Gig Corsair Force Series 3 SSD system | WD Caviar Black 2TB data, Tuners Black Gold BGT3595 dual DVB-S/S2, dual DVB-T, Video nVIDIA GeForce GT 520, 1024MB, Sound Intel® High Definition Audio (onboard), OS Windows 7 x64

1743 posts

Uber Geek
+1 received by user: 337

Trusted
Spark NZ

  Reply # 741892 8-Jan-2013 20:13 Send private message

Kellysu: Hi plambrechtsen / all

What kind of diagnostics could help? TRACERTS from affected sites to affected pods?  We have a number of clients affected and they are all connected via Xtra - its quite frustraiting that a problem like this can take so long to resolve. 

A similar issue happened about 6 months ago (July 2012) and the problem was actually resolved by Microsoft - it appeared to be a routing issue, symptoms identical, and the final explanation from MS was vague - but it was one of their edge servers.

What may be helpful for others is to change the DNS servers from the xtra DNS servers to google DNS (8.8.8.8) - this allows users to get on to OWA which is better that nothing, as google will resolve different FE servers.

in the meantime - problem still apparent.


Strangely enough the guy I spoke with at global gateway said the same thing about the outage 6 months ago and the challenges he had finding someone in Microsoft to own the issue.

That's why I'm recommending to everyone to log a case with Microsoft and then use that to escalate to CTS in Telecom.

I'm 99% sure its a problem on Microsoft's end but happy to be corrected if I am wrong.




I work for Spark, but as always my views are my own.

694 posts

Ultimate Geek
+1 received by user: 140

Trusted
Snap Internet

  Reply # 741895 8-Jan-2013 20:17 Send private message

Hi All,

Traceroutes can be very helpful in diagnosing faults like this, but what you really need is a traceroute back from Microsoft to your IP Address, Singtel have a looking glass here (unfortunately they don’t have any BGP information) but here is some interesting info which seems to indicate that there is no return path back to Telecom from Singtel for some of their networks (example the network www.telecom.co.nz sits on)

 If you put your client’s IP Address into the below tool and do a traceroute this could be helpful for their faults desk.

 http://www.stixlg.singtel.com/open/public/lgmain/lgquery/#

 Orchard, SG Router traceroute to to www.telecom.co.nz:

traceroute www.telecom.co.nz

Translating "www.telecom.co.nz"...domain server (165.21.100.88)

 

Type escape sequence to abort.

Tracing the route to www.telecom.co.nz (210.55.11.13)

VRF info: (vrf in name/id, vrf out name/id)

1 203.208.150.37 0 msec 0 msec 0 msec

2 203.208.153.202 0 msec

3 203.208.151.253 32 msec

4 ge-2-0-0-0.nycec-ar1.ix.singtel.com (203.208.151.218) 32 msec

5 203.208.155.179 32 msec 32 msec

6 * * *

 

Traceroute incomplete due to query timeout

 

 Traceroute to a random XT IP Address, which goes to LA -> San Jose -> NZ (but does get there and www.singtel.com is accessible)

 

traceroute 222.153.223.1

Type escape sequence to abort.

Tracing the route to 222-153-223-1.mobile.telecom.co.nz (222.153.223.1)

VRF info: (vrf in name/id, vrf out name/id)

1 203.208.150.37 0 msec 0 msec 0 msec

2 203.208.173.185 0 msec 0 msec 0 msec

3 203.208.152.194 4 msec 4 msec 0 msec

4 ae-2.r21.newthk02.hk.bb.gin.ntt.net (129.250.3.10) [AS 2914] [MPLS: Label 301952 Exp 0] 4 msec 0 msec 0 msec

5 p64-5-0-0.r23.tokyjp01.jp.bb.gin.ntt.net (129.250.2.120) [AS 2914] [MPLS: Label 382690 Exp 0] 52 msec 52 msec 52 msec

6 as-2.r05.sydnau01.au.bb.gin.ntt.net (129.250.3.238) [AS 2914] 176 msec 176 msec 176 msec

7 xe-0-1-0.a06.sydnau01.au.ra.gin.ntt.net (202.68.64.21) [AS 2914] 176 msec 172 msec 176 msec

8 202.68.67.158 [AS 2914] 152 msec 144 msec 148 msec

9 ae0-10.sgbr1.global-gateway.net.nz (202.50.232.245) 152 msec 144 msec 148 msec

10 xe3-0-0.akbr4.global-gateway.net.nz (202.50.232.9) [MPLS: Label 361520 Exp 0] 172 msec 168 msec 172 msec

11 ae5-2.akbr5.global-gateway.net.nz (202.50.232.78) 176 msec 172 msec 180 msec

12 * * *

 

Versus a traceroute to a Snap IP

 

Orchard, SG Router traceroute to 202.37.101.1:

 

traceroute 202.37.101.1

 

Type escape sequence to abort.

Tracing the route to bender.snap.net.nz (202.37.101.1)

 

1 203.208.155.2 0 msec 4 msec 0 msec

2 203.208.171.169 4 msec 0 msec 4 msec

3 203.208.153.245 0 msec 4 msec 0 msec

4 203.208.166.173 [MPLS: Label 429536 Exp 0] 4 msec 4 msec 0 msec

5 so-5-2-3-0.toknf-cr3.ix.singtel.com (203.208.154.18) 72 msec 72 msec 72 msec

6 i-0-4-0-0.siko02.bi.telstraglobal.net (202.84.219.46) 152 msec 152 msec 149 msec

7 i-9-0-0.siko-core02.bi.telstraglobal.net (202.84.219.45) 144 msec 148 msec 144 msec

8 i-0-6-0-2.sydo-core01.bx.telstraglobal.net (202.84.140.137) 161 msec 168 msec 169 msec

9 i-0-3-2-0.sydo-core02.bi.telstraglobal.net (202.84.220.189) 176 msec 169 msec 176 msec

10 i-2-2-1.hptw-core01.bx.telstraglobal.net (202.84.142.150) 192 msec 196 msec 193 msec

11 unknown.telstraglobal.net (134.159.174.42) 200 msec 175 msec 200 msec

12 xe-0-3-0-1037.mdr.core.snap.net.nz (111.69.27.66) 197 msec 200 msec 201 msec

 

 So it would seem to indicate a problem where Singtel cant see/is not accepting some of Telecom/Global Gateways networks into their routing tables.

 

 




Snap

0800 BROADBAND (276 232)
www.snap.net.nz

@SnapInternet on Twitter
Snap Internet on Facebook

Our Social Media Team:
^MC Michelle - Technical Lead
^RO Ricky - Technical Lead
^AT Ashleigh - Retail Marketing Coordinator



1743 posts

Uber Geek
+1 received by user: 337

Trusted
Spark NZ

  Reply # 741898 8-Jan-2013 20:24 Send private message

bigal_nz: Bugger.

Having said that I have spoken to the MS Cloud Manager in NZ. He is aware and states they are working on it together with Xtra.

Perhaps the guy from Xtra watching this thread can confirm?

I sure hope MS sort this out real quick. What happend to redundant servers? Single point of failure????


Who's the MS Cloud manager in NZ?

If you have any email trails or contact of people in NZ or better yet a contact email and number for the Microsoft NOC as I am sure its a routing screwup on MS's end so the global gateway and SingTel and Microsoft NOC guys can sit down and sort it out.

Feel free to email me pl at telecom dot co dot nz and I will see if I can connect the right people.




I work for Spark, but as always my views are my own.

BDFL
49914 posts

Uber Geek
+1 received by user: 4621

Administrator
Trusted
Geekzone
Subscriber

  Reply # 741903 8-Jan-2013 20:34 Send private message

I agree with Peter that Microsoft should "own" this problem, or at least show they're putting some effort into it.




Infrastructure Geek
3702 posts

Uber Geek
+1 received by user: 87

Trusted
Microsoft NZ
Subscriber

  Reply # 741908 8-Jan-2013 20:42 Send private message

i'm now able to connect my Exchange Management Console, connect a remote powershell and kick off on-premises -> cloud mailbox migrations - something I haven't been able to do all yesterday or today. I'm hoping that means that something has been fixed....




Technical Evangelist
Microsoft NZ
about.me/nzregs
Twitter: @nzregs


Infrastructure Geek
3702 posts

Uber Geek
+1 received by user: 87

Trusted
Microsoft NZ
Subscriber

  Reply # 741919 8-Jan-2013 20:51 Send private message

blur: Yep, I was considering calling Microsoft this afternoon and asking for our mailboxes to be moved from the Singapore server farm to the US as a workaround.

What concerns me the most about his email to me is that he will call me tomorrow afternoon to check, by which time another entire day will have passed.

I sincerly hope that I am wrong, but I am gutted that this issue just doesn't seem to be getting the priority that is needed.


you cant just 'move mailboxes from Singapore to US'... its a little more complex than that.  The datacentres are separate.

The Microsoft NZ staff are not directly involved with running the global network/routing, nor the office 365 server farms, so all they can really do is try and escalate internally.  when they get a number of NZ customers with the same problem, especially from multiple ISPs, they can make a bigger noise and are more likely to be 'heard' when trying to escalate. 

if there is a single customer with an issue and Microsoft Ops teams are not sitting on a pile of other service requests - then its going to be hard to convince them that there is a 'major' issue.  For all they know, it could be a problem isolated to a change on a customers firewall... or an ISP who messed up their routing tables...





Technical Evangelist
Microsoft NZ
about.me/nzregs
Twitter: @nzregs


335 posts

Ultimate Geek


  Reply # 741926 8-Jan-2013 21:08 Send private message

Regs:
blur: Yep, I was considering calling Microsoft this afternoon and asking for our mailboxes to be moved from the Singapore server farm to the US as a workaround.

What concerns me the most about his email to me is that he will call me tomorrow afternoon to check, by which time another entire day will have passed.

I sincerly hope that I am wrong, but I am gutted that this issue just doesn't seem to be getting the priority that is needed.


you cant just 'move mailboxes from Singapore to US'... its a little more complex than that.  The datacentres are separate.

The Microsoft NZ staff are not directly involved with running the global network/routing, nor the office 365 server farms, so all they can really do is try and escalate internally.  when they get a number of NZ customers with the same problem, especially from multiple ISPs, they can make a bigger noise and are more likely to be 'heard' when trying to escalate. 

if there is a single customer with an issue and Microsoft Ops teams are not sitting on a pile of other service requests - then its going to be hard to convince them that there is a 'major' issue.  For all they know, it could be a problem isolated to a change on a customers firewall... or an ISP who messed up their routing tables...



Talking to the support guys in the US, they explained that it is not uncommon for you to be moved around servers every 3 - 7 days for load balancing reasons. This is why the autodiscover entries in the DNS are so critical.




My HTPC - Case Antec Fusion Remote, MOBO Intel DH67BLB3, CPU Intel Core i5-2400S 2.5 GHz, RAM 8GB  DDR3 1333, HDD 120Gig Corsair Force Series 3 SSD system | WD Caviar Black 2TB data, Tuners Black Gold BGT3595 dual DVB-S/S2, dual DVB-T, Video nVIDIA GeForce GT 520, 1024MB, Sound Intel® High Definition Audio (onboard), OS Windows 7 x64

335 posts

Ultimate Geek


  Reply # 741927 8-Jan-2013 21:10 Send private message

freitasm: I agree with Peter that Microsoft should "own" this problem, or at least show they're putting some effort into it.


Here here!

My contract is with Microsoft, and Microsoft's contract is with SingTel.

It falls squarely at Microsoft's feet in my view.




My HTPC - Case Antec Fusion Remote, MOBO Intel DH67BLB3, CPU Intel Core i5-2400S 2.5 GHz, RAM 8GB  DDR3 1333, HDD 120Gig Corsair Force Series 3 SSD system | WD Caviar Black 2TB data, Tuners Black Gold BGT3595 dual DVB-S/S2, dual DVB-T, Video nVIDIA GeForce GT 520, 1024MB, Sound Intel® High Definition Audio (onboard), OS Windows 7 x64

24 posts

Geek


  Reply # 741928 8-Jan-2013 21:12 Send private message

Customer that been having issues last 2 days is now working. anyone else seeing an improvement?

335 posts

Ultimate Geek


  Reply # 741932 8-Jan-2013 21:15 Send private message

Just tested - seems to be working here too...




My HTPC - Case Antec Fusion Remote, MOBO Intel DH67BLB3, CPU Intel Core i5-2400S 2.5 GHz, RAM 8GB  DDR3 1333, HDD 120Gig Corsair Force Series 3 SSD system | WD Caviar Black 2TB data, Tuners Black Gold BGT3595 dual DVB-S/S2, dual DVB-T, Video nVIDIA GeForce GT 520, 1024MB, Sound Intel® High Definition Audio (onboard), OS Windows 7 x64

Infrastructure Geek
3702 posts

Uber Geek
+1 received by user: 87

Trusted
Microsoft NZ
Subscriber

  Reply # 741957 8-Jan-2013 22:03 Send private message

blur:
Talking to the support guys in the US, they explained that it is not uncommon for you to be moved around servers every 3 - 7 days for load balancing reasons. This is why the autodiscover entries in the DNS are so critical.


you'll be moved around servers in the Singapore/Hong Kong datacenter, but not from APAC to US, as far as i know...




Technical Evangelist
Microsoft NZ
about.me/nzregs
Twitter: @nzregs


4 posts

Wannabe Geek


  Reply # 741958 8-Jan-2013 22:04 Send private message

Testing fine here as well. Both my Office365 mailboxes connected to exchange and synching ok. Hoping it stays this way tomorrow morning.

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic




Twitter »
Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





Trending now »

Hot discussions in our forums right now:

Who Audits IRD?
Created by gundar, last reply by charsleysa on 22-Oct-2014 15:52 (18 replies)
Pages... 2


22nd Only: PB Tech BROTHER HL1110 Mono laser Printer $15 shipped(after $30 cashback)
Created by loceff13, last reply by Jaxson on 22-Oct-2014 15:44 (17 replies)
Pages... 2


American legal jurisdiction in New Zealand
Created by ajobbins, last reply by gzt on 21-Oct-2014 14:58 (30 replies)
Pages... 2


Spark Socialiser
Created by freitasm, last reply by Lizard1977 on 22-Oct-2014 17:17 (29 replies)
Pages... 2


Another Trade Me competitor: SellShed
Created by freitasm, last reply by SellShed on 22-Oct-2014 11:54 (42 replies)
Pages... 2 3


HERE Maps beta available to all Android 4.4 devices and up
Created by freitasm, last reply by ajobbins on 22-Oct-2014 16:27 (13 replies)

Abnormal upstream data usage - Vodafone Cable Wellington
Created by otherside, last reply by otherside on 22-Oct-2014 17:11 (12 replies)

Snap have failed our company!
Created by dafman, last reply by toejam316 on 22-Oct-2014 13:03 (25 replies)
Pages... 2



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.

Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.