News, Trade Me and other sites outages - is this a Bigpipe cache problem?

Forums › New Zealand Broadband › News, Trade Me and other sites outages - is this a Bigpipe cache problem?

1 | 2

freitasm

BDFL - Memuneh
80659 posts

Uber Geek
+1 received by user: 41075

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#2720963 8-Jun-2021 22:25

@electricboat:

Must be a shared data centre or CDN for both Trademe and RNZ to go down together (though RNZ continued working for a few minutes longer after trademe went down)

Sites that use Fastly are affected.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

Wellingtondave

160 posts

Master Geek
+1 received by user: 88

#2720964 8-Jun-2021 22:25

electricboat:

Trademe and RNZ both down with the same error - curiously it provides "Guru Mediation" instead of "Guru Meditation". Maybe the gurus have been fighting?

Must be a shared data centre or CDN for both Trademe and RNZ to go down together (though RNZ continued working for a few minutes longer after trademe went down)

Error 503 Service Unavailable

Service Unavailable

Guru Mediation:

Details: cache-akl10326-AKL 1623147384 1052598505

Currently on with people from around the world, it's a global issue.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#2720965 8-Jun-2021 22:26

(Although their status lists Auckland as operational, and right now, it's not)

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

ResponseMediaNZ

518 posts

Ultimate Geek
+1 received by user: 196

ID Verified

Trusted

#2720969 8-Jun-2021 22:36

BBC now reporting it

BBC News - Major newspaper and government websites down https://www.bbc.co.uk/news/technology-57399628

Dial111

979 posts

Ultimate Geek
+1 received by user: 656

#2720970 8-Jun-2021 22:37

Has anyone tried turning it on and off at the wall?

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#2720972 8-Jun-2021 22:39

https://www.youtube.com/watch?v=ckIMuvumYrg&t=30s

Cheers - N :-)

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Dyson

Shop now for Dyson appliances (affiliate link).

Fred99

13684 posts

Uber Geek
+1 received by user: 10018

#2720973 8-Jun-2021 22:44

Dial111: Has anyone tried turning it on and off at the wall?

Yeah. I just did and everything is working fine now.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#2720974 8-Jun-2021 22:46

Graphs are starting to bump back up. Our nodes are serving traffic again.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

freitasm

BDFL - Memuneh
80659 posts

Uber Geek
+1 received by user: 41075

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#2721295 9-Jun-2021 16:43

Summary of June 8 outage | Fastly

On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances.

Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

Sideface

9652 posts

Uber Geek
+1 received by user: 15602

Trusted

Lifetime subscriber

#2723694 10-Jun-2021 12:44

The Guardian - Fastly says single customer triggered bug behind mass internet outage

09 June 2021

An internet blackout that knocked out some of the world’s biggest websites on Tuesday was ultimately caused by a single customer updating their settings, the infrastructure provider Fastly has revealed.

A bug in Fastly’s code introduced in mid-May had lain dormant until Tuesday morning, according to Nick Rockwell, the company’s head of engineering and infrastructure. When the unnamed customer updated their settings, it triggered the flaw, which ultimately took down 85% of the company’s network. ...

The content delivery network (CDN) operated by Fastly is one of the largest on the internet, along with similar networks operated by Akamai, Cloudflare and Amazon’s CloudFront.

All operate on the same principle: that the internet is faster and more stable if users can connect to servers physically close to them, optimised for handling lots of traffic. ...

... the outage highlighted the risks associated with a concentration of critical internet infrastructure in the hands of just a few companies.

[my bold]

Sideface

1 | 2