News, Trade Me and other sites outages - is this a Bigpipe cache problem?

Forums › New Zealand Broadband › News, Trade Me and other sites outages - is this a Bigpipe cache problem?

1 | 2

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#2720963 8-Jun-2021 22:25

@electricboat:

Must be a shared data centre or CDN for both Trademe and RNZ to go down together (though RNZ continued working for a few minutes longer after trademe went down)

Sites that use Fastly are affected.

Wellingtondave

156 posts

Master Geek

#2720964 8-Jun-2021 22:25

electricboat:

Trademe and RNZ both down with the same error - curiously it provides "Guru Mediation" instead of "Guru Meditation". Maybe the gurus have been fighting?

Must be a shared data centre or CDN for both Trademe and RNZ to go down together (though RNZ continued working for a few minutes longer after trademe went down)

Error 503 Service Unavailable

Service Unavailable

Guru Mediation:

Details: cache-akl10326-AKL 1623147384 1052598505

Currently on with people from around the world, it's a global issue.

Talkiet

4792 posts

Uber Geek

Trusted

#2720965 8-Jun-2021 22:26

(Although their status lists Auckland as operational, and right now, it's not)

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

ResponseMediaNZ

518 posts

Ultimate Geek

ID Verified

Trusted

#2720969 8-Jun-2021 22:36

BBC now reporting it

BBC News - Major newspaper and government websites down https://www.bbc.co.uk/news/technology-57399628

Dial111

978 posts

Ultimate Geek

#2720970 8-Jun-2021 22:37

Has anyone tried turning it on and off at the wall?

Talkiet

4792 posts

Uber Geek

Trusted

#2720972 8-Jun-2021 22:39

https://www.youtube.com/watch?v=ckIMuvumYrg&t=30s

Cheers - N :-)

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Fred99

13684 posts

Uber Geek

#2720973 8-Jun-2021 22:44

Dial111: Has anyone tried turning it on and off at the wall?

Yeah. I just did and everything is working fine now.

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

Talkiet

4792 posts

Uber Geek

Trusted

#2720974 8-Jun-2021 22:46

Graphs are starting to bump back up. Our nodes are serving traffic again.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#2721295 9-Jun-2021 16:43

Summary of June 8 outage | Fastly

On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances.

Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors.

Sideface

9350 posts

Uber Geek

Trusted

Lifetime subscriber

#2723694 10-Jun-2021 12:44

The Guardian - Fastly says single customer triggered bug behind mass internet outage

09 June 2021

An internet blackout that knocked out some of the world’s biggest websites on Tuesday was ultimately caused by a single customer updating their settings, the infrastructure provider Fastly has revealed.

A bug in Fastly’s code introduced in mid-May had lain dormant until Tuesday morning, according to Nick Rockwell, the company’s head of engineering and infrastructure. When the unnamed customer updated their settings, it triggered the flaw, which ultimately took down 85% of the company’s network. ...

The content delivery network (CDN) operated by Fastly is one of the largest on the internet, along with similar networks operated by Akamai, Cloudflare and Amazon’s CloudFront.

All operate on the same principle: that the internet is faster and more stable if users can connect to servers physically close to them, optimised for handling lots of traffic. ...

... the outage highlighted the risks associated with a concentration of critical internet infrastructure in the hands of just a few companies.

[my bold]

Sideface

1 | 2