Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 


14785 posts

Uber Geek
+1 received by user: 2750

Trusted
Subscriber

  # 1728320 1-Mar-2017 13:51
One person supports this post
Send private message

Groucho:

 

 

 

I think I've seen something similar online and was way over my pay grade but agree with you.  Xero with their 862,000 subscribers probably can't justify the expense unlike another AWS client calling themselves Netflix whose "NZ content" is streamed from a lot closer to home.

 

 

Expense varies with the level of redundancy you want:

 

  • "Cold", where you have a backup of your data and server images outside the region in something like AWS Glacier, wouldn't be too expensive. The time to get the environment running could be days, it depends how much preparation is done.
  • "Pilot light", where you have a minimal mirror environment running and can scale it up, costs more, but not too much.
  • "Active - active", where services run across two sites, is more complex and expensive, with data synchronisation issues. You also have to decide whether you size each environment for the full load in case one fails or half the current load with the ability to scale if one site goes down.

 

 

Highly critical things like NASA mission control and anything where lives can depend on it might use multiple active sites. Accounting software, well, customers can probably wait an hour for it to scale up.




14785 posts

Uber Geek
+1 received by user: 2750

Trusted
Subscriber

  # 1728324 1-Mar-2017 14:01
Send private message

Brumfondl:

 

Think AWS is back up :)

 

 

yeah it was back up a couple of hours ago. Downtime was 4-5 hours I think. Some services still have "elevated error rates".


 
 
 
 


656 posts

Ultimate Geek
+1 received by user: 163


  # 1728340 1-Mar-2017 14:16
Send private message

 

 

Here is a good'ish youtube video on why not to rely on external services.

 

https://youtu.be/pptGgNO8d3g

 

 








14785 posts

Uber Geek
+1 received by user: 2750

Trusted
Subscriber

  # 1728343 1-Mar-2017 14:29
One person supports this post
Send private message

There are also very good reasons to rely on external services. There's no way a small organisation can get access to the security and features of AWS or other cloud providers themselves economically. I run a small server for my business websites, it costs me about NZ$10/month for a small VM. For that I get firewalls (security groups and NACLS), backups (disk snapshots, plus I sync data and database to dropbox), analytics (CloudWatch, free), near real time alerts of problems (Cloudwatch again), a plethora of services I could use, plus the ability to spin up my website in any of around 30 different data centers with very little effort.

 

AWS is pretty complex though. I've done hundreds or perhaps low thousands of hours study towards AWS certifications.


1491 posts

Uber Geek
+1 received by user: 350

Trusted

  # 1728345 1-Mar-2017 14:38
Send private message

timmmay:

 

It's interesting to see how many services are poorly architected. Yes S3 stores data across multiple AZs in order to increase durability and availability, but it seems something must be shared, or they deployed to all AZs in the same time period. Cross region replication is recommended for high reliability.

 

TLDR: many places put all their eggs in one basket.

 

It's coming back up now, but may be an hour before it's fully working.

 

--

 

Update at 12:52 PM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.

 

 

 

 

Surprised at Xero - having only migrated to AWS late last year would've thought they would have designed more resiliency




14785 posts

Uber Geek
+1 received by user: 2750

Trusted
Subscriber

  # 1728348 1-Mar-2017 14:45
Send private message

It's interesting that there are no reports of outages on amazon.com. Werner Vogels the Amazon.com CTO famously said "Everything fails all the time", and they have more AWS experience than anyone (completely separate business units), so I assume they've architected for failure.


2255 posts

Uber Geek
+1 received by user: 703

Subscriber

  # 1728589 2-Mar-2017 05:34
Send private message

Ive read reports stating S3 buckets were down across multiple regions so not sure how cross region replication would have helped here?


 
 
 
 




14785 posts

Uber Geek
+1 received by user: 2750

Trusted
Subscriber

  # 1728599 2-Mar-2017 07:03
Send private message

lxsw20:

 

Ive read reports stating S3 buckets were down across multiple regions so not sure how cross region replication would have helped here?

 

 

Amazon are pretty transparent on outages, there's nothing in their status history (bottom of the page), and I haven't read anything about a multi region outage. Can you share link to a reputable source?


2255 posts

Uber Geek
+1 received by user: 703

Subscriber

  # 1728864 2-Mar-2017 12:30
Send private message

Nope, just people on reddit at the time saying they couldn't create buckets in Singapore/Ireland either. 




14785 posts

Uber Geek
+1 received by user: 2750

Trusted
Subscriber

  # 1728878 2-Mar-2017 13:01
Send private message

Regions should be independent. However US-EAST-1 is their "home" region so it's possible that there's a dependency. S3 buckets do need to be globally unique, which suggests a central registry.

 

The silver lining of the outage: AWS probably won't fail in this way again, and they'll likely do significant work on making sure regions really are independent and the status page is accurate.


2255 posts

Uber Geek
+1 received by user: 703

Subscriber

  # 1729210 3-Mar-2017 06:34
Send private message

https://aws.amazon.com/message/41926/

 

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

 

 

 

We've all done it. Just not on something that takes down a large amount of the internet.




14785 posts

Uber Geek
+1 received by user: 2750

Trusted
Subscriber

  # 1729212 3-Mar-2017 06:48
Send private message

Ah, a PEBKAC error. Well, I bet they won't make that particular mistake again.


Infrastructure Geek
4057 posts

Uber Geek
+1 received by user: 195

Trusted
Microsoft NZ
Subscriber

  # 1731092 6-Mar-2017 16:31
Send private message

lxsw20:

 

https://aws.amazon.com/message/41926/

 

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

 

 

 

We've all done it. Just not on something that takes down a large amount of the internet.

 

 

 

 

"The Amazon Simple Storage Service (S3) team was debugging an issue causing the S3 billing system to progress more slowly than expected."





Technical Evangelist
Microsoft NZ
about.me/nzregs
Twitter: @nzregs


1 | 2 
View this topic in a long page with up to 500 replies per page Create new topic



Twitter and LinkedIn »



Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

Anyone can broadcast with Kordia Pop Up TV
Posted 13-Jun-2019 10:51


Volvo and Uber present production vehicle ready for self-driving
Posted 13-Jun-2019 10:47


100,000 customers connected to fibre broadband network through Enable
Posted 13-Jun-2019 10:35


5G uptake even faster than expected
Posted 12-Jun-2019 10:01


Xbox showcases 60 anticipated games
Posted 10-Jun-2019 20:24


Trend Micro Turns Public Hotspots into Secure Networks with WiFi Protection for Mobile Devices
Posted 5-Jun-2019 13:24


Bold UK spinoff for beauty software company Flossie
Posted 2-Jun-2019 14:10


Amazon Introduces Echo Show 5
Posted 1-Jun-2019 15:32


Epson launches new 4K Pro-UHD projector technology
Posted 1-Jun-2019 15:26


Lenovo and Qualcomm unveil first 5G PC called Project Limitless
Posted 28-May-2019 20:23


Intel introduces new 10th Gen Intel Core Processors and Project Athena
Posted 28-May-2019 19:28


Orcon first to trial residential 10Gbps broadband
Posted 28-May-2019 11:20


Video game market in New Zealand passes half billion dollar mark
Posted 24-May-2019 16:15


WLG-X festival to celebrate creativity and innovation
Posted 22-May-2019 17:53


HPE to acquire supercomputing leader Cray
Posted 20-May-2019 11:07



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.