Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 
timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted
Lifetime subscriber

  #1728320 1-Mar-2017 13:51
Send private message

Groucho:

 

 

 

I think I've seen something similar online and was way over my pay grade but agree with you.  Xero with their 862,000 subscribers probably can't justify the expense unlike another AWS client calling themselves Netflix whose "NZ content" is streamed from a lot closer to home.

 

 

Expense varies with the level of redundancy you want:

 

  • "Cold", where you have a backup of your data and server images outside the region in something like AWS Glacier, wouldn't be too expensive. The time to get the environment running could be days, it depends how much preparation is done.
  • "Pilot light", where you have a minimal mirror environment running and can scale it up, costs more, but not too much.
  • "Active - active", where services run across two sites, is more complex and expensive, with data synchronisation issues. You also have to decide whether you size each environment for the full load in case one fails or half the current load with the ability to scale if one site goes down.

 

 

Highly critical things like NASA mission control and anything where lives can depend on it might use multiple active sites. Accounting software, well, customers can probably wait an hour for it to scale up.




timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted
Lifetime subscriber

  #1728324 1-Mar-2017 14:01
Send private message

Brumfondl:

 

Think AWS is back up :)

 

 

yeah it was back up a couple of hours ago. Downtime was 4-5 hours I think. Some services still have "elevated error rates".


jimbob79
673 posts

Ultimate Geek
+1 received by user: 165


  #1728340 1-Mar-2017 14:16
Send private message

 

 

Here is a good'ish youtube video on why not to rely on external services.

 

https://youtu.be/pptGgNO8d3g

 

 




timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted
Lifetime subscriber

  #1728343 1-Mar-2017 14:29
Send private message

There are also very good reasons to rely on external services. There's no way a small organisation can get access to the security and features of AWS or other cloud providers themselves economically. I run a small server for my business websites, it costs me about NZ$10/month for a small VM. For that I get firewalls (security groups and NACLS), backups (disk snapshots, plus I sync data and database to dropbox), analytics (CloudWatch, free), near real time alerts of problems (Cloudwatch again), a plethora of services I could use, plus the ability to spin up my website in any of around 30 different data centers with very little effort.

 

AWS is pretty complex though. I've done hundreds or perhaps low thousands of hours study towards AWS certifications.


jonb
1796 posts

Uber Geek
+1 received by user: 545

Trusted

  #1728345 1-Mar-2017 14:38
Send private message

timmmay:

 

It's interesting to see how many services are poorly architected. Yes S3 stores data across multiple AZs in order to increase durability and availability, but it seems something must be shared, or they deployed to all AZs in the same time period. Cross region replication is recommended for high reliability.

 

TLDR: many places put all their eggs in one basket.

 

It's coming back up now, but may be an hour before it's fully working.

 

--

 

Update at 12:52 PM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.

 

 

 

 

Surprised at Xero - having only migrated to AWS late last year would've thought they would have designed more resiliency


timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted
Lifetime subscriber

  #1728348 1-Mar-2017 14:45
Send private message

It's interesting that there are no reports of outages on amazon.com. Werner Vogels the Amazon.com CTO famously said "Everything fails all the time", and they have more AWS experience than anyone (completely separate business units), so I assume they've architected for failure.


 
 
 

Support Geekzone with one-off or recurring donations Donate via PressPatron.
lxsw20
3689 posts

Uber Geek
+1 received by user: 2174

Subscriber

  #1728589 2-Mar-2017 05:34
Send private message

Ive read reports stating S3 buckets were down across multiple regions so not sure how cross region replication would have helped here?


timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted
Lifetime subscriber

  #1728599 2-Mar-2017 07:03
Send private message

lxsw20:

 

Ive read reports stating S3 buckets were down across multiple regions so not sure how cross region replication would have helped here?

 

 

Amazon are pretty transparent on outages, there's nothing in their status history (bottom of the page), and I haven't read anything about a multi region outage. Can you share link to a reputable source?


lxsw20
3689 posts

Uber Geek
+1 received by user: 2174

Subscriber

  #1728864 2-Mar-2017 12:30
Send private message

Nope, just people on reddit at the time saying they couldn't create buckets in Singapore/Ireland either. 


timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted
Lifetime subscriber

  #1728878 2-Mar-2017 13:01
Send private message

Regions should be independent. However US-EAST-1 is their "home" region so it's possible that there's a dependency. S3 buckets do need to be globally unique, which suggests a central registry.

 

The silver lining of the outage: AWS probably won't fail in this way again, and they'll likely do significant work on making sure regions really are independent and the status page is accurate.


lxsw20
3689 posts

Uber Geek
+1 received by user: 2174

Subscriber

  #1729210 3-Mar-2017 06:34
Send private message

https://aws.amazon.com/message/41926/

 

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

 

 

 

We've all done it. Just not on something that takes down a large amount of the internet.


 
 
 

Support Geekzone with one-off or recurring donations Donate via PressPatron.
timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted
Lifetime subscriber

  #1729212 3-Mar-2017 06:48
Send private message

Ah, a PEBKAC error. Well, I bet they won't make that particular mistake again.


Regs
4066 posts

Uber Geek
+1 received by user: 206

Trusted
Snowflake

  #1731092 6-Mar-2017 16:31
Send private message

lxsw20:

 

https://aws.amazon.com/message/41926/

 

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

 

 

 

We've all done it. Just not on something that takes down a large amount of the internet.

 

 

 

 

"The Amazon Simple Storage Service (S3) team was debugging an issue causing the S3 billing system to progress more slowly than expected."





1 | 2 
View this topic in a long page with up to 500 replies per page Create new topic








Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.