Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 
timmmay

20574 posts

Uber Geek

Trusted
Lifetime subscriber

  #1728320 1-Mar-2017 13:51
Send private message

Groucho:

 

 

 

I think I've seen something similar online and was way over my pay grade but agree with you.  Xero with their 862,000 subscribers probably can't justify the expense unlike another AWS client calling themselves Netflix whose "NZ content" is streamed from a lot closer to home.

 

 

Expense varies with the level of redundancy you want:

 

  • "Cold", where you have a backup of your data and server images outside the region in something like AWS Glacier, wouldn't be too expensive. The time to get the environment running could be days, it depends how much preparation is done.
  • "Pilot light", where you have a minimal mirror environment running and can scale it up, costs more, but not too much.
  • "Active - active", where services run across two sites, is more complex and expensive, with data synchronisation issues. You also have to decide whether you size each environment for the full load in case one fails or half the current load with the ability to scale if one site goes down.

 

 

Highly critical things like NASA mission control and anything where lives can depend on it might use multiple active sites. Accounting software, well, customers can probably wait an hour for it to scale up.




timmmay

20574 posts

Uber Geek

Trusted
Lifetime subscriber

  #1728324 1-Mar-2017 14:01
Send private message

Brumfondl:

 

Think AWS is back up :)

 

 

yeah it was back up a couple of hours ago. Downtime was 4-5 hours I think. Some services still have "elevated error rates".


jimbob79
673 posts

Ultimate Geek


  #1728340 1-Mar-2017 14:16
Send private message

 

 

Here is a good'ish youtube video on why not to rely on external services.

 

https://youtu.be/pptGgNO8d3g

 

 




timmmay

20574 posts

Uber Geek

Trusted
Lifetime subscriber

  #1728343 1-Mar-2017 14:29
Send private message

There are also very good reasons to rely on external services. There's no way a small organisation can get access to the security and features of AWS or other cloud providers themselves economically. I run a small server for my business websites, it costs me about NZ$10/month for a small VM. For that I get firewalls (security groups and NACLS), backups (disk snapshots, plus I sync data and database to dropbox), analytics (CloudWatch, free), near real time alerts of problems (Cloudwatch again), a plethora of services I could use, plus the ability to spin up my website in any of around 30 different data centers with very little effort.

 

AWS is pretty complex though. I've done hundreds or perhaps low thousands of hours study towards AWS certifications.


jonb
1771 posts

Uber Geek

Trusted

  #1728345 1-Mar-2017 14:38
Send private message

timmmay:

 

It's interesting to see how many services are poorly architected. Yes S3 stores data across multiple AZs in order to increase durability and availability, but it seems something must be shared, or they deployed to all AZs in the same time period. Cross region replication is recommended for high reliability.

 

TLDR: many places put all their eggs in one basket.

 

It's coming back up now, but may be an hour before it's fully working.

 

--

 

Update at 12:52 PM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.

 

 

 

 

Surprised at Xero - having only migrated to AWS late last year would've thought they would have designed more resiliency


timmmay

20574 posts

Uber Geek

Trusted
Lifetime subscriber

  #1728348 1-Mar-2017 14:45
Send private message

It's interesting that there are no reports of outages on amazon.com. Werner Vogels the Amazon.com CTO famously said "Everything fails all the time", and they have more AWS experience than anyone (completely separate business units), so I assume they've architected for failure.


lxsw20
3552 posts

Uber Geek

Subscriber

  #1728589 2-Mar-2017 05:34
Send private message

Ive read reports stating S3 buckets were down across multiple regions so not sure how cross region replication would have helped here?


 
 
 

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.
timmmay

20574 posts

Uber Geek

Trusted
Lifetime subscriber

  #1728599 2-Mar-2017 07:03
Send private message

lxsw20:

 

Ive read reports stating S3 buckets were down across multiple regions so not sure how cross region replication would have helped here?

 

 

Amazon are pretty transparent on outages, there's nothing in their status history (bottom of the page), and I haven't read anything about a multi region outage. Can you share link to a reputable source?


lxsw20
3552 posts

Uber Geek

Subscriber

  #1728864 2-Mar-2017 12:30
Send private message

Nope, just people on reddit at the time saying they couldn't create buckets in Singapore/Ireland either. 


timmmay

20574 posts

Uber Geek

Trusted
Lifetime subscriber

  #1728878 2-Mar-2017 13:01
Send private message

Regions should be independent. However US-EAST-1 is their "home" region so it's possible that there's a dependency. S3 buckets do need to be globally unique, which suggests a central registry.

 

The silver lining of the outage: AWS probably won't fail in this way again, and they'll likely do significant work on making sure regions really are independent and the status page is accurate.


lxsw20
3552 posts

Uber Geek

Subscriber

  #1729210 3-Mar-2017 06:34
Send private message

https://aws.amazon.com/message/41926/

 

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

 

 

 

We've all done it. Just not on something that takes down a large amount of the internet.


timmmay

20574 posts

Uber Geek

Trusted
Lifetime subscriber

  #1729212 3-Mar-2017 06:48
Send private message

Ah, a PEBKAC error. Well, I bet they won't make that particular mistake again.


Regs
4066 posts

Uber Geek

Trusted
Snowflake

  #1731092 6-Mar-2017 16:31
Send private message

lxsw20:

 

https://aws.amazon.com/message/41926/

 

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

 

 

 

We've all done it. Just not on something that takes down a large amount of the internet.

 

 

 

 

"The Amazon Simple Storage Service (S3) team was debugging an issue causing the S3 billing system to progress more slowly than expected."





1 | 2 
View this topic in a long page with up to 500 replies per page Create new topic





News and reviews »

Air New Zealand Starts AI adoption with OpenAI
Posted 24-Jul-2025 16:00


eero Pro 7 Review
Posted 23-Jul-2025 12:07


BeeStation Plus Review
Posted 21-Jul-2025 14:21


eero Unveils New Wi-Fi 7 Products in New Zealand
Posted 21-Jul-2025 00:01


WiZ Introduces HDMI Sync Box and other Light Devices
Posted 20-Jul-2025 17:32


RedShield Enhances DDoS and Bot Attack Protection
Posted 20-Jul-2025 17:26


Seagate Ships 30TB Drives
Posted 17-Jul-2025 11:24


Oclean AirPump A10 Water Flosser Review
Posted 13-Jul-2025 11:05


Samsung Galaxy Z Fold7: Raising the Bar for Smartphones
Posted 10-Jul-2025 02:01


Samsung Galaxy Z Flip7 Brings New Edge-To-Edge FlexWindow
Posted 10-Jul-2025 02:01


Epson Launches New AM-C550Z WorkForce Enterprise printer
Posted 9-Jul-2025 18:22


Samsung Releases Smart Monitor M9
Posted 9-Jul-2025 17:46


Nearly Half of Older Kiwis Still Write their Passwords on Paper
Posted 9-Jul-2025 08:42


D-Link 4G+ Cat6 Wi-Fi 6 DWR-933M Mobile Hotspot Review
Posted 1-Jul-2025 11:34


Oppo A5 Series Launches With New Levels of Durability
Posted 30-Jun-2025 10:15









Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.