Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 


14760 posts

Uber Geek
+1 received by user: 2746

Trusted
Subscriber

  # 1728320 1-Mar-2017 13:51
One person supports this post
Send private message

Groucho:

 

 

 

I think I've seen something similar online and was way over my pay grade but agree with you.  Xero with their 862,000 subscribers probably can't justify the expense unlike another AWS client calling themselves Netflix whose "NZ content" is streamed from a lot closer to home.

 

 

Expense varies with the level of redundancy you want:

 

  • "Cold", where you have a backup of your data and server images outside the region in something like AWS Glacier, wouldn't be too expensive. The time to get the environment running could be days, it depends how much preparation is done.
  • "Pilot light", where you have a minimal mirror environment running and can scale it up, costs more, but not too much.
  • "Active - active", where services run across two sites, is more complex and expensive, with data synchronisation issues. You also have to decide whether you size each environment for the full load in case one fails or half the current load with the ability to scale if one site goes down.

 

 

Highly critical things like NASA mission control and anything where lives can depend on it might use multiple active sites. Accounting software, well, customers can probably wait an hour for it to scale up.




14760 posts

Uber Geek
+1 received by user: 2746

Trusted
Subscriber

  # 1728324 1-Mar-2017 14:01
Send private message

Brumfondl:

 

Think AWS is back up :)

 

 

yeah it was back up a couple of hours ago. Downtime was 4-5 hours I think. Some services still have "elevated error rates".


 
 
 
 


652 posts

Ultimate Geek
+1 received by user: 163


  # 1728340 1-Mar-2017 14:16
Send private message

 

 

Here is a good'ish youtube video on why not to rely on external services.

 

https://youtu.be/pptGgNO8d3g

 

 








14760 posts

Uber Geek
+1 received by user: 2746

Trusted
Subscriber

  # 1728343 1-Mar-2017 14:29
One person supports this post
Send private message

There are also very good reasons to rely on external services. There's no way a small organisation can get access to the security and features of AWS or other cloud providers themselves economically. I run a small server for my business websites, it costs me about NZ$10/month for a small VM. For that I get firewalls (security groups and NACLS), backups (disk snapshots, plus I sync data and database to dropbox), analytics (CloudWatch, free), near real time alerts of problems (Cloudwatch again), a plethora of services I could use, plus the ability to spin up my website in any of around 30 different data centers with very little effort.

 

AWS is pretty complex though. I've done hundreds or perhaps low thousands of hours study towards AWS certifications.


1486 posts

Uber Geek
+1 received by user: 343

Trusted

  # 1728345 1-Mar-2017 14:38
Send private message

timmmay:

 

It's interesting to see how many services are poorly architected. Yes S3 stores data across multiple AZs in order to increase durability and availability, but it seems something must be shared, or they deployed to all AZs in the same time period. Cross region replication is recommended for high reliability.

 

TLDR: many places put all their eggs in one basket.

 

It's coming back up now, but may be an hour before it's fully working.

 

--

 

Update at 12:52 PM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.

 

 

 

 

Surprised at Xero - having only migrated to AWS late last year would've thought they would have designed more resiliency




14760 posts

Uber Geek
+1 received by user: 2746

Trusted
Subscriber

  # 1728348 1-Mar-2017 14:45
Send private message

It's interesting that there are no reports of outages on amazon.com. Werner Vogels the Amazon.com CTO famously said "Everything fails all the time", and they have more AWS experience than anyone (completely separate business units), so I assume they've architected for failure.


2249 posts

Uber Geek
+1 received by user: 698

Subscriber

  # 1728589 2-Mar-2017 05:34
Send private message

Ive read reports stating S3 buckets were down across multiple regions so not sure how cross region replication would have helped here?




14760 posts

Uber Geek
+1 received by user: 2746

Trusted
Subscriber

  # 1728599 2-Mar-2017 07:03
Send private message

lxsw20:

 

Ive read reports stating S3 buckets were down across multiple regions so not sure how cross region replication would have helped here?

 

 

Amazon are pretty transparent on outages, there's nothing in their status history (bottom of the page), and I haven't read anything about a multi region outage. Can you share link to a reputable source?


2249 posts

Uber Geek
+1 received by user: 698

Subscriber

  # 1728864 2-Mar-2017 12:30
Send private message

Nope, just people on reddit at the time saying they couldn't create buckets in Singapore/Ireland either. 




14760 posts

Uber Geek
+1 received by user: 2746

Trusted
Subscriber

  # 1728878 2-Mar-2017 13:01
Send private message

Regions should be independent. However US-EAST-1 is their "home" region so it's possible that there's a dependency. S3 buckets do need to be globally unique, which suggests a central registry.

 

The silver lining of the outage: AWS probably won't fail in this way again, and they'll likely do significant work on making sure regions really are independent and the status page is accurate.


2249 posts

Uber Geek
+1 received by user: 698

Subscriber

  # 1729210 3-Mar-2017 06:34
Send private message

https://aws.amazon.com/message/41926/

 

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

 

 

 

We've all done it. Just not on something that takes down a large amount of the internet.




14760 posts

Uber Geek
+1 received by user: 2746

Trusted
Subscriber

  # 1729212 3-Mar-2017 06:48
Send private message

Ah, a PEBKAC error. Well, I bet they won't make that particular mistake again.


Infrastructure Geek
4057 posts

Uber Geek
+1 received by user: 195

Trusted
Microsoft NZ
Subscriber

  # 1731092 6-Mar-2017 16:31
Send private message

lxsw20:

 

https://aws.amazon.com/message/41926/

 

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

 

 

 

We've all done it. Just not on something that takes down a large amount of the internet.

 

 

 

 

"The Amazon Simple Storage Service (S3) team was debugging an issue causing the S3 billing system to progress more slowly than expected."





Technical Evangelist
Microsoft NZ
about.me/nzregs
Twitter: @nzregs


1 | 2 
View this topic in a long page with up to 500 replies per page Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

Video game market in New Zealand passes half billion dollar mark
Posted 24-May-2019 16:15


WLG-X festival to celebrate creativity and innovation
Posted 22-May-2019 17:53


HPE to acquire supercomputing leader Cray
Posted 20-May-2019 11:07


Techweek starting around NZ today
Posted 20-May-2019 09:52


Porirua City Council first to adopt new council software solution Datascape
Posted 15-May-2019 12:00


New survey provides insight into schools' technology challenges and plans
Posted 15-May-2019 09:30


Apple Music now available on Alexa devices in Australia and New Zealand
Posted 15-May-2019 09:11


Make a stand against cyberbullying this Pink Shirt Day
Posted 14-May-2019 20:23


Samsung first TV manufacturer to launch the Apple TV App and Airplay 2
Posted 14-May-2019 20:11


Vodafone New Zealand sold
Posted 14-May-2019 07:25


Kordia boosts cloud performance with locally-hosted Microsoft Azure ExpressRoute
Posted 8-May-2019 10:25


Microsoft Azure ExpressRoute in New Zealand opens up faster, more secure internet for Kiwi businesses
Posted 8-May-2019 09:39


Vocus Communications to deliver Microsoft Azure Cloud Solutions through Azure ExpressRoute
Posted 8-May-2019 09:25


Independent NZ feature film #statusPending to premiere during WLG-X
Posted 6-May-2019 22:13


The ultimate dog photoshoot with Nokia 9 PureView #ForgottenDogsofInstagram
Posted 6-May-2019 09:41



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.