CloudFlare and CDN for cloud failover

Forums › IT Pro and developers › CloudFlare and CDN for cloud failover

1 | 2 | 3

2102 posts

Uber Geek
+1 received by user: 557

#1101573 4-Aug-2014 11:28

@Zeon

Your absolutely right it will not always meet unique needs. Especially in the case of NZ based being essential, personally found it doesnt affect my customers, but am not moving massive amounts of data. Not sure what you mean by CPU cures, but out of interest are you talking about the pricing of RDS? It is a managed DB solution, so to compare it to running and managing your own server with it installed is not really apples with apples.

jasnz

41 posts

Geek
+1 received by user: 3

#1101622 4-Aug-2014 12:12

As Zeon says the cost of running in the cloud is actually much higher. In fact we actually run out of one of our houses! The colocation costs we found were fairly high in NZ as well. We could be shipping servers around NZ but things in NZ are just expensive... Or so I found when I looked at a number of options. (If you are an ISP in the south Island and can provide colocation with decent bandwidth (i.e. the same sorts of bandwidth you can get by buying home version ADSL or fiber) ie Something like 20MBs down / 5MBs up sort of thing, with a cost that is not more than power_costs * 1.4 (i.e. you are making a 40% profit per month on this, then it really makes a lot more sense to host out of our house with normal ADSL / VDSL / fiber. With 4U of servers we were looking at a minimum of something like NZ $400 a month. Right now we pay NZ$100 in ISP charges (it was being paid anyway) and about roughly NZ$150 in power per month. So we are saving arguably NZ$250 per month, plus we can just bring on more servers, etc as we need.

Plus we get a free heater in the winter :)

The only really downside to home hosting is that we would love to be able to have a /29 or something and have a few more IP's to play with, but in the end for our mail server we can just run that on digital ocean for US$20 per month. It was that or spend at least NZ$1000 trying to get some sort of managed solution with a /29. Again if there is a good option here I don't know about I would love to know.

In any case the servers on DigitalOcean would be around something like US$400 a month. (Actually they have really fast SSD's whereas we only have Raid 15K disks in our servers so we have found it is not so bad to use a lot of swap and then we might be able to get away with 3G or even 2G machines but it would be tight...) AWS is something like double or triple that... So basically much more expensive!

I think it is much better idea to just fail over into the cloud if things go down and just live with the fact we will be down for 5 minutes and then back up. That is really not so bad... I actually find it quite strange that this kind of architecture is not more common. Ie when we looked into it and saw a few server centres with gen-i etc, I saw ultra reliable this and that, where they have obviously spent a huge amount of money on all sorts of protection... but of course things happen. e.g. the Christchurch earthquake. I am sure a lot of data centres went down in Christchuch at the time. Instead I would think it would be simply much better to have a much cheaper system that you almost expect to go down a couple of times a year and just fail over to it somewhere else and smoothly at that... At least this is how we are trying to architect things for the moment.

Thanks,
Jas

freitasm

BDFL - Memuneh
80657 posts

Uber Geek
+1 received by user: 41065

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#1101646 4-Aug-2014 12:20

Failover into the cloud seems interesting, until you think of the complexities.

Is your app a transactional app, or does it capture any information? Because once you failover any transactions performed in the cloud need to be replicated back to your master database. If you don't have the code there then you need to develop and test. If you have the code, a five minutes failover may in fact be a much longer thing - failover via DNS takes at most five minutes, your server gets back in five minutes (as you said), then DNS changes take another five minutes. But then some DNS around the world don't respect TTL (Telecom, ahem) and you will see a lot of the traffic going to the cloud IP addresses which in theory shouldn't be up anymore. So you get frustrated users who can access your site for a few hours perhaps. But the cloud server needs to be up to replicate the transactions back to your master database so it may be still responding to these stray customers - when do you cut off access if you are only doing a DNS trick?

Also if you have a transactional system how to you keep the cloud database in sync with the master? You need it running at least some time to get the database always ready in a failover. Of course if it is a static site then it's not a problem, but your description above seems to indicate diferently.

I don't think you are looking at the whole business continuity, uptime/failover from the right angle.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

jasnz

41 posts

Geek
+1 received by user: 3

#1101688 4-Aug-2014 13:12

Interesting points, Actually we are planning to do postgres warm standby replication. It just so happens we have the other transactions into the database available from another source so this is probably unusual compared to other more typical situations. Moreover we can keep standbys of multiple VM's on one machine. Or this is the plan. So our failover control machine will be always up and doing the warm standby replication for all of the other VM's.

Happily the way things are we can actually ship the data back fairly easily. Also it is not so hard. Moreover the machine is fairly stateless as far as the end users go. So we don't actually have many problems there. Eg an example of something different but maybe similar, is a real time system producing information on stock market prices for stocks and other commodities. Users can come in, look at some prices, do some charting, look at some real time tickers, etc. But in the end if we fall over into the cloud we can just grab the missing data again. Our data is not so readily available as this kind of public data but we can get bits of it back when we need. Also there is no real question of syntonisation where one system thinks one thing and the other system thinks something else. Everyone will agree that at 9:56.22 the price of IBM was $17.32 etc. As long as the users are just using this to look at graphs, or some aggregated data then we don't really have so much of a problem with data synchronisation. Or at least this is what it appears to us when we have thought about it. But we might be missing something of course :)

Interesting that the TTL is not respected. Then maybe we should be using CloudFlare as a CDN, and so the address of our site would always be the same then, if I understand this? Then we just point CloudFlare to a different server on the back end but the front end is always CloudFlare and always the same?...

freitasm

BDFL - Memuneh
80657 posts

Uber Geek
+1 received by user: 41065

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#1101701 4-Aug-2014 13:40

Is this service in production yet or just testing?

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

insane

3324 posts

Uber Geek
+1 received by user: 1006

ID Verified

Trusted

2degrees

Subscriber

#1102160 4-Aug-2014 22:04

freitasm: The best thing anyway is not use DNS as failover management because DNS changes can take some time to be used since there's a TTL for those records. If the TTL is too high DNS resolvers will take too long to refresh. If TTL is too low changes are too frequent impacting performance on client side.

The correct way of implementing failover is through a load balancer in front of a failover cluster.

DNS will not be the answer.

freitasm:

...- failover via DNS takes at most five minutes, your server gets back in five minutes (as you said), then DNS changes take another five minutes. But then some DNS around the world don't respect TTL (Telecom, ahem) and you will see a lot of the traffic going to the cloud IP addresses which in theory shouldn't be up anymore. So you get frustrated users who can access your site for a few hours perhaps....

Zeon: Talk to Insane, he works for Vocus and they have some kind of geo-seperate solution for that.

I used to work for Maxnet / Vocus ;)

But yes they use F5 Global Traffic Managers (GTMs), and Local Traffic Managers (LTMs), the GTMs being what you're referring to which provide ultra fast failover (talking seconds not minutes) to provide geographic redundancy through smart DNS (GTMs are really just DNS servers with fancy health checks and a lot of smarts). They don't have to host your domain name, you just setup a bunch of CNAMES for each of services you need, and they use an internal/service domain name to do the rest. Most of their workloads are balanced between their Auckland and Christchurch Data Centers, but there's ultimately no reason why they couldn't be used for pointing traffic to any number of locations.

Of course they also have LTMs for balancing traffic between pools of servers, again with some serious smarts around the health checks in a similar way to something like a Citrix Netscaler.

They have done, and still do load balancing for some very popular NZ websites, eg NZ Herald, Southern Cross Travel Insurance etc, and in the 4 or so years I was there with that product I never came across people having issues with stale DNS records, browser cache was actually the only issue which affected some people for up to 30 seconds.

I'm actually working on a huge project right now which is using F5 GTMs to provide geographical redundancy for up to 1 million users. TradeMe actually also uses them for their Auckland to Wellington fail-over, just bigger boxes :)

Dell

Shop now for Dell laptops and other devices (affiliate link).

jasnz

41 posts

Geek
+1 received by user: 3

#1102284 5-Aug-2014 01:14

@BDFL Well out main service is in production, but the failover is in testing. (We have our ansible scripts that do everything for failover but we are in the process of generalising these a bit and robustifying them (e.g. just to get things going some things were hard coded, and now that we sees that the launch times etc are acceptable, we are going back over the scripts and making them "good", etc.),

We can do programatic changing of the DNS via CloudFlare, we are in the process of changing the bits which send our main site the information, and getting ready to put it all together... Hence this chain of emails :)

@insane, ahhh! This sounds like something we should definitely investigate, and would be better than CloudFlare (or at least we should compare it's speeds etc.) Is there some person you would recommend that we could call and chat this over with? Thanks!

Jas

freitasm

BDFL - Memuneh
80657 posts

Uber Geek
+1 received by user: 41065

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#1102299 5-Aug-2014 07:37

Thanks for your reply.

If I understand your previous posts you are running this service from home then?

My opinion is that if you want a robust service you should first move the servers from home to a colocation facility. Your availability levels should go up a lot from this move alone - you get an environment with regulated temperature and humidity, you get resilient network with multiple paths, you get managed firewall, etc. The network alone would be a lot better than a DSL connection, which has limited upload speeds, limiting your site anyway. For example if you do use a CDN I would be happier with the static assets being distributed from their POP in Australia than a DSL connection in New Zealand.

Once this is done you then start thinking about failover. Your current environment is not good enough to get you in "production" these days.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

jasnz

41 posts

Geek
+1 received by user: 3

#1102610 5-Aug-2014 13:31

@freitasm Thanks for your reply.

Q: If I understand your previous posts you are running this service from home then?
A: Yep. There is a dedicated room for this, so pretty much like running out of some office or something, but yep.

Q: My opinion is that if you want a robust service you should first move the servers from home to a colocation facility. Your availability levels should go up a lot from this move alone - you get an environment with regulated temperature and humidity, you get resilient network with multiple paths, you get managed firewall, etc. The network alone would be a lot better than a DSL connection, which has limited upload speeds, limiting your site anyway. For example if you do use a CDN I would be happier with the static assets being distributed from their POP in Australia than a DSL connection in New Zealand. Once this is done you then start thinking about failover. Your current environment is not good enough to get you in "production" these days.

A: Interesting, this is the standard line I / we have heard. I just don't really understand it. Having a temperature controlled and humidity controlled environment and everything else that goes along with it quite a lot. But I have had PC's on my desk running 24/7 for years without problems at times, and sometimes they fail. So it looks like a lot of effort is generally put into making something ultra-reliable (which is really hard) and even then it is not always successful: e.g. just look up AWS outgoes and you will see some, e.g.: http://www.zdnet.com/amazon-web-services-suffers-outage-takes-down-vine-instagram-flipboard-with-it-7000019842/ Or Eg the Christchurch earthquake. Colocation in a good server place in Christchurch would have down nothing when the quake struck. Again having some cloud failover of a server stuck in a garden shed would have been better.

So I would think the first thing to do is have decent failover. Once you have that then all the other bits don't need to be ultra-reliable... You can just move them around easily as the need arises. And we are very close to this hence my questions...

Re: colocation. If it costs say NZ$500 a month more to colocate our 4 X 1U servers, and we are doing this to save 5 minutes downtime a month (assuming we go down that often due to hardware failure, etc which we are not even close to doing.) Then that implicitly values are business as being worth NZ$100 a minute (well there is the reputation hit as well if we are off line for 5 to 10 minutes or so) but still at NZ$100 a minute that works out at our company value being 100 * 60 mins / hour * 24 hours / day * 365 days / year at around NZ$52 million yearly sales. We don't have that. Not even close yet :) Of course we can't have spotty service and if we reach a patch that things are going bad with our servers or our config then we just simply easily flip over to the cloud and continue...

Thus I would see having cloud failover (or at least all the scripts and launching capabilities to do it fully automated and easy to run) very important. Then once we have this why would we not set it up to do so?

Also the colocation bandwidths we were offered were not that great. Certainly nothing like the 100MBs down / 50 MBs up you can get on fibre...

Note if we were in the US then this would be different. It appears you can colocate in many places for a price not much above what you could do this from your own home. However in NZ things are quite expensive. I just looked up on VPS city for a colocation and it worked out at around NZ$270 for one of our 1U servers. (I am not sure if the 4U below is a single unit or we could get away with 4 units in there but this comes out at:

Server Colocation - 4U Rack Unit Power Required : 1500-2000 Watt PSU $150.00 NZD LAN Port Speed : 1Gbps Ethernet Port $25.00 NZD Extra LAN Port : Not Required Internet Bandwidth : 100Mbps Shared $50.00 NZD NAS Storage : Not Required IP Addresses : /29 IP Address Range (6 Usable IP's) $30.00 NZD USB Rotation Service : Not Required Web based IP KVM : No $394.95 NZD + $49.95 NZD Setup Fee Subtotal : $444.90 NZD GST @ 15.00% : $66.74 NZD Total Due Today $511.64 NZD

That is not cheap! We could better spend the $6000 a year on other things I think. Maybe we are wrong and then we will colocate our servers later, but since we can fail over to the cloud we can simply run there for a while we seek other options, if things go badly for us :)

So in conclusion, it seems to me that easy failover and being able to be agile in configuring our infrastructure around is quite important and probably one of the first things to architect in as we expand...

Cheers,
Jas

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1102620 5-Aug-2014 13:41

With the home internet connection it can potentially go down for two days at a time, not just 5 minutes. Once you get your cloud backup going that'll account for it.

F5 LTM's are extensively used, I've worked with them in a bunch of places. I'd never heard of the GTM, that could be a good option. Some F5 solutions can be run as VMs rather than dedicated hardware, which might be an option for you. Put that one virtual machine somewhere in the cloud or hosted, it can check every minute to see if the main site is up and if not initiate failover. There are probably free software versions as well.

jnimmo

1098 posts

Uber Geek
+1 received by user: 255

#1102638 5-Aug-2014 14:14

Not wanting to be pushy or negative about your decision, but there are many other advantages to co-location not mentioned above

- Security against fire and theft, any serious datacenter will have fire supression, and be much less vulnerable to break ins than a home.
- Good, stable power - high quality UPSes with onsite generators
- Much better internet connectivity - you may not realise that although your fiber connection may be 100/50, there is probably only a 2.5Mbps committed information rate on that link, and home ISPs probably have no CIR.
- Price - shop around, you'll be able to find much better deals than that for colocation
- Actually, you mentioned that being in a datacenter in christchurch wouldn't have helped during the earthquake - I'm not aware of any having unexpected downtime because of the earthquakes
- In the event of a natural disaster, who is going to do the migration to the cloud? You will be unlikely to get any internet connectivity to do it yourself. If it is all automated, how are you going to avoid false failovers if your home internet connection goes down or gets congested?
- Scale - if the business grew, you can grow in a datacenter, sure you can at home too but the risk gets that much greater the more you are hosting

Whatever you do, like others have mentioned- a load balancer would be preferable otherwise you might get some users pointing to the onsite and some pointing to the cloud etc.

EDIT: i.e. HD currently are offering 4U, a /30 IP address for $60/month. I can't speak for their reliability etc but probably better than a home

Dyson

Shop now for Dyson appliances (affiliate link).

jasnz

41 posts

Geek
+1 received by user: 3

#1102705 5-Aug-2014 16:30

@timmmay , @jnimmo I guess I have not been clear but our monitoring solution which determines failover launch is not at our office / home. It is a small DigitalOcean droplet that is monitoring via nagios our setup, there is then a python script which we are going to tune for when to launch in the cloud. We will launch the cloud versions proactively after around 2 minutes of down time. All the VM's take around 3 minutes to come up on digital ocean. (I assume this would be the same sorts of speeds if we were using rackspace or AWS). Then at the 5 minute mark or so we see if things are still down and if so we fail over.

Fire suppression, again why do we really need that? How much more dangerous is it to have servers than a desktop that is always on? Are servers known to explode more often or something? Of course if you are in a room with 1200 servers or something then I can image through odds alone it would need all those fancy things.

It will be *all* automated. Avoiding the odd false fail over is likely not that important. If our link is congested then it is likely better to fail over anyway. Maybe some exchange in Dunedin is having problems, etc. Why let users struggle with a congested link? better to fail over... Ie if we false failover we haven't lost much. Of course if we do this every day or something then we will look at it (in fact we will look at it seriously any time it fails over of course) but it certainly isn't a huge problem.

I am not opposed to running for weeks or even maybe a month in the cloud it is just a lot more expensive. Failing over for an hour is very cheap, i.e. in the order of 60 cents per hour (which is still of course US$475 per month) , so if we have to use the cloud due to growth or something, then we do that and then maybe try and cut a deal with some colocation place get more hardware...

In any case thanks for the comments of essentially using a cloud load balancer / redirect through to our site so when we flip the back end over to a different site things still resolve correctly. I think we will try using CloudFlare then and pound on that configuration with our testing extensively...

I just checked out HD and it is not $60 per month it is more like NZ$600. Here was the page I used: https://my.hd.net.nz/cart.php?a=confproduct&i=0

How did you get to NZ$60?!? is there some super special I missed somewhere? $500 to $600 was the price range I typically saw when looking around in NZ...

And here was the configuration...

[code]
Colocation - 4U Rack Unit 4U Rack Unit $200.00 NZD » Electricity Required: 1500-2000 Watt PSU $150.00 NZD » Internet Port Speed: 100Mbit Ethernet Port $50.00 NZD » National Bandwidth: 2Mbps CIR / 5Mbps PIR $72.00 NZD » International Bandwidth: Unlimited Shared Pool [included] $0.00 NZD » IP Addresses: Single IP Address [included] $0.00 NZD » APC Remote Hands?: No $0.00 NZD » USB Rotation Service: No USB Service [included] $0.00 NZD » Extra Ethernet Port (NIC): Not Required $0.00 NZD » Server Management: Not Required $0.00 NZD » Advanced Monitoring: Not Required $0.00 NZD » Managed Firewall: Not Required $0.00 NZD Setup fee (excluding GST): $0.00 NZD Monthly Total (including GST): $542.80 NZD
[/code]

jnimmo

1098 posts

Uber Geek
+1 received by user: 255

#1103192 6-Aug-2014 12:10

Fire won't be caused by the servers - it would be caused by leaving an iron on, an electrical fault, a heater catching something.

You're right - there was a special listed on HD website yesterday, but not today! I should have screenshotted it for you. It was $40 for 1 or 2U, $60 for 3 or 4U.
http://www.hd.net.nz/business-solutions/datacentre-services/colocation/

I suspect you'll generally get better colocation pricing if you call up a few places rather than looking at their online price.

Anyway, if you've got a setup which lets you easily failover to the cloud and failback just as easily then that's cool, but if it ends up being complicated then dealing with server problems will become time consuming instead of focusing on your core business - but like you say you can just see how it goes

EDIT: Re cloudflare, may need the paid version if you're dealing with SSL

Ragnor

8279 posts

Uber Geek
+1 received by user: 585

Trusted

#1104137 7-Aug-2014 14:55

Why would you run it from home and failover to some form of cloud/hosting provider instead of just running it from the cloud/hosting provider in the first place?

Running from home just has too many single points of failure (power, internet, fire, theft etc).

Yes it costs money to host things properly but presumably you have a business model or plan in mind where when you have x customers it exceeds the y cost of running the servers. Most new online services run at a loss initially and require investment of time and money to build reputation and customers. Skimping on hosting will damage your reputation at some point most likely.

The whole reason Azure, S3 or even a small VPS at some hosting provider exist is to allow you to start small/cheap and scale up if you don't want to start with a high cost initial cost of owning your own servers and co-locating them.

freitasm

BDFL - Memuneh
80657 posts

Uber Geek
+1 received by user: 41065

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#1104143 7-Aug-2014 14:59

Remember, in case of fire your house insurance probably won't pay for your servers and if the fire was started by your servers they might not even pay anything (unless you declared to them you run a business from home).

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

1 | 2 | 3