AWS VPC NAT Instance Failover and High Availability

Amazon Virtual Private Cloud (VPC) is a great way to setup an isolated portion of AWS and control the network topology. It is a great way to extend your data center and use AWS for burst requirements. With the latest VPC for Everyone announcement, what was earlier "Classic" and "VPC" in AWS will  soon be only VPC. That is, every deployment in AWS will be on a VPC even though one might not need all the additional features that VPC provides. One might eventually start looking at utilizing VPC features such as multiple Subnets, Network isolation, Network ACLs, etc.. Those who have already worked with VPC's understand the role of NAT Instance in a VPC.

When you create a VPC, you create them with multiple Subnets (Public and Private). Instances launched in the Public Subnet have direct internet connectivity to send and receive internet traffic through the internet gateway of the VPC. Typically, internet facing servers such as web servers are kept in the Public Subnet. A Private Subnet can be used to launch Instances that do not require direct access from the internet. Instances in a Private Subnet can access the Internet without exposing their private IP address by routing their traffic through a Network Address Translation (NAT) instance in the Public Subnet. AWS provides an AMI that can be launched as a NAT Instance. Following diagram is the representation of a standard VPC that gets provisioned through the AWS Management Console wizard.

Standard Private and Public Subnets in a VPC

The above architecture has

• A Public Subnet that has direct internet connectivity through the Internet Gateway. Web Instances can be placed within the Public Subnet
• The custom Route Table associated with Public Subnet will have the necessary routing information to route traffic to the Internet Gateway
• A NAT Instance is also provisioned in the Public Subnet
• A Private Subnet that has outbound internet connectivity through the NAT Instance in the Public Subnet
• The Main Route Table is by default associated with the Private Subnet. This will have necessary routing information to route internet traffic to the NAT Instance
• Instances in the Private Subnet will use the NAT Instance for outbound internet connectivity. For example, DB backups from standby that needs to be stored in S3. Background programs that make external web services calls

Of course, the above architecture has limited High Availability since all the Subnets are created within the same Availability Zone. We can avoid this by creating multiple Subnets in multiple Availability Zones.

Public and Private Subnets with multiple Availability Zones

• Additional Subnets (Public and Private) are created in one another Availability Zone
• Both Private Subnets are attached to the Main Routing Table
• Both Public Subnets are attached to the same Custom Routing Table
• Instances in the Private Subnet still continue to use the NAT Instance for outbound internet connectivity

Though we increased the High Availability by utilizing multiple Availability Zones, the NAT Instance is still a Single Point of Failure. NAT Instance is just another EC2 Instance that can become unavailable any time. The updated architecture below uses two NAT Instances to provide failover and High Availability for the NAT Instances

NAT Instance High Availability

• Each Subnet is associated with its own Route Table
• NAT1 is provisioned in Public Subnet 1
• NAT2 is provisioned in Public Subnet 2
• Private Subnet 1's Route Table (RT) has routing entry to NAT1 for internet traffic
• Private Subnet 2's Route Table (RT) has routing entry to NAT2 for internet traffic

NAT Instance HA Illustration

A script can be installed on both the NAT Instances to monitor each other and swap the routing table association if one of them fails. For example, if NAT1 detects that NAT2 is not responding to its ping requests, it can change the Route Table of Private Subnet 2 to NAT1 for internet traffic. Once NAT2 becomes operational again, a reverse swapping can happen. AWS has a pretty good documentation on this and a sample script for the swapping.

Apart from HA, the above architecture also provides better overall throughput, since during normal conditions, both NAT Instances can be used to drive the outbound internet requirements of the VPC. If there are workloads that requires a lot of outbound internet connectivity, having more than one NAT Instance would make sense. Of course, you are still limited with one NAT Instance per Subnet.

Another AWS outage. And how do you overcome it?

Disclaimer I: I don't work at AWS :) but seriously think that building your own cloud (as some people call it) or setting up a private cloud doesn't really address anything apart from posting a comment in article about an outage.

Disclaimer II: Though not every one admits, some people who have not been affected during outages have been lucky. They are lucky that their Instances and EBS Volumes weren't in the affected AZ. Though people spend time on building sophisticated architecture, it is only during such outages, the architecture is put to real test. Just like how AWS themselves uncover new failure areas, the architecture will also open up new failure points during outages. And you need to work on improving them.

We just had another outage in AWS. And this time too there were a good number of popular websites that were affected by the outage - Reddit, Foursquare and Heroku are few from the list. AWS has published a detailed analysis of what caused the outage. The previous outage was on June 29th which was even more widespread taking down Netflix and others. AWS did provide a detailed analysis on what went wrong and how they are taking actions to prevent such incidents in future.

The aftermath and reactions of such outages can be classified in to two - from people who understand Cloud and people who don't. No one likes outages. But people who understand Cloud, they clearly see outages as an opportunity to build better systems. Better architectures. And there are people who would start blaming it on the Cloud. Saying things like "this is the reality behind Cloud", or even say it is better to own than rent. Public Cloud versus Private Cloud is a huge debate. And if you have been in the Cloud market you will even hear more types of Cloud - like Hybrid Cloud.

Why Cloud?

Be it public cloud, private cloud or hybrid cloud. Be it Open Stack or Cloud Stack. One should definitely know why they need a Cloud strategy. Are you getting into Cloud for

• Scalability
• High Availability
• Cost
• Getaway from infrastructure management
• Everyone is getting in to Cloud
• My resume needs to show I know Cloud

Be honest in your answer to this evaluation because the benefits vary depending upon where you fit in to the above classification. Of course, every one is concerned about cost and would like to keep costs low. But there are other invisible costs that doesn't stand out normally. The cost of slow performing website, the cost of a lost customer are not seen directly as against infrastructure cost. And designing scalable systems within a specific cost is an even bigger challenge.

Hardware fails
It has to be understood that hardware fails all the time. The Cloud Service Provider doesn't setup some magic data center where hardware doesn't fail. And if some one advises you towards private cloud because of such outages in public cloud, be wary that such hardware failures will happen in private cloud setup as well. In contrast, because of the scale at which the public cloud service provider is operating, there is massive automation in place. There are systems what will detect failures well ahead of time and act on its own. There are disk's that reach their lifetime and they get replaced automatically. Those disks will go through a purge cycle to be destroyed completely as per the policy of service provider. Servers get replaced, security systems get updated and network gets upgraded. All these are happening continuously and transparently. Probably as an every day routine in all the data centers. So such things need to happen if you are going to setup a private cloud

Efficiency
Amazon.com is successful because they mastered retail and are operating it at cut throat efficiency. They have successfully replicated the same model into AWS. At any given point of time, AWS tries to operate their data centers at maximum efficiency. That's where they make the most of the buck. By continuously operating at that efficiency and because of the economies of scale, they are also able to pass on the benefits as cost reduction to end customers (without even any significant competition). Whether a private cloud setup or another service provider that would operate at that efficiency and continue to automate is something that one has to evaluate.

With that as a background, here are some thoughts on keeping your website highly available.

Availability Zones
I read your mind. "Yeah. I know. You gonna tell me to use multiple Availability Zones right?". Yes, almost the first thing that you will notice in any article which talks about overcoming outages is utilizing multiple Availability Zones (AZs). So is there something new that I am gonna talk about? Yes.

• AZs are data centers and they are isolated. Some AZ's might have multiple data centers supporting them. In the sense that they have their own independent power, diesel generators, cooling and outbound internet connectivity. And they are not isolated across coast or cities. They are just near by data centers having low latency network between them 
• AZs have low latency between them. But there is a latency. And it might be considerable for you depending upon the number of concurrent requests that your application is expected to handle. So if you have the application servers distributed across AZ-1 and AZ-2 and database only in AZ-1, the AZ-2 application will experience a latency when talking to the DB

So make use of multiple AZ's. If you are planning for High Availability at web layer you will have to have a minimum of two EC2 Instances. Launch them across two different AZ's. Each region in AWS has at least two AZ's and any new regions that will be launched by AWS will have at least two AZ's. If you look at the latest outage, it was specific to a particular AZ. So infrastructure spanning across multiple AZ's weren't completely down. But then why Reddit went down? I will talk about that a little later.

Elastic Load Balancing

Elastic Load Balancing (ELB) is the load balancing service from AWS. When you look at ELB, there are two things that you should pay attention to. One - ELB servers. The EC2 Instances that AWS will be using to run the ELB service. Two - the servers behind the ELB which are your web/app servers.

• ELB, being a service from AWS has HA, scalability and fault tolerance built into it. The Instances that AWS will be provisioning to support the ELB service will span across multiple AZ's and will automatically scale on its own. All of this are abstracted to us and it is an obvious guess that AWS will utilize multiple AZ's by themselves

Configuring ELB's to distribute traffic across multiple AZ's

• For your web/app Instances that sit behind ELB, you have an option to specify the AZ's that the ELB has to support while creating it. For example, when you create an ELB in US-East-1 region, you can mention "us-east-1a" and "us-east-1b" to be supported by the ELB. ELB will now distribute the incoming traffic between the AZ's. Now why is this important? Because, if you specify multiple AZ's for ELB, it results in two things
• The Instances that ELB creates for load balancing will created in all the AZ's that you mention. This means there is a AZ level HA for load balancing Instances itself
• If there is an AZ level failure, then the ELB service will route the traffic to the load balancing Instances in the other AZ thereby continuing the service operation
• So why ELB service was interrupted during the outage? It appears that there was a software bug in the traffic diversion software which resulted in traffic still being routed to the load balancing Instances in the affected AZ itself. Looks like AWS is fixing that bug pretty soon

AutoScaling

AutoScaling will automatically scale up/down your Instances according to the pre-configured triggers (based on CPU, Network, etc...). AutoScaling works in conjunction with ELB where it also gets signals from ELB. AutoScaling can be configured with multiple AZ's as follows

as-create-auto-scaling-group AutoScalingGroupName  --availability-zones  value[,value...] --launch-configuration  value  --max-size  value  --min-size  value [--cooldown  value ] [--load-balancers  value[,value...] ] [General Options]

Once configured, when AutoScaling launches/terminates Instances it will distribute across the configured AZ's. So if the scale up condition is "2" (launching two Instances), and AutoScaling has been configured with "us-east-1a" and "us-east-1b" then AutoScaling will launch one Instance in each AZ.

Go HA with 3 not 2

We know that we have to use multiple AZs. And regions like US-East has 5 AZs. How many do you use in ELB and AutoScaling? It is always better to use three AZ's at a minimum. If you are using only two AZ's and if one of the AZ is down, then the other AZ is receiving traffic twice its normal since ELB will redirect all the traffic to the other AZ. Your AutoScaling is configured to scale up by 1 and now an AZ is down as well. So by the time your AutoScaling reacts you are probably flooded with loads of traffic. And,

• You aren't alone - you are not the only one affected. There are 100s of other customers whose 1000's of Instances are affected. And they are all going to spin up Instances in an alternate AZ. Now AWS might run out of Instances in the alternate AZ's because they are already operating at 20% reduced capacity (in case of US-East)
• API error rates - normally with any such outage, the APIs start throwing up errors. You provision the infrastructure through API's or through AWS Management Console which is a UI wrapper around APIs. During an outage, AWS definitely doesn't want the other AZ's to be flooded with requests which may cause a spiral effect and make the other AZ's also reach critical infrastructure thresholds. And AWS API's themselves will be running in multi-AZ mode and probably are affected because of non-availabilty of an AZ. In such cases, I am sure there is a lot of throttling that AWS will employ at the API level to protect the API from customer requests causing a spiral effect in other AZ's
• Other AWS services - there are other services like ELB, Route53, SQS, SES, etc...which are services that will have HA built on to the them. For example, the ELB service itself to support its load balancing service not going down during an outage, might provision it's own ELB Instances in an alternate AZ's

So in effect, though multi AZ setup is the recommended setup in AWS, it is always better to use more than two AZ's (if a region has). In case of US-East, you can, say, utilize three AZ's in your AutoScaling configuration and ELB settings. The probability of two AZ's going down at the same time is less and there is a better distribution of traffic even if one AZ is down. Of course, it can be argued that we are just avoiding an eventuality. Yes, but during an outage the primary goal of anyone would be to sustain operations (even at reduced capacity) during the outage and return to normalcy at the earliest.

Understand AWS Services that are multi-AZ
The following services from AWS are inherently multi-AZ and are highly available. During an outage, these services will be still available and they would have internally failed over to another AZ.

• Route53 - DNS
• Elastic Load Balancing - Load Balancer
• CloudFront - Content Distribution Network
• Amazon S3 - Storage Service
• DynamoDB - NoSQL Database Service
• SQS - Simple Queue Service
• SES - Simple Email Service

RDS Multi-AZ supports multiple AZ's but you unlike the other services listed above, the application will notice the RDS failover. During an AZ outage, when RDS internally triggers a failover to the standby instance in another AZ, the application will lose connectivity to the RDS endpoint. Depending upon the volume of data and transactions that were happening at the time of failure, the failover time could be around 3-5 minutes.

But even Multi AZ RDS Instances were affected
Yes, and that again seems to be because of bugs in the automatic failover software. RDS relies on EBS for database storage and this outage started with problems in EBS Volumes (which was a result of improper internal DNS propagation). If the EBS of primary RDS Instance is stuck it would result in an automatic failover to the standby immediately. The failover didn't happen immediately because the software didn't anticipate the type of EBS I/O issue that happened during this outage. But for some of the RDS Instances, looks like the primary lost connectivity to the standby and the primary was still accepting transactions. This should have immediately triggered a failover. Because the standby is not receiving updates and can result in data inconsistency. But again, there seems to be a software bug which prevented such a failover. This has to be addressed immediately because RDS is a service where AWS has developed synchronous replication to the standby which isn't available in native MySQL. And that needs to definitely work perfectly all the time.

Perceived Availability
Different applications need different levels of availability. And it is easier to achieve 99.99% of availability. But pushing things forward, every extra nine comes with additional complexity in the architecture and cost. As application architects, it is also required to understand how the application end users perceive availability of a system. For example, let's consider a photo sharing website built on top of an open source CMS. If you break down the different areas in the architecture, you will be looking at the following

• A DNS service
• Content Distribution Network - serving static assets from the nearest edge location
• A front end web load balancer
• A page cache - to cache the HTML
• The web server - storing templates, etc...
• Object Cache - to cache the content, store user sessions
• Database - storing user data and other transactional data

AWS Services and High Availability

With such kind of integration of services that Amazon offers, you have certain portions of your website still available during an outage. For example, your users can still view photos (through CDN) but might not be able to upload new photos (at normal throughput) during outage. Users will still be able to see data from cache which is not up-to-date. Likewise, each area in the application needs to be looked at from a failure point of view and addressed accordingly.

Improve
As architects, one needs to continuously ask questions on failures in each area/tier in the architecture. What if I pull out that web server behind the ELB, will my users sessions be lost? What if that EBS volume becomes unresponsive? What if my RDS Instance is flooded with database connections? Such probing can lead in to small improvements in the architecture which can increase the overall site availability.

So why did Reddit and Heroku go down?
I am sure the engineers at Reddit and Heroku are well aware of all these and would have designed an architecture that scales and tries to keep the service up and running all the time. While, there is no information on what went really wrong, we all know that software systems are not perfect at all the time. No one designs systems that anticipates all probabilities of failure and has a fool proof mechanism to address them. Only when the system is put to use, pushed to boundaries and scales it opens up new flaws in the architecture that once we thought was perfect. I am sure Reddit is learning from areas in their architecture that went down during the outage and will make it better. Just like how Netflix wrote an article on what lessons they learnt from the previous outage.

Making your apps truely HA on AWS - lessons from the recent outage

During last week, AWS had an outage in one of their data centers. Unlike last year, this year's outage had lesser impact on only one Availability Zone in the US-East region of AWS. None of the customers for whom we had setup their infrastructure in AWS suffered any downtime during this outage. One of the customer panicked and reported Instances becoming unreachable only to find that the Instances in other AZs were running fine and the website continued to be available.  What was more interesting was, the next day, one of India's leading entertainment content provider got in touch with us and wanted us to evaluate their existing AWS infrastructure. While I was talking to the customer, I casually asked the prime reason for this evaluation. He mentioned that their website is completely hosted in AWS and the website went down during the outage. And he was not aware that the website was down until some one from their team accidentally found the website to be down.

Here's the update from AWS on what went wrong. The full information is available on AWS Service Health Dashboard page but might become difficult to access by next week as the date widget moves ahead

"At approximately 8:44PM PDT, there was a cable fault in the high voltage Utility power distribution system. Two Utility substations that feed the impacted Availability Zone went offline, causing the entire Availability Zone to fail over to generator power. All EC2 instances and EBS volumes successfully transferred to back-up generator power. At 8:53PM PDT, one of the generators overheated and powered off because of a defective cooling fan. At this point, the EC2 instances and EBS volumes supported by this generator failed over to their secondary back-up power (which is provided by a completely separate power distribution circuit complete with additional generator capacity). Unfortunately, one of the breakers on this particular back-up power distribution circuit was incorrectly configured to open at too low a power threshold and opened when the load transferred to this circuit. After this circuit breaker opened at 8:57PM PDT, the affected instances and volumes were left without primary, back-up, or secondary back-up power. Those customers with affected instances or volumes that were running in multi-Availability Zone configurations avoided meaningful disruption to their applications; however, those affected who were only running in this Availability Zone, had to wait until the power was restored to be fully functional."

Utilize Multiple Availability Zones
Or AZ's in short. AWS provides multiple Availability Zones in each region. Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region. Additionally in this FAQ answer, AWS clearly states that common points of failures like generators, cooling systems are not shared between AZs and each AZ is physically separate and independent. This recent outage was primarily because of a fault in the power distribution system and since AZs are isolated only one of the AZ (us-east-1c) in the US-East region got affected. Remaining AZ's were running smoothly without any interruption. Hence it is prudent to have the infrastructure distributed across multiple AZ's. Distribute your EC2 Instances across multiple AZ's and bring up your HA quotient.
Note: You need not worry about any performance issues because of Instances deployed across different AZs. All AZs have very low network latency between them and can be ignored for web applications.

Configure ELB to use Multi AZ
AWS Elastic Load Balancer (ELB) can be configured to work across multiple AZ's. This way you can launch EC2 Instances in multiple AZs and have them tied to a single ELB. In event of a failure of one AZ, you will still have Instances running out of the other AZs and ELB will failover to them and your website will continue to be operational. When you create an ELB through the AWS Management Console you will not have the option to specify the AZ distribution. However, when you add Instances running in multiple AZ's to the ELB, the ELB automatically gets configured to work with multi AZs.

If you are using the command line interface or the API's, while creating the ELB itself you can mention the AZ's that the ELB needs to handle:

elb-create-lb LoadBalancerName  --availability-zones  value[,value...]  --listener "protocol=value, lb-port=value, instance-port=value, [cert-id=value]" [ --listener "protocol=value, lb-port=value, instance-port=value, [cert-id=value]" ...]  [General Options]

Additionally you create an AutoScalingGroup and mention the same list of Availability Zones, you are automating the whole failover process.

as-create-auto-scaling-group AutoScalingGroupName  --availability-zones  value[,value...] --launch-configuration  value  --max-size  value  --min-size  value [--cooldown  value ] [--load-balancers  value[,value...] ] [General Options]

In this case, the moment the Instances in one AZ went down,

• AutoScaling would have tried to launch new Instances to replace the failed ones
• AWS API's would have thrown errors due to AZ unavailability
• AutoScaling would have launched the new Instances in another AZ

RDS Multi AZ
Utilize the Multi-AZ feature of RDS where RDS will run a standby database Instance in another AZ in addition to the primary database Instance. In the event of a primary failure, RDS will internally initiate a failover to the standby and the database will continue to serve the traffic. AWS has mentioned in the RDS status dashboard that during the outage none of their Multi-AZ deployments were affected (failover got triggered). Only the Single AZ RDS had to be recovered by AWS team.

"As described in the EC2 post, a number of EBS volumes in the affected Availability Zone were impacted, disabling many Single Availability Zone (Single-AZ) RDS instances. Multi-AZ RDS instances were able to immediately begin failing over to other healthy Availability Zones. We started the recovery process for the affected Single-AZ RDS instances after EBS APIs became available at approximately 10:40 PM PST. Upon completing storage volume checks, we began processing EBS recovery volumes and recovered most of the affected RDS instances by 5:00 AM PDT on June 15th."

There are couple of things to note here:

• Unlike ELB, RDS failover is not instant. In simple words, the front end application that connects to the RDS will see a down time when the failover happens. During the time RDS completely fails over to the standby, connectivity to the RDS endpoint will not be available. We have seen that the time taken is approximately 5-7 minutes for around 100GB of data. But with more data probably it might take little longer.
• When it comes to database the impact is different when compare to, say, web servers. With database one will worry more about the data consistency and data durability than just availability. So when AWS talks above about recovering the Single RDS, it is actually the recovery of the EBS behind the RDS. These Single RDS EBS became unavailable and since there was a power failure it potentially resulted in a write corruption. Hence customers were asked to restore to the nearest point in time snapshot. With Multi-AZ such data corruption issue will not arise since there will be an entire replica on another AZ and data is written on both concurrently on a commit.
• Of course, Multi-AZ comes with additional cost. Probably double. The cost of running a Single RDS large Instance in US-East is about $300/mo give or take. In case of a Multi-AZ setup, this cost will double and AWS would be charging around $600/mo. Availability, consistency and data durability versus cost is something individual application owners need to decide

Stateless Servers

Apart from the persistent store (database) of the application, it is recommended to make all other layers in the infrastructure stateless. Web and application servers specifically needs to be designed to be stateless. Most of the applications store the web session data on the web server memory itself. Similarly there are other assets like user uploaded files, server and application log files that typically reside on the web/app servers. All of these assets needs to move to a central repository. For example, session information can be stored on caching servers or databases and files can be periodically rotated to, say, Amazon S3. Once we design such an application, then it is easy to make modifications in the infrastructure to maintain HA and also to scale on demand. Spreading the infrastructure across multiple AZ's can definitely help you achieve HA. But the failover will not be transparent to the user if the session information is stored in memory. If 100s of users session state is maintained in Instances in AZ1 and that AZ becomes unavailable, the front end Load Balancer will immediately failover to the Instances running in AZ2. But users' session state will be unavailable in these AZ2 Instances and will probably be asked to login again.

With these simple principles coupled with utilizing AWS building blocks effectively one can definitely build an infrastructure that will be truely Highly Available. Of course, these are lessons that one learn as Cloud delivery model matures and as architects one has to keep fine tuning the architecture post live as well.

Note: All the above thoughts are to increase uptime during problems with a single AZ. Last year we saw a different problem in AWS where a configuration change coupled with the architecture of EBS an entire region (US-East) had a service disruption. Infrastructure architecture can be designed to withstand such huge disruption as well. However those come with very high costs of replicating data over the internet. And in most of the cases the cost of operating such an infrastructure outweighs the benefits.