Few more AWS updates

RDS Provisioned IOPS
Just as I mentioned in previous article that Provisioned IOPS (PIOPS) will be naturally extended to RDS as well, AWS just released PIOPS feature for RDS. You can read the full update here. Couple of points to be noted

• This is currently available only for newly provisioned RDS Instances. If you create a new RDS Instance you will be able to specify the required IOPS
• At the time of writing this article, specifying PIOPS is not yet available in Singapore region. It is available in US-East, US-West and EU-West regions
• As the article notes, AWS will release an automate way to migrate existing RDS Instances to PIOPS RDS. I believe it will come with its own down time. I am wondering if AWS will make it without an endpoint change so that there will not be any change in the application side
• For now, if you want to migrate the RDS Instance to PIOPS based, you can follow the below steps
• Take an application downtime and stop all transactions hitting the RDS Instance
• Create a new DB Snapshot from your existing RDS Instance
• Create a new RDS Instance from the Snapshot and mention the required PIOPS
• The new RDS Instance will come with its own endpoint which needs to specified in your application

Non BGP device support in VPC

AWS Virtual Private Cloud (VPC) is a great way for enterprises to extend their data center to AWS and use AWS for burst capacities. Workloads can be setup in public and private subnets within VPC with a secured IPSec tunnel to existing data centers. For a VPN connectivity there are two sides of the VPN tunnel - one at AWS side and the other at the Customer side. The AWS side of the tunnel is automatically taken care by AWS whenever we create a VPC. But on the customer side, one needs to have a compatible device that can talk to AWS VPC. Till now, these customer devices needs to support BGP protocol to successfully establish a VPN connection with VPC. Not anymore. AWS has released a new static routing feature that can create VPN connections from VPC to customer devices which do not support BGP protocol.

• Most of the enterprises will have a device which support BGP protocol. But in our experience, we have seen that the network team sometimes find it hard to configure. Especially when the enterprise has existing VPN connections to other data centers. A lot of care should be taken while adding another VPN connectivity and advertising the right set of IP ranges. Otherwise, it might result in loops where traffic can be diverted elsewhere bringing critical systems down
• With Static routing, testing a VPC setup becomes very easy. Most of the offices and corporate branches might have devices which support IPSec but not BGP. In such cases, this comes pretty handy to test out a VPC setup. It would be worthwhile to quote a real life example that we faced. As a practice, we generally test out all architecture recommendations before we make the recommendations to our customer. For one of the large enterprise customer, the architecture involved VPC with VPN connectivity to their data center. We were able to successfully create the VPC architecture in AWS and configure all the routing. But to test it out, we needed to have a device at our office that supports BGP. We had a simple device that supported IPSec but not BGP. We had to procure a new device and continue with the testing. This will not be the case anymore
• Here's the complete list of devices that AWS VPC supports - both BGP (dynamic) and non BGP (static) ones

SQL Server RDS in VPC

RDS for SQL Server is now available within VPC. This has been long pending from AWS. VPC is the way any typical enterprise would like to move their workloads into AWS and use AWS for their burst capacities. With enterprises, SQL server is still the widely used database and non-availability of SQL Server RDS within VPC has been a long pending issue. Not anymore. We can now provision a RDS SQL Server within a VPC and let AWS manage the database such as patching, upgrades and automated backups.

• Multi-AZ deployments are not yet available. Unlike RDS MySQL, RDS SQL Server still does not support Multi-AZ deployments. So in case of any primary failure, we will have to recover the DB Instance from the latest backup (latest automated snapshot)
• Read Replicas for RDS SQL Server are not yet available.
• Scaling Storage - MySQL RDS allows us to scale the storage space on demand. If we start with 100GB MySQL RDS and once we near the 20% available free space, we can always increase the storage space to, say, 200GB. RDS will automatically increase the underlying storage space without causing any database downtime. This is currently not available for RDS SQL Server
• RDS SQL Server currently supports only SQL Server authentication. Windows authentication is not yet supported

AWS updates that makes architects happy

AWS keeps adding a lot of features to its existing services and also keeps launching new services. With the pace of innovation from AWS and the rate at which new services and updates are released, one may lose track of them pretty easily. As architects it is essential to know all of these all the time so that one proposes the right solution for a given requirement. Here are some of the features and services that has made the lives of architects easy and can help in simpler infrastructure design addressing specific needs. There is no order to this list - I am just listing them down. And here's a summary before we go detail in to each of them

High I/O EC2 Instance Type - This is a new family of Instance types from AWS in addition to the existing High Memory and High CPU Instance types. The first one in this family is hi1.4xlarge which comes with 60GB of memory and 8 virtual cores. The best part of this Instance type is the whopping 2 TB of SSB attached to them. These SSDs are the ephemeral disks that come as part of the Instance and are of course non-persistent. AWS promises around 120K random read IOPS and between 10K and 85K write IOPS. This Instance type is suitable for I/O intensive applications and running NoSQL databases. Here are couple of pointers about this Instance type

• SSD disks are ephemeral - you will lose data on Instance termination (of course if you stop and start the Instance)
• Data will be persistent between reboot like any other EC2 Instance
• You will see two 1TB volumes on the Instance. Can be striped to double the throughput
• If you are launching a Windows Instance, choose the "Microsoft Windows 2008 R2 64-bit for Cluster Instances" AMI
• Currently this Instance type is available only in US-East and EU-West regions
• And its priced at $3.10 an hour :)

I am sure AWS will be releasing lower Instance types (such as High I/O large) since not everyone will be looking to use an 4xlarge Instance. There are already some benchmarks that you can refer here and specifically netflix has performed a detailed benchmark of this Instance type.

Provisioned IOPS for EBS - EBS is great for storing persistent data and the ability to attach and detach it to/from any Instance. But from day zero the performance and throughput has been an issue with EBS. The IO throughput that one gets out of EBS is very poor and all the more it is not consistent. If the network load during that time is not heavy then one might see large throughput from EBS. If not the throughput will hover around 100 IOPS. Fast forward to now and with provisioned IOPS we can set the IOPS that we expect from EBS and AWS will guarantee the IOPS at all times. This is great news for hosting databases on EC2 (if you are not using RDS). Here are couple of pointers on this:

• Currently the max IOPS that can be set is 1000. AWS will soon increase this limit
• We can of course drive more throughput by attaching multiple EBS volumes and striping them
• Costs little extra than a normal EBS Volume - 0.125/GB against 0.10/GB. Of course the benefits will be far more than the cost difference. Guess everyone will eventually move to provisioned IOPS. Nice way to increase AWS revenue :)

I am assuming that provisioned IOPS will be extended to RDS as well in near future. Just like DynamoDB where one can specify the required throughput, AWS should let the user specify the throughput required from RDS. RDS already uses EBS behind the scenes for storing the data and hence this should definitely be in the pipeline. At least I wish so.

EBS-Optimized EC2 Instances - One thing with EBS that everyone understands is that it is network attached. Unlike local ephemeral storage which is directly attached to the EC2 Instance, there is a pipe which transfers all the data that will be written to EBS. So, even if we use provisioned IOPS to increase the throughput from an EBS, the network throughput can be significantly low that we may not utilize the increased EBS throughput; especially when the network is shared between many EC2 Instances. AWS has introduced EBS-Optimized EC2 Instances which have dedicated network throughput to the EBS that are attached to it. An EBS-Optimized m1.large Instance will have about 500 Mbits/s of guaranteed network throughput to EBS.

• Currently only m1.large, m1.xlarge and m2.4xlarge are available as EBS-Optimized Instances
• The guaranteed throughput is 500 Mbits/s for m1.large and 1000 Mbits/s for m1.xlarge and m2.4xlarge
• This should not be confused with the regular network throughput of the EC2 Instance - for example talking to another EC2 Instance. This guaranteed throughput is only for the EBS related traffic. All other traffic is through the other network pipe given to the EC2 Instance (which is not guaranteed and differs by Instance)
• Comes with additional hourly cost - Optimized m1.large will cost $0.345/hr against a normal on demand m1.large costing $0.320/hr
• Always use EBS-Optimized Instance when using Provisioned IOPS

Elastic Network Interface (ENI) - This feature is available for Instances launched within Virtual Private Cloud (VPC). When you launch Instances within VPC you normally specify the subnet to launch it in and also specify the private IP address of that EC2 Instance. If the Instance is in a public subnet, you will attach and Elastic IP so that you can access the Instance over the internet. With ENI, you no longer specify a subnet while launching the Instance. You create an ENI (which is attached to a subnet) and attach the ENI to the Instance. So what are the benefits of using ENI?

• You can attach multiple ENI (belonging to different subnets) to an EC2 Instance
• One ENI can be configured to have a public IP (Elastic IP) and private IP and the other ENI can be configured to have just a private IP
• With such a setup, you can allow traffic on port 80 through the first ENI and SSH traffic (port 22) through the other ENI
• This way you can further secure your VPN architecture by allowing SSH traffic only from your corporate datacenter

Image Courtesy - AWS

Multiple IP Addresses per Instance - with this feature release, AWS addressed the long waited requested for hosting multiple SSL websites on a given Instance. This feature is an extension of  the ENI feature, where each ENI is now allowed to have multiple public and private IPs. We can create an ENI and create multiple Elastic IPs and attach them to the ENI. The ENI will be then attached to the EC2 Instance (during launch or hot attach during running) and the Instance gets multiple IP addresses.

• The multiple public IP address and private IP addresses are attached to the ENI. And to the Instance when we attach the ENI to the Instance. The difference is that, if we terminate an Instance and relaunch a new one with the same ENI, all the mappings will remain and the new Instance will also get all the public IPs and their private IP mappings
• Available only within VPC currently
• Additional cost for each of the additional public IPs that we attach. No cost for the additional private IPs attached

ELB Security Group - When you use an Elastic Load Balancer (ELB), you will obviously place EC2 Instances behind it to accept incoming traffic. For quite a while, on the EC2 Instances Security Group, you will have to open the incoming port (to which ELB will be sending traffic) to all IPs (0.0.0.0/0). This was because of the fact that ELB will scale itself and AWS will not know at a given point of time how many ELB Instances will be running. But now, ELB's come with their own security groups. So on your EC2 Instances (that sit behind the ELB) Security Group, you will have to open up the port to only ELB's security group. This way you secure your EC2 Instances to accept traffic only from ELB.

Internal Load Balancer - You can launch ELB within VPC as an internal load balancer. By this fashion, the ELB's do not have a public IP addresses and will only have private IP addresses. Such a load balancer can be used for internal load balancing requirements between different tiers in the architecture. I wrote a separate article explaining how it can be used.

Custom Security Group of ELB - If you launch ELBs within VPC (either internal or public facing) you have an option of specifying a custom Security Group of your own. This way custom rules can be configured on the ELB's Security Group instead of using the ELB's default Security Group (which cannot be edited).

S3 Object Expiration

This feature is great for storing logs and scheduling them to be deleted after a definite period of time. When you have multiple web or application servers you will definitely consider pushing the logs generated on them to S3 for centralized storage. Even if you are running a minimal setup, you certainly do not want run out of space on your Instance. But you do not want to store the logs perpetually - one month old logs will be good enough for most of the applications. Instead of writing custom scripts that periodically deletes the logs stored in S3, use this feature so that Amazon will automatically delete them. Saves some cost.

Route 53 Alias record

Route53 is a scalable DNS service from AWS. With pay-as-you-go pricing, there are whole lot of benefits (in terms of flexibility) with Route53. It is definitely not sophisticated like other DNS service providers in terms of feature set. Be it latency based routing or weighted round robin, the service is updated with features. One feature that will be of interest is "Alias Record". Let's say you have a website called "example.com". And you are hosting the infrastructure in AWS. If your load balancer layer uses AWS Elastic Load Balancing, then you will be provided with a CNAME for the ELB. ELB internally may employ "n" number of servers to scale itself and hence it may have more than one IP. Hence AWS provides the CNAME for an ELB which may resolve to one or more IPs at any time. With that background,

• You want to create a DNS record for "www.example.com"
• You also need to create a DNS record for "example.com"; the naked domain or top level domain. Most of the users will use only the naked domain

By design, DNS does not support adding a CNAME entry for the naked domain - simply put you cannot point "http://example.com" to an ELB but you can point "http://www.example.com" to an ELB CNAME. The naked domain can be pointed to only an A-record. Having only A-records for the naked domain will introduce complexity in HA and Scalability for the load balancing layer. And if your load balancing layer employs ELB, there is no way you will know the IPs before hand. One way to address this constraint is to put a separate Apache server which can do the translation. But this will be a serious bottleneck in the architecture since most of the users will use only the naked domain and soon the infrastructure will crumble even though we have build HA and Scalability in all other layers. With the "Alias" feature

• First create a Route53 record set for "http://www.example.com" and point it to the ELB CNAME
• Create another record set for "example.com" and set the type as "Alias". Point the record value to the record set created above. Or directly point it to the ELB CNAME
• With the "Alias" option you will be able to point the record value to either an ELB CNAME or another record set that is available in Route53.

This way, you take care of both naked domain and "www" with both pointing to the load balancing layer in the AWS setup.

Of course there are a whole lot of features and updates that AWS keeps pushing across its services. The above are some of the features that helps architects while designing the infrastructure architecture. These have personally helped us build better architectures. Prior to these features, we always had to look for alternatives such has tuning, custom solutions or sacrifice on certain parameters to achieve others. I am sure there are many more such updates that we can uncover and will be coming from AWS in future. That will be for another post here.

The AdWantageS

Every customer has a reason to move in to the cloud. Be it cost, scalability, on demand provisioning, there are plenty of reasons why one moves into the Cloud. The latest whitepaper "The Total Cost of (Non) Ownership of Web Applications in the Cloud" by Jinesh Varia, Technical evangelist of Amazon Web Services provides a good insight between hosting an infrastructure in-house and in the Cloud (AWS). There are plenty of pricing models available with AWS currently which can provide cost benefits ranging from 30% to 80% when compared to hosting the servers in-house.

On-Demand Instances - this is where every one starts with. You simply add your credit card to your AWS account and start spinning up Instances. You provision them on demand and pay for how long you run them. You of course have the option of stopping and starting them whenever needed. You are charged for every hour of running an Instance. For example, a Large Instance (2 CPU, 7.5GB Memory, Linux) you will pay $0.32/hr (US-East).

Reserved Instances - let's say after you migrate/host your web application to AWS and run multiple web servers and DB servers in AWS. After couple of months, you may notice that some of your servers will run 24 hours/day. You may spin up additional web servers during peak load but will at least run 2 of them always; plus a DB server. For such cases, where you know that you will always run the Instances, AWS provides an option for reducing your cost - Reserved Instances. This is purely a pricing model, where you purchase the required Instance type (say Large/X-Large) in a region for a minimum amount. You will then be charged substantially less for your on-demand charge of that Instance. This way there is a potential savings of 30% over a period of one year when compared to on-demand Instances. The following provides an illustration of the cost savings for purchasing an m1.large Reserved Instance against using it on demand through an year

Cost comparison between On-Demand and Reserved Instance for m1.large Linux Instance in US-East

Be careful that,

• Reserved Instances are purchased against an Instance type. If you purchase an m1.large Reserved Instance, at the end of the month when your bill is generated, only m1.large usage will be billed at the reduced Instance hourly charge. Hence, on that given month if you figure out m1.large is not sufficient and move up to m1.xlarge, you will not be billed at the reduced hourly charge. In such a case, you may end up paying more to AWS on an yearly basis. So, examine your usage pattern, fix your Instance type and purchase a Reserved Instance.
• Reserved Instances are for a region - if you buy one in the US-East region and later decide to move your Instances to US-West, the cost reduction will not be applicable for Instances running out of US-West region

Of course, you have the benefits of,

• Reduced cost - a minimum of 30-40% cost savings which increases if you purchase a 3-year term
• Guaranteed Capacity - AWS will always provide you the number of Reserved Instances you have purchased (you will not get an error saying "Insufficient Capacity")

Spot Instances - in this model, you will bid against the market price of an Instance and if your's is the highest bid then you will get an Instance at your bid price. The Spot Market price is available through an API which can be queried regularly. You can write code that will check for the Spot Market price and keep placing bids mentioning the maximum price that you are willing to pay against the current Spot Market price. If your bid exceeds then you will get an Instance provisioned. The spot price will be substantially low than on demand pricing. For example, at the time of writing this article, the spot instance pricing for a m1.large Linux Instance in US-East was $0.026/hr as against $0.32/hr on demand pricing. This provides about 90% cost reduction on an hourly basis. But the catch is,

• The Spot Market price will keep changing as other users place their bids and AWS has additional excess capacity
• If your maximum bid price falls below the Spot Market price, then AWS may terminate your Instance. Abruptly
• Hence you may loose your data or your code may terminate unfinished.
• Jobs that you anticipate to be completed in one hour may take few more hours to complete

Hence Spot Instances are not suitable for all kind of work loads. Certain work loads like log processing, encoding can exploit Spot Instances but requires writing careful algorithms and deeper understanding. Here are some of the use cases for using Spot Instances.

Now, with that basic understanding, let's examine the whitepaper little carefully not just from cost point of view. Web applications can be classified in to three types based on traffic nature - Steady Traffic, Periodic Burst and Always Unpredictable. Here is a summary of the comparison of benefits of hosting them in AWS

AWS benefits for different type of web applications

Steady Traffic
The website has steady traffic. You are running couple of servers on-premise and consider moving it to AWS. Or you are hosting a new web application on AWS. Here's the cost comparison from the whitepaper

Source: AWS Whitepaper quoted above

• You will most likely start with spinning up On-Demand Instances. You will be running couple of them for web servers and couple of them for your database (for HA)
• Over long run (3 years) if you only use On-Demand Instances, you may end up paying more than hosting it on-premise. Do NOT just run On-Demand Instances if you your usage is steady
• If you are on AWS for about couple of months and are OK with the performance from your setup, you should definitely consider purchasing Reserved Instances. You will end up with a minimum of 40% savings against on-premise infrastructure and about 30% against running On-Demand Instances
• You will still benefit from spinning up infrastructure on demand. Unlike on-premise, where you need to plan and purchase ahead, here you have the option of provisioning on demand; just in time
• And in case, you grow and your traffic increases, you have the option to add more capacity to your fleet and remove it. You can change server sizes on demand. And pay only for what you use. This will go a long way in terms of business continuity, user experience and more sales
• You can always mix and match Reserved and On-Demand Instances and reduce your cost whenever required. Reserved Instances can be purchased anytime

Periodic Burst

In this scenario, you have some constant traffic to your website but periodically there are some spikes in the traffic. For example, every quarter there can be some sales promotion. Or during thanks giving or Christmas you will have more traffic to the website. During other months, the traffic to the website will be low. Here's the cost comparison for such a scenario

Source: AWS Whitepaper quoted above

• You will spin up On-Demand Instances to start with. You will run couple of them for web servers and couple of them for the database
• During the burst period, you will need additional capacity to meet the burst in traffic. You need to spin up additional Instances for your web tier and application tier to meet the demand
• Here is where you will enjoy the benefits of on demand provisioning. This is something that is not possible in on-premise hosting. In on-premise hosting, you will purchase the excess capacity required well ahead and keep running them even though the traffic is less. With on demand provisioning, you will only provision them during the burst period. Once the promotion is over, you can terminate those extra capacity
• For the capacity that you will always run as the baseline, you can purchase Reserved Instances and reduce the cost up to 75%
• Even if you do not purchase Reserved Instances, you can run On-Demand Instances and save around 40% against on-premise infrastructure. Because, for the periodic burst requirement, you can purchase only during the burst period and turn off later. This is not possible in an on-premise setup where you will anyways purchase this ahead of time

Always Unpredictable

In this case, you have an application where you cannot predict the traffic all the time. For example, a social application that is in experimental stage and you expect it to go viral. If it goes viral and gains popularity you will need to expand the infrastructure quickly. If it doesn't, then you do not want to risk a heavy cap-ex. Here's the cost comparison for such a scenario

Source: AWS Whitepaper quoted above

• You will spin up On-Demand Instances and scale them according to the traffic
• You will use automation tools such as AutoScaling to scale the infrastructure on demand. You can align your infrastructure setup according to the traffic
• Over a 3 year period, there will be some initial steady growth of the application. As the application goes viral you will need to add capacity. And beyond its lifetime of say, 18 months to 2 years the traffic may start to fall
• Through monitoring tools such as CloudWatch you can constantly tweak the infrastructure and arrive at a baseline infrastructure. You will figure out that during the initial growth and "viral" period you will need certain baseline servers. You can go ahead and purchase Reserved Instances for them and mix them with On-Demand Instances when you scale them. You will enjoy a cost saving benefit of around 70% against on-premise setup
• It is not advisable to plan for full capacity and run at full capacity. Or purchase full Reserved Instances for the full capacity. If the application doesn't go well as anticipated, you may end up paying more to AWS than the actual

As you can see, whether you have a steady state application or trying out a new idea, AWS proves advantageous from different perspectives for different requirements. Cost, on-demand provisioning, scalability, flexibility, automation tools are things that even a startup can think off and get on board quickly. One question that you need to ask yourself is "Why am I moving in to AWS?". Ask this question during the early stages and spend considerable time in design and architecture for the infrastructure setup. Otherwise, you may end up asking yourself "Why did I come into AWS?".

Yet another post on High Availability

Looks like High Availability, Scalability are becoming the most abused words / jargons these days. And Cloud Computing has helped these words reaching the common man. I was really annoyed at an article on High Availability on CloudAve. The article was posted immediately after the latest AWS outage and talks about how companies like Netflix and Pinterest have all got it wrong and how to "program" for failover. Or is that failure? Here are the main items that the article talks about

1. Programming for failover / failure. Anyone who doesn't is a fool
2. Doing an S3 sync can help you avoid outages
3. Elastic Load Balancing can ensure regions are healthy

1. Agreed. Any large scale system should definitely be designed for failure. And I am sure the folks at Netflix have not ignored this fact. Netflix is at a scale where they are probably one of the largest customer of AWS and they indeed work very closely with AWS. And to imagine what their scale is, they made their own custom AWS management solution called Asgard  and open sourced it. They were not happy with the feature set of AWS management console and built their own. And the news also appeared in AWS blog. Now, I am at loss of words if Netflix hasn't thought about using multiple AZs, load balancers, etc...That is like a bare minimum for their scale.

2. Syncing between S3 buckets. Here we are talking about Syncing large amount of objects between S3 buckets located across continents. Over the Internet. Now, this looks all the more simple task to do in paper. Anyone who has thought about the actual implementation will fairly appreciate the complexity and cost involved in doing so. Of course that may outweigh the benefits of being available during an outage. But before we get to that there are many implementation bottlenecks to be addressed. You just cannot imagine putting a Single EC2 Instance (oh yeah we are talking about HA. lets make it one Instance in each AZ) and writing a custom script to perform the sync. Bandwidth availability from a single EC2, availability of that setup during an outage and inter region internet bandwidth charges should definitely be considered to go for such a design. And in large scale architecture, companies heavily rely on Content Distribution Network (CDN) such as Akamai / AWS CloudFront to deliver the media assets. If the content is stored in S3, it is most likely served through a CDN. In such case, even if the origin server is not available (in this case an outage at AWS), the content is always available at the edge location. Apart from the media assets, companies on the likes/scale of Netflix will heavily use other technologies like NoSQL (Mongo, Cassandra), Relational Databases (RDS) and Cache Clusters. And they are probably running 100s in each. Replicating them, building failover is a slow and complex process.

3. Elastic Load Balancing is a load balancing service from AWS. It can be used to bring HA in to your architecture - within a region. You cannot use ELB to ensure regions are healthy. ELB is a service that is restricted to a specific region and operates within it. To detect multiple region health status, one needs a solution at the DNS level. Such as going for a managed DNS solution which can perform health checks between multiple data centers (geo distributed) and failover. AWS Route53 can do that. Route53 Latency Based Routing (LBR) can detect multi region latency and route traffic accordingly.

Building a Highly Available solution is not about just utilizing multiple data centers or syncing data. It involves very deep understanding of the application and the infrastructure. And how they work together. Availability is perceived by the user. Hence it varies from application to application. For example, a photo sharing website might have three major tiers - photo viewer, photo upload, commenting. If the website is completely hosted on AWS, during at outage (depending upon which AWS services are affected), some of the tiers can still be perceived to be working. Photo viewer might continue to serve the pictures from a CDN while users might not be able to add new comments. Or, new comments might be working for the users who are posting but might be delayed for the ones who are viewing (achieved through queues and cache clusters). Architects can think of various ways to build such systems and that's how one increases the overall availability of the system. It is not an one time activity but a continuous process requiring patience, careful examination and planning. Guess such architectures will quietly work behind the scenes and let others abuse words like High Availability, Scalability for eternity.

Making your apps truely HA on AWS - lessons from the recent outage

During last week, AWS had an outage in one of their data centers. Unlike last year, this year's outage had lesser impact on only one Availability Zone in the US-East region of AWS. None of the customers for whom we had setup their infrastructure in AWS suffered any downtime during this outage. One of the customer panicked and reported Instances becoming unreachable only to find that the Instances in other AZs were running fine and the website continued to be available.  What was more interesting was, the next day, one of India's leading entertainment content provider got in touch with us and wanted us to evaluate their existing AWS infrastructure. While I was talking to the customer, I casually asked the prime reason for this evaluation. He mentioned that their website is completely hosted in AWS and the website went down during the outage. And he was not aware that the website was down until some one from their team accidentally found the website to be down.

Here's the update from AWS on what went wrong. The full information is available on AWS Service Health Dashboard page but might become difficult to access by next week as the date widget moves ahead

"At approximately 8:44PM PDT, there was a cable fault in the high voltage Utility power distribution system. Two Utility substations that feed the impacted Availability Zone went offline, causing the entire Availability Zone to fail over to generator power. All EC2 instances and EBS volumes successfully transferred to back-up generator power. At 8:53PM PDT, one of the generators overheated and powered off because of a defective cooling fan. At this point, the EC2 instances and EBS volumes supported by this generator failed over to their secondary back-up power (which is provided by a completely separate power distribution circuit complete with additional generator capacity). Unfortunately, one of the breakers on this particular back-up power distribution circuit was incorrectly configured to open at too low a power threshold and opened when the load transferred to this circuit. After this circuit breaker opened at 8:57PM PDT, the affected instances and volumes were left without primary, back-up, or secondary back-up power. Those customers with affected instances or volumes that were running in multi-Availability Zone configurations avoided meaningful disruption to their applications; however, those affected who were only running in this Availability Zone, had to wait until the power was restored to be fully functional."

Utilize Multiple Availability Zones
Or AZ's in short. AWS provides multiple Availability Zones in each region. Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region. Additionally in this FAQ answer, AWS clearly states that common points of failures like generators, cooling systems are not shared between AZs and each AZ is physically separate and independent. This recent outage was primarily because of a fault in the power distribution system and since AZs are isolated only one of the AZ (us-east-1c) in the US-East region got affected. Remaining AZ's were running smoothly without any interruption. Hence it is prudent to have the infrastructure distributed across multiple AZ's. Distribute your EC2 Instances across multiple AZ's and bring up your HA quotient.
Note: You need not worry about any performance issues because of Instances deployed across different AZs. All AZs have very low network latency between them and can be ignored for web applications.

Configure ELB to use Multi AZ
AWS Elastic Load Balancer (ELB) can be configured to work across multiple AZ's. This way you can launch EC2 Instances in multiple AZs and have them tied to a single ELB. In event of a failure of one AZ, you will still have Instances running out of the other AZs and ELB will failover to them and your website will continue to be operational. When you create an ELB through the AWS Management Console you will not have the option to specify the AZ distribution. However, when you add Instances running in multiple AZ's to the ELB, the ELB automatically gets configured to work with multi AZs.

If you are using the command line interface or the API's, while creating the ELB itself you can mention the AZ's that the ELB needs to handle:

elb-create-lb LoadBalancerName  --availability-zones  value[,value...]  --listener "protocol=value, lb-port=value, instance-port=value, [cert-id=value]" [ --listener "protocol=value, lb-port=value, instance-port=value, [cert-id=value]" ...]  [General Options]

Additionally you create an AutoScalingGroup and mention the same list of Availability Zones, you are automating the whole failover process.

as-create-auto-scaling-group AutoScalingGroupName  --availability-zones  value[,value...] --launch-configuration  value  --max-size  value  --min-size  value [--cooldown  value ] [--load-balancers  value[,value...] ] [General Options]

In this case, the moment the Instances in one AZ went down,

• AutoScaling would have tried to launch new Instances to replace the failed ones
• AWS API's would have thrown errors due to AZ unavailability
• AutoScaling would have launched the new Instances in another AZ

RDS Multi AZ
Utilize the Multi-AZ feature of RDS where RDS will run a standby database Instance in another AZ in addition to the primary database Instance. In the event of a primary failure, RDS will internally initiate a failover to the standby and the database will continue to serve the traffic. AWS has mentioned in the RDS status dashboard that during the outage none of their Multi-AZ deployments were affected (failover got triggered). Only the Single AZ RDS had to be recovered by AWS team.

"As described in the EC2 post, a number of EBS volumes in the affected Availability Zone were impacted, disabling many Single Availability Zone (Single-AZ) RDS instances. Multi-AZ RDS instances were able to immediately begin failing over to other healthy Availability Zones. We started the recovery process for the affected Single-AZ RDS instances after EBS APIs became available at approximately 10:40 PM PST. Upon completing storage volume checks, we began processing EBS recovery volumes and recovered most of the affected RDS instances by 5:00 AM PDT on June 15th."

There are couple of things to note here:

• Unlike ELB, RDS failover is not instant. In simple words, the front end application that connects to the RDS will see a down time when the failover happens. During the time RDS completely fails over to the standby, connectivity to the RDS endpoint will not be available. We have seen that the time taken is approximately 5-7 minutes for around 100GB of data. But with more data probably it might take little longer.
• When it comes to database the impact is different when compare to, say, web servers. With database one will worry more about the data consistency and data durability than just availability. So when AWS talks above about recovering the Single RDS, it is actually the recovery of the EBS behind the RDS. These Single RDS EBS became unavailable and since there was a power failure it potentially resulted in a write corruption. Hence customers were asked to restore to the nearest point in time snapshot. With Multi-AZ such data corruption issue will not arise since there will be an entire replica on another AZ and data is written on both concurrently on a commit.
• Of course, Multi-AZ comes with additional cost. Probably double. The cost of running a Single RDS large Instance in US-East is about $300/mo give or take. In case of a Multi-AZ setup, this cost will double and AWS would be charging around $600/mo. Availability, consistency and data durability versus cost is something individual application owners need to decide

Stateless Servers

Apart from the persistent store (database) of the application, it is recommended to make all other layers in the infrastructure stateless. Web and application servers specifically needs to be designed to be stateless. Most of the applications store the web session data on the web server memory itself. Similarly there are other assets like user uploaded files, server and application log files that typically reside on the web/app servers. All of these assets needs to move to a central repository. For example, session information can be stored on caching servers or databases and files can be periodically rotated to, say, Amazon S3. Once we design such an application, then it is easy to make modifications in the infrastructure to maintain HA and also to scale on demand. Spreading the infrastructure across multiple AZ's can definitely help you achieve HA. But the failover will not be transparent to the user if the session information is stored in memory. If 100s of users session state is maintained in Instances in AZ1 and that AZ becomes unavailable, the front end Load Balancer will immediately failover to the Instances running in AZ2. But users' session state will be unavailable in these AZ2 Instances and will probably be asked to login again.

With these simple principles coupled with utilizing AWS building blocks effectively one can definitely build an infrastructure that will be truely Highly Available. Of course, these are lessons that one learn as Cloud delivery model matures and as architects one has to keep fine tuning the architecture post live as well.

Note: All the above thoughts are to increase uptime during problems with a single AZ. Last year we saw a different problem in AWS where a configuration change coupled with the architecture of EBS an entire region (US-East) had a service disruption. Infrastructure architecture can be designed to withstand such huge disruption as well. However those come with very high costs of replicating data over the internet. And in most of the cases the cost of operating such an infrastructure outweighs the benefits.

AWS VPC, Internal Load Balancer and Scaling Databases

There is more good news for solutions built using AWS VPC. AWS recently announced the availability of Internal Elastic Load Balancer within Virtual Private Cloud (VPC). This has been a request by the AWS community for quite a while and AWS always listens to the community and prioritizes product offerings accordingly.

Amazon's Elastic Load Balancing (ELB) is the internet exposed load balancing service. It can be used to load balance web servers and the web server layer can be scaled behind ELB. Usually the scaling requirements doesn't end with web layer. Often the subsequent backend layers such as application tier, database needs to scale as well. When you have a dynamic web layer and a dynamic application layer, architects will obviously have the challenge of how the dynamic servers identify each other. The solution would be to introduce a "app load balancer" in between which can load balance the backend application tier Instances. The web tier Instances will be configured to talk to the "app load balancer". This way the app tier can also be scaled behind the "app load balancer" while web tier continues to connect to the load balancer.

But the architecture has the following drawbacks:

• Single Point of Failure - the "app load balancer" becomes the single point of failure in the entire architecture. If that Instance becomes unavailable for some reason then essentially there will be a down time
• Scalability - the "app load balancer" layer is not scalable and even if we scale we are back to square one of dynamic infrastructure unable to be identified by the web tier
• Management - another layer that needs to be managed and monitored

AWS Internal Elastic Load Balancer (i am naming it iELB :) ) is the solution for this requirement. We can place iELB between the web and application tier and bring in scalability for the application tier and leave the rest of architecture concerns to AWS. iELB is available only within VPC and hence is useful for infrastructure that is residing within VPC.

Apart from using iELB for scaling and load balancing the application tier, I found it useful for another use case - Scaling Database

Most of the web applications are read intensive in nature. The resultant database queries will primarily be read transactions. To achieve better database throughput we can separate out the write and read transactions to different database Instances. If you are using Amazon RDS, you can launch RDS read replicas and use them to serve the read transactions. To scale the read operations we can launch multiple copies of read replicas according to the load condition. But there aren't good JDBC drivers that can load balance between multiple read replica Instances. Even if we are to use the drivers' capability to load balance between multiple DB Instances, it would be difficult to add and remove Instances dynamically. We will have to push configuration changes to the application tier whenever we scale out or scale in the read replicas. This can be addressed by utilizing iELB as the load balancer between the application tier and database. The resultant architecture will look like below.

Note: As of today, one can add only EC2 Instances behind the iELB. Whenever we create a read replica we do not know the Instance Id of that read replica. Rather we know only the end point URL. Hence this architecture is currently not implementable as such.

Here are some of the highlights of this architecture:

Security

• VPC - the entire infrastructure is in VPC providing network isolation within AWS and ability to connect to your datacenter over VPN
• Public and Private Subnets - separate different layers between public and private subnets. Public subnet Instances will be exposed to the internet and private subnet Instances will be accessible only within VPC
• Security Groups - create Security Groups for public Elastic Load Balancer and Internal Load Balancer. Configure the web servers to be accessible only from public ELB. Configure the iELB to accept requests only from Web Security Group. Configure RDS Security Group to accept requests only from the iELB

Scalability

• Web/App Server  - web servers in the public subnet can be scaled automatically and Elastic Load Balancing will load balance and failover between the available web EC2 Instances
• Database - launch additional read replicas as and when required to scale read requests. Scaling action will remain transparent to the application tier above

High Availability

• Multiple Subnets - utilize multiple subnets for all the layers in the architecture. Each subnet will be created on a Availability Zone (AZ). Hence using multiple subnets can mitigate AZ level failure (which is a rare case)
• RDS Multi AZ - launch RDS in Multi-AZ mode and use automatic failover from the primary to standby in case of a primary failure

Expanding Windows Azure

In my previous post I wrote about how the line between an IaaS and PaaS service provider is fading away with overlaps between the services they offer. Now, Azure is taking another step forward towards IaaS. Today is going to be a big day in the Azure world as some interesting announcements are expected in the Meet Windows Azure event. Azure team has spilled some beans in their blog about what's coming in.


Virtual Machines - ability to spin up Virtual Machines in the Azure cloud. Till now, you deploy the application on Azure and Azure takes care of the infrastructure provisioning. If you suspect any problems you can always login remotely to the Virtual Machine that Azure has provisioned for your application. You can probably perform some troubleshooting tasks on the VM but you cannot perform any changes and persist them. If you want to deploy a new change that needs to come through the Visual Studio route where you deploy a new version of the application. Azure will then take care of pushing the changes to the underlying VMs or spin up new ones. In AWS terms, you cannot create new server images from a running VM. But looks like that is going to change from today. Windows Azure Virtual Machines are going to be available where one can spin up new VMs.


Linux Instances - Now this is a major push from Azure. The ability to spin up Linux based VMs within Windows Azure. Yes, on Window Azure. Yes, pinch yourself. Read it again - Linux Instances on Windows Azure. I am wondering whether MS will go for re-branding by removing Windows from Windows Azure :)


Virtual Private Network - Creating a Virtual Private Network within Windows Azure with IPSec connectivity to your data center. A service very similar to Amazon Virtual Private Cloud. I believe, using this service, one will be able to create an isolated portion within Azure and setup the infrastructure within that. It will be interesting to see what will be possible within a Virtual Private Network. And this is a must have feature to tap into enterprises.


Azure Websites - Ability to directly deploy popular open source stack like Wordpress, Joomla, Drupal within Azure. I guess this service is going to be something similar to AWS Elastic Beanstalk. The Azure team is probably little late in realizing that currently the major players who are adopting public cloud are SMBs. SMBs heavily utilize open source technologies and Azure has realized that they need to get those customers on board as well. And with "Azure Websites" they are probably extending the simplicity of deployment and management that a pure PaaS application enjoys to Wordpress, Joomla and the likes.


These are the main offerings that has come out yet and may be the Azure team has something more as a surprise in the event today. But how far customers will embrace these and move into Azure is something that needs to be watched. AWS is way ahead in terms of the sheer number of services and offerings.


Azure is adding more IaaS elements to its offerings on top of the PaaS offering that it originally started off with. Similarly AWS is also adding PaaS like services (may be not pure PaaS) on top of the IaaS services and components. I think a new model is evolving where few years down the line we will see what is called as "IT as a Service". Service providers are not going to be pure IaaS or PaaS providers. Rather something like an ITaaS. The entire IT is going to be offered as a service where different environments like dev, staging, production will be running out of the Cloud. The service providers will also offer source control repositories and testing services. Web application developers will just use their IDE's which will have all required plugins and will carry out their development. When production ready the artifacts can be directly pushed to the production environment. This will also enable flexible development models where developers can work from anywhere and seamlessly integrate. All one needs will be the internet connectivity.