Friday, December 28, 2012

Log archive & analysis with Amazon S3 and Glacier - Part III

A recap from the previous posts:
In this post, we will see how to push logs from the local storage to the central log storage - Amazon S3 and what the considerations are.

Amazon S3 folder structure
The logs are going to be generated throughout the day and hence we need to have a proper folder structure to store them in S3. Logs will be particularly useful to perform analysis such as a production issue or to find a usage pattern such as feature adoption by users/month. Hence it will make sense to store them by year/month/day/hour_of_the_day structure.

Multiple Instances
The web tier will be automatically scaled and we will have multiple Instances always running for High Availability and Scalability. So, even if we have logs stored hourly basis, we will be having log files with similar names from multiple Instances. Hence the folder structure needs to factor multiple Instances as well. The resultant folder structure in S3 will look something like this

Amazon S3 Log Folder Structure
Amazon S3 Log Folder Structure
Note in the above picture (as encircled) that we are storing "Instance" wise logs for every hour.

Log Rotation
Every logging framework will have an option to rotate the log files on size, date, etc...We will be periodically pushing the log files to Amazon S3 and hence it might make sense to say, rotate the log file every hour and push it to S3. But the downside to that is, we cannot anticipate the traffic to the web tier and that's the reason we have the web tier scaling automatically on demand. If there is a sudden surge in the traffic which may result in large log files generated, it will start filling up the file system eventually making the Instance unavailable. Hence it is better to rotate the log files by size.

You can use the default logrotate available in Linux systems to rotate the log files on size. Logrotate can be configured to call a post script after the rotation to enable us push the newly rotated files to S3. A sample logrotate implementation will look like this:

Note: If you are using logrotate, make sure your logging framework isn't configured to rotate

/var/log/applogs/httpd/web {
        rotate 52
        size 50M
        create 644 root root
                /usr/local/admintools/ web &> /var/log/custom/web_logrotate.log

The above set of commands rotate the "httpd" log files whenever the size reaches 50M. It also calls a "postrotate" script to compress the rotated file and upload it to S3.

Upload to S3
The next step is to upload the rotated log file to S3.
  • We need a mechanism to access the S3 API from the shell to upload the files. S3cmd is a command line tool that is widely used and recommended for accessing all S3 APIs through the command line. We need to setup the CLI in the Instance
  • We are rotating by size but we will be uploading to a folder structure that maintains log files by the hour
  • We will also be uploading from multiple Instances and hence we need to fetch the Instance Id to store in the corresponding folder. Within EC2, there is an easy way to get Instance meta data. If we "wget" "" it will provide the Instance meta-data such as InstanceId, public DNS, etc.. For example if we "wget" "" we will get the current Instance Id
The following set of commands will compress the rotated file and upload them into the corresponding S3 bucket

# Perform Rotated Log File Compression
tar -czPf /var/log/httpd/"$1".1.gz /var/log/httpd/"$1".1

# Fetch the instance id from the instance
EC2_INSTANCE_ID="`wget -q -O -`"
if [ -z $EC2_INSTANCE_ID ]; then
echo "Error: Couldn't fetch Instance ID .. Exiting .."
        # Upload log file to Amazon S3 bucket
        /usr/bin/s3cmd -c /.s3cfg put /var/log/httpd/"$1".1.gz s3://$BUCKET_NAME/$(date +%Y)/$(date +%m)/$(date +%d)/$EC2_INSTANCE_ID/"$1"/$(hostname -f)-$(date +%H%M%S)-"$1".gz
# Removing Rotated Compressed Log File
rm -f /var/log/httpd/"$1".1.gz

Now that the files are automatically getting rotated, compressed and uploaded to S3 there is one last thing to be taken care of.

Run While Shutdown
Since the web tier will automatically scale depending upon the load, Instances can be pulled off (terminated) when load decreases. During such scenarios, we might be still left with some log files (maximum of 50MB) that didn't get rotated and uploaded. During shutdown, we can have a small script, that will forcefully call the logrotate to rotate the final set of files, compress and upload.
stop() {
echo -n "Force rotation of log files and upload to s3 intitiated"
/usr/sbin/logrotate -f /etc/logrotate.d/web
exit 0

We need to provide Access Key and Secret Access Key to the S3cmd utility for S3 API access. Do NOT provide the AWS account's Access Key and Secret Access Key. Create an IAM user who has access to only the specific S3 bucket where we are uploading the files and use the IAM user's Access Key and Secret Access Key. A sample policy allowing access for the IAM user to the S3 log bucket would be

  "Statement": [
      "Sid": "Stmt1355302111002",
      "Action": [
      "Effect": "Allow",
      "Resource": [
  • The above policy allows the IAM user to perform all actions on the S3 bucket. The user will not have permission to access any other buckets or services
  • If you want to restrict further, instead of allowing all actions on the S3 bucket, we can allow only PutObject (s3:PutObject) for uploading the files
  • Through the above approach, you will be storing the IAM credentials on the EC2 Instance itself. An alternative approach is to use IAM Roles so that the EC2 Instance will obtain the API credentials at runtime
With that we have the web tier log files automatically getting rotated, compressed and uploaded to Amazon S3 and stored in a central location. We have access to log information by the year/month/day/hour and Instance-wise.

Wednesday, December 19, 2012

Log archive & analysis with Amazon S3 and Glacier - Part II

In the previous post, we saw how to configure logging in AWS CloudFront and start collecting access logs. Let's now move on to the next tier in the architecture - web/app. The following are the key considerations for  logging in web/app layer:
  • Local Log Storage - choosing the local storage for logging. Using a storage option that is sufficient, cost-effective and meets the performance requirement for logging
  • Central Log Storage - how do we centrally store log files for log analysis in future
  • Dynamic Infrastructure - how do we collect logs from multiple servers that are provisioned on demand
Local Log Storage
Except very few cases, EBS-backed Instances are the most sought after Instance type. They launch quickly and easier to build Images out of them. But they come with couple of limitations from logging perspective
  • Limited storage - EBS-backed AMIs that are provided by AWS or third-party providers come with limited storage. For example, a typical RHEL AMI comes with around 6GB of EBS attached as the root partition. Similarly Windows AMI come with 30GB EBS attached as C:\
  • Growing EBS - log files tend to grow faster. And it becomes difficult to grow the root EBS (or an additional EBS) as the log file sizes grow
  • Performance - any I/O operation on an EBS Volume is over the network. And it tends to be slower than local disk writes. Specifically for logging, it is always better to remove the I/O bottleneck. Otherwise lot of system resources could be spent towards logging
Every EC2 Instance comes with ephemeral storage. These are local storage directly attached to the host on which the Instance is running. Ephemeral storage do not persist between stop-start cycles of an Instance (EBS-backed) but they are available when the Instance is running and persist during reboots. There are couple of advantages of Ephemeral storage:
  • They are locally attached on the physical host on which the Instance runs and hence have better I/O throughput when compared to EBS Volumes
  • They come in pretty good size - for example a m1.large Instance comes with 850GB of ephemeral storage
  • And it comes free of cost - you aren't charged per GB or for any I/O operations on the ephemeral storage unlinke EBS
This makes ephemeral storage the ideal candidate for storing log files. For an EBS-backed Instance, the ephemeral storage is not mounted and readily available. Hence one needs to follow the following steps to start using the ephemeral storage for storing log files
  • The logging framework usually comes with a configuration file to configure logging parameters. The log file path needs to be configured to point to the ephemeral storage mount directory that we create below
  • All application related files (such as binaries, configuration files, web/app server) will be installed on the root EBS. Before the final AMI is created, the ephemeral storage needs to be setup and configured
  • Run fdisk to list all the storage devices that are mounted
fdisk -l
  • Created a directory such as "/applogs". This is the directory where the ephemeral storage will be mounted
mkdir /var/log/applogs
  • Mount the storage device in this directory using the "mount" command
mount /dev/xvdj /var/log/applogs
  • Add "fstab" entries so that the ephemeral storage is mounted in the same directory after stop/start or when new Instances are launched out of this AMI
/dev/xvdj  /var/log/applogs xfs defaults,noatime,nobarrier,allocsize=64k,logbufs=8 0 2
/dev/xvdj /var/log/applogs    ext3    defaults        0   0

The last step is essential especially from AutoScaling point of view. When AutoScaling launches new Instances, the ephemeral storage needs to be automatically mounted in the directory so that the application can start logging. Now, we can go ahead create the final AMI and launch Instances from them. The new Instances will have the ephemeral storage automatically mounted in the "/var/log/applogs" directory and applications can start storing the log files in them.

Friday, December 14, 2012

Log archive & analysis with Amazon S3 and Glacier - Part I

Since this would be a multi part article, here's an outline in terms of how the different parts will be arranged
  1. First we will set the context in terms of taking a web application and identifying the areas of log generation for analysis
  2. Next we will define the overall log storage structure since we have logs being generated from different sources
  3. We will then look at each tier and look at how logs can be collected, uploaded to a centralized storage, what are the considerations
  4. Finally, we will look at other factors such as cost implications, alternate storage options, how to utilize the logs for analysis

Let's take an e-commerce web application which has the following tiers
  • Content Distribution Network - a CDN to serve the static assets of the website. AWS CloudFront
  • Web/App Server running Apache / Nginx / Tomcat on Amazon EC2
  • Search Server running Solr on Amazon EC2
  • Database - Amazon RDS
The first three areas are the major source of log information. Your CDN provider will provide access logs in a standard format with information such as the edge location serving the request, the client IP address, the referrer, the user agent, etc...The web servers and search servers will write access logs, error logs and application logs (custom logging by your application).

Log Analysis Architecture
Log Analysis Architecture

In AWS, Amazon S3 becomes the natural choice for centralized log storage. Since S3 comes with unlimited storage and is internally replicated for redundancy, it will be the right choice for storing the log files generated by the CDN provider, web servers and search servers. Per above architecture, all of the above tiers will be configured/setup to push their respective logs to Amazon S3. We will evaluate each layer independently and look at how to setup logging and the different considerations associated.

S3 Log Storage Structure
Since we have logs coming in front different sources, it is better to create a bucket structure to organize them. Let's say we have the following S3 bucket structure
S3 Log Storage Bucket Structure
S3 Log Storage Bucket Structure
  • my-global-logs: Bucket containing all the logs
  • cf-logs: Folder under the bucket for storing CloudFront logs
  • web-logs: Folder under the bucket for storing Web Server logs
  • solr-logs: Folder under the bucket for storing Solr Server logs

AWS CloudFront
AWS CloudFront is the Content Distribution Network service from AWS. With a growing list of 37 edge locations, it serves as a vital component in e-commerce applications hosted in AWS for serving static content. By using CloudFront, one can deliver static assets and streaming videos to users from the nearest edge location and thereby reducing latency, round trips and also off loading such delivery from the web servers.

Enable CloudFront Access Logging
You can configure CloudFront to log all access information during the "Create Distribution" step. You "Enable Logging" and specify the bucket to which CloudFront should push the logs.
Configure CloudFront for Access Logging
Configure CloudFront for Access Logging

  • Specify the bucket that we created above in the "Bucket for Logs" option. This field will accept only a bucket in your account and not any sub-folders in the bucket
  • Since we have a folder called "cf-logs" under the bucket to store the logs, mention the name of that folder in the "Log Prefix" option
  • CloudFront will start pushing access logs to this location every hour. The logs will be in W3C extended format. The logs will be compressed by AWS since the original size could be significantly large for websites that attract massive traffic
Once this is setup CloudFront will periodically start pushing access logs to this folder.
CloudFront Logs
CloudFront Logs
In the next post, we will see how to configure the web tier to push logs to S3 and what are the different considerations.

Tuesday, December 11, 2012

Log archive & analysis with Amazon S3 and Glacier - Introduction

Logging is an essential part of any system. It let's you understand what's going on in your system especially serving as a vital source for debugging. Primarily many systems uses logging to let developers debug issues in the production environment. But there are systems where logging becomes the essential component to understand the following
  • User Behavior - understanding user behavior patterns such as which areas of the system is being used by the user
  • Feature Adoption - evaluate new feature adoption by tracking how a new feature is being used by the users. Do they vanish after a particular step in a particular flow? Are people from a specific geography use this during a specific time of the day?
  • Click through analysis - let's say you are placing relevant ads across different pages in your websites. You would like to know how many users clicked them, the demographic analysis and such
  • System performance
    • Any abnormal behavior in certain areas in the system - a particular step in a workflow resulting in error/exception conditions
    • Analyzing performance of different areas in the system - such as finding out if a particular screen takes more time to load because of a longer query getting executed. Should we optimize the database? Should we probably introduce a caching layer?
Any architect would enforce logging as a core component in the technical architecture. While logging is definitely required, many a times, inefficient logging such as too much logging, using inappropriate log levels might lead to the following
  • Under performance of the system - the system could be spending more resources in logging than actively serving requests
  • Huge log files - generally log files grow very fast, especially when inappropriate log levels are used such as "debug" levels for all log statements
  • Inadequate data - if the log contains only debug information by the developer there will not be much of an analysis that can be performed
On the other hand, the infrastructure architecture also needs to support for efficient logging and analysis
  • Local Storage - how do you efficiently store the log files on the local server without running out of disk space; especially when log files tend to grow
  • Central Log Storage - how do you centrally store log files so that it can be used later for analysis
  • Dynamic Server Environment - how do you make sure you collect & store all the log files in a dynamic server environment where servers will be provisioned and de-provisioned on demand depending upon load
  • Multi source - handling log files from different sources - like your web servers, search servers, Content Distribution Network logs, etc...
  • Cost effective - when your application grows, so does your log files. How do you store the log files in the most cost effective manner without burning a lot of cash
In this multi-post article let's take up a case of a typical e-commerce web application with the above characteristics and setup a best practice architecture for logging, analysis and archiving in AWS. We will see how different AWS services can be used effectively to store and process the logs from different sources in a cost effective and efficient manner.

Tuesday, December 4, 2012

Amazon VPC Essentials

Amazon Virtual Private Cloud (VPC) is a great way to setup an isolated portion of AWS and control the network topology. It is a great way to extend your data center and use AWS for burst requirements. In this post, I will list down the key areas that one needs to consider when working with VPC. This will help one decide the best architecture / solution that may fit the given requirement.

Except Cluster Compute Quadruple Extra Large instances all other Instance types (including Micro) are available within VPC. Of course, be sure to check which Instance types are available in a region. For example Second generation Standard On-demand instances are available only in US-East

Know which services and which features of a service are available within VPC
    • Dynamodb, ElastiCache, SQS, SES, SNS and CloudSearch are not yet available in VPC
    • Amazon RDS Micro Instances are not available in a VPC at this time - Update: AWS just announced (few minutes after posting this article) availability of Micro RDS Instances in VPC #paceofawsinnovation
    • RDS Instances launched in VPC cannot be accessed over the internet (through the end point). You will need a jump box (bastion) in the Public Subnet to connect to RDS or of course you can connect through the VPN from DC
    • DevPay AMIs cannot be launched within VPC
BGP is no longer a requirement for VPN connectivity from your devices. Static routing can be used for non BGP devices. But BGP is recommended since the liveness check that is performed is better. Also, each VPN conenctions has two tunnels for redundancy. If one of them goes down, the failover process is fairly simple in case of BGP

Customer Gateway Compatibility
The following customer gateway devices are tested and known to be working with AWS VPC. Make sure you are using one of these devices at the Customer Gateway side. Other devices may be compatible as well
  • Static Routing
    • Cisco ASA 5500 Series version 8.2 (or later) software
    • Cisco ISR running Cisco IOS 12.4 (or later) software
    • Juniper J-Series Service Router running JunOS 9.5 (or later) software
    • Juniper SRX-Series Services Gateway running JunOS 9.5 (or later) software
    • Juniper SSG running ScreenOS 6.1, or 6.2 (or later) software
    • Juniper ISG running ScreenOS 6.1, or 6.2 (or later) software
    • Microsoft Windows Server 2008 R2 (or later) software
    • Yamaha RTX1200 router
  • Dynamic Routing using BGP
    • Astaro Security Gateway running version 8.3 (or later)
    • Astaro Security Gateway Essential Firewall Edition running version 8.3 (or later)
    • Cisco ISR running Cisco IOS 12.4 (or later) software
    • Fortinet Fortigate 40+ Series running FortiOS 4.0 (or later) software
    • Juniper J-Series Service Router running JunOS 9.5 (or later) software
    • Juniper SRX-Series Services Gateway running JunOS 9.5 (or later) software
    • Juniper SSG running ScreenOS 6.1, or 6.2 (or later) software
    • Juniper ISG running ScreenOS 6.1, or 6.2 (or later) software
    • Palo Alto Networks PA Series running PANOS 4.1.2 (or later) software
    • Vyatta Network OS 6.5 (or later) software
    • Yamaha RTX1200 router
IP Ranges
When setting up a VPC you are essentially fixing the network of the VPC. And if the VPC requires VPN connectivity (as in most of the cases), care should be taken to choose the IP range of the VPC and avoid any IP conflicts.
  • Assess the IP ranges used at the customer gateway side
  • If the customer gateway side has more than one data center which already have VPN connectivity between them, assess all the data centers
  • Also check the MPLS link between the data centers and the IP range used by the MPLS provider
The above two points (Device Support & IP Range) becomes the critical point in setting up the VPC. If the chosen IP ranges result in IP conflicts, this will jeopardize the entire network architecture and can potentially bring down existing systems in the customer data center

Public and Private Subnets
The VPC network can be divided further in to smaller network segments called as Subnets. Any VPC will have at least one Subnet
  • You can setup a Public Subnet which will have internet connectivity. Instances launched within a Public Subnet will have both outbound and inbound (through EIP) internet connectivity through the Internet Gateway attached to the Public Subnet
  • Private Subnets are completely locked down. They do not have internet connectivity by default
  • Create number of Public and Private Subnets depending upon your architecture. Place all public facing servers such as web servers, search servers in the public subnet. Keep DB servers, cache nodes, application servers in the private subnet
Security Groups
VPC Security Groups are different from normal EC Security Groups. With EC2 Security Groups you can control the ingress into your EC2 Instance. With VPC Security Groups, you have the option to control both inbound and outbound traffic. When something is not accessible you have to check both inbound and outbound rules set in the VPC Security Group

ELB Security Group 
When you launch an ELB within VPC, you have the option to specify a VPC Security Group to be attached with the ELB. This is not available for ELB launched outside VPC in normal EC2. With this additional option, you can control access to specific ELB ports from specific IP sources. On the backend EC2 Instances' Security Group, you can allow access to the VPC Security Group that you associated with the ELB

Internal ELB 
When you launch an ELB within VPC, you also have additional option to launch it as an "Internal Load Balancer". You can use an "Internal Load Balancer" to load balance your application tier from the web tier above. Please refer to my earlier article on Internal Load Balancer

NAT Instance 
By default the Private Subnets in a VPC do not have internet connectivity. They cannot be accessed over the internet and neither can they make outbound connections to internet resources. But let's say you have setup a database on an EC2 Instance in the Private Subnet and have implemented a backup mechanism. You would want to push the backups to Amazon S3. But the Private Subnet's cannot access S3 since there is no internet connectivity. You can achieve it by placing  a NAT Instance in the VPC
  • Through NAT Instance outbound connectivity for Private Subnet Instances can be achieved. The Instances will still not be reachable from the internet (inbound)
  • You need to configure the VPC Routing Table to enable all outbound internet traffic for the Private Subnet to go through the NAT Instance
  • AWS provides a ready NAT AMI (ami-f619c29f) which you can use to launch the NAT Instance
  • You can have only one NAT Instance per VPC
  • Since you can have only one NAT Instance per VPC, you need to be aware that it becomes a Single Point Of Failure in the architecture. If the architecture depends on the NAT Instance for any critical connectivity, it is an area to be reviewed
  • And you are limited by the bandwidth availability of a single NAT Instance. So do not build architecture that will have internet bandwidth requirements from the Private Subnet with NAT
Elastic Network Interface
This is a powerful feature of VPC which lets you solve many problems around licensing, hosting multiple SSL websites, etc...
  • You can attach multiple ENI's per Instance essentially creating multiple private IPs and possibility of attaching multiple EIP's per Instance
  • One interesting result of attaching multiple ENI's per Instance is this: previously you are used to launching EC2 Instances in a Subnet. Now EC2 Instances can span across Subnets. For example 
    • You can have two ENIs - one attached to the public Subnet and other attached to the private Subnet
    • You attach a Security Group to each of these ENIs. The Security Group of public Subnet ENI will allow access on port 80 from the internet but no access on, let's say, port 22
    • The other Security Group attached to the Private Subnet ENI will allow port 22 access from the VPN gateway essentially securing the Instances more
  • If there are products that have licenses tied to the MAC address, you can now set the MAC address to the ENI and have the EC2 Instance derive the them from ENI
Private IP and Elastic IP
  • You can fix the Private IP of the Instance when you launch Instances within the VPC.  Let's say, you plan to launch multiple tiers (web servers and search servers) within a Subnet and have multiple Instances for each tier. Make a logical allocation of IP ranges for each tier. For example, to for web servers. Whenever you launch any Web Server Instance use one of the private IPs from the logical list. This will help in management of the potentially vast number of Instances
  • Private IP of an Instance do not change if you stop and start the Instance
  • If you attach an Elastic IP to the Instance, the EIP does not change if you stop and start the Instance

Monday, October 29, 2012

Another AWS outage. And how do you overcome it?

Disclaimer I: I don't work at AWS :) but seriously think that building your own cloud (as some people call it) or setting up a private cloud doesn't really address anything apart from posting a comment in article about an outage.

Disclaimer II: Though not every one admits, some people who have not been affected during outages have been lucky. They are lucky that their Instances and EBS Volumes weren't in the affected AZ. Though people spend time on building sophisticated architecture, it is only during such outages, the architecture is put to real test. Just like how AWS themselves uncover new failure areas, the architecture will also open up new failure points during outages. And you need to work on improving them.

We just had another outage in AWS. And this time too there were a good number of popular websites that were affected by the outage - Reddit, Foursquare and Heroku are few from the list. AWS has published a detailed analysis of what caused the outage. The previous outage was on June 29th which was even more widespread taking down Netflix and others. AWS did provide a detailed analysis on what went wrong and how they are taking actions to prevent such incidents in future.

The aftermath and reactions of such outages can be classified in to two - from people who understand Cloud and people who don't. No one likes outages. But people who understand Cloud, they clearly see outages as an opportunity to build better systems. Better architectures. And there are people who would start blaming it on the Cloud. Saying things like "this is the reality behind Cloud", or even say it is better to own than rent. Public Cloud versus Private Cloud is a huge debate. And if you have been in the Cloud market you will even hear more types of Cloud - like Hybrid Cloud.

Why Cloud?
Be it public cloud, private cloud or hybrid cloud. Be it Open Stack or Cloud Stack. One should definitely know why they need a Cloud strategy. Are you getting into Cloud for
  • Scalability
  • High Availability
  • Cost
  • Getaway from infrastructure management
  • Everyone is getting in to Cloud
  • My resume needs to show I know Cloud
Be honest in your answer to this evaluation because the benefits vary depending upon where you fit in to the above classification. Of course, every one is concerned about cost and would like to keep costs low. But there are other invisible costs that doesn't stand out normally. The cost of slow performing website, the cost of a lost customer are not seen directly as against infrastructure cost. And designing scalable systems within a specific cost is an even bigger challenge.

Hardware fails
It has to be understood that hardware fails all the time. The Cloud Service Provider doesn't setup some magic data center where hardware doesn't fail. And if some one advises you towards private cloud because of such outages in public cloud, be wary that such hardware failures will happen in private cloud setup as well. In contrast, because of the scale at which the public cloud service provider is operating, there is massive automation in place. There are systems what will detect failures well ahead of time and act on its own. There are disk's that reach their lifetime and they get replaced automatically. Those disks will go through a purge cycle to be destroyed completely as per the policy of service provider. Servers get replaced, security systems get updated and network gets upgraded. All these are happening continuously and transparently. Probably as an every day routine in all the data centers. So such things need to happen if you are going to setup a private cloud

Efficiency is successful because they mastered retail and are operating it at cut throat efficiency. They have successfully replicated the same model into AWS. At any given point of time, AWS tries to operate their data centers at maximum efficiency. That's where they make the most of the buck. By continuously operating at that efficiency and because of the economies of scale, they are also able to pass on the benefits as cost reduction to end customers (without even any significant competition). Whether a private cloud setup or another service provider that would operate at that efficiency and continue to automate is something that one has to evaluate.

With that as a background, here are some thoughts on keeping your website highly available.

Availability Zones
I read your mind. "Yeah. I know. You gonna tell me to use multiple Availability Zones right?". Yes, almost the first thing that you will notice in any article which talks about overcoming outages is utilizing multiple Availability Zones (AZs). So is there something new that I am gonna talk about? Yes.
  • AZs are data centers and they are isolated. Some AZ's might have multiple data centers supporting them. In the sense that they have their own independent power, diesel generators, cooling and outbound internet connectivity. And they are not isolated across coast or cities. They are just near by data centers having low latency network between them 
  • AZs have low latency between them. But there is a latency. And it might be considerable for you depending upon the number of concurrent requests that your application is expected to handle. So if you have the application servers distributed across AZ-1 and AZ-2 and database only in AZ-1, the AZ-2 application will experience a latency when talking to the DB
So make use of multiple AZ's. If you are planning for High Availability at web layer you will have to have a minimum of two EC2 Instances. Launch them across two different AZ's. Each region in AWS has at least two AZ's and any new regions that will be launched by AWS will have at least two AZ's. If you look at the latest outage, it was specific to a particular AZ. So infrastructure spanning across multiple AZ's weren't completely down. But then why Reddit went down? I will talk about that a little later.

Elastic Load Balancing
Elastic Load Balancing (ELB) is the load balancing service from AWS. When you look at ELB, there are two things that you should pay attention to. One - ELB servers. The EC2 Instances that AWS will be using to run the ELB service. Two - the servers behind the ELB which are your web/app servers.
  • ELB, being a service from AWS has HA, scalability and fault tolerance built into it. The Instances that AWS will be provisioning to support the ELB service will span across multiple AZ's and will automatically scale on its own. All of this are abstracted to us and it is an obvious guess that AWS will utilize multiple AZ's by themselves
Configuring ELB's to distribute traffic across multiple AZ's
Configuring ELB's to distribute traffic across multiple AZ's

  • For your web/app Instances that sit behind ELB, you have an option to specify the AZ's that the ELB has to support while creating it. For example, when you create an ELB in US-East-1 region, you can mention "us-east-1a" and "us-east-1b" to be supported by the ELB. ELB will now distribute the incoming traffic between the AZ's. Now why is this important? Because, if you specify multiple AZ's for ELB, it results in two things
    • The Instances that ELB creates for load balancing will created in all the AZ's that you mention. This means there is a AZ level HA for load balancing Instances itself
    • If there is an AZ level failure, then the ELB service will route the traffic to the load balancing Instances in the other AZ thereby continuing the service operation
  • So why ELB service was interrupted during the outage? It appears that there was a software bug in the traffic diversion software which resulted in traffic still being routed to the load balancing Instances in the affected AZ itself. Looks like AWS is fixing that bug pretty soon
AutoScaling will automatically scale up/down your Instances according to the pre-configured triggers (based on CPU, Network, etc...). AutoScaling works in conjunction with ELB where it also gets signals from ELB. AutoScaling can be configured with multiple AZ's as follows

as-create-auto-scaling-group AutoScalingGroupName  --availability-zones  value[,value...] --launch-configuration  value  --max-size  value  --min-size  value [--cooldown  value ] [--load-balancers  value[,value...] ] [General Options]

Once configured, when AutoScaling launches/terminates Instances it will distribute across the configured AZ's. So if the scale up condition is "2" (launching two Instances), and AutoScaling has been configured with "us-east-1a" and "us-east-1b" then AutoScaling will launch one Instance in each AZ.

Go HA with 3 not 2
We know that we have to use multiple AZs. And regions like US-East has 5 AZs. How many do you use in ELB and AutoScaling? It is always better to use three AZ's at a minimum. If you are using only two AZ's and if one of the AZ is down, then the other AZ is receiving traffic twice its normal since ELB will redirect all the traffic to the other AZ. Your AutoScaling is configured to scale up by 1 and now an AZ is down as well. So by the time your AutoScaling reacts you are probably flooded with loads of traffic. And,
  • You aren't alone - you are not the only one affected. There are 100s of other customers whose 1000's of Instances are affected. And they are all going to spin up Instances in an alternate AZ. Now AWS might run out of Instances in the alternate AZ's because they are already operating at 20% reduced capacity (in case of US-East)
  • API error rates - normally with any such outage, the APIs start throwing up errors. You provision the infrastructure through API's or through AWS Management Console which is a UI wrapper around APIs. During an outage, AWS definitely doesn't want the other AZ's to be flooded with requests which may cause a spiral effect and make the other AZ's also reach critical infrastructure thresholds. And AWS API's themselves will be running in multi-AZ mode and probably are affected because of non-availabilty of an AZ. In such cases, I am sure there is a lot of throttling that AWS will employ at the API level to protect the API from customer requests causing a spiral effect in other AZ's
  • Other AWS services - there are other services like ELB, Route53, SQS, SES, etc...which are services that will have HA built on to the them. For example, the ELB service itself to support its load balancing service not going down during an outage, might provision it's own ELB Instances in an alternate AZ's
So in effect, though multi AZ setup is the recommended setup in AWS, it is always better to use more than two AZ's (if a region has). In case of US-East, you can, say, utilize three AZ's in your AutoScaling configuration and ELB settings. The probability of two AZ's going down at the same time is less and there is a better distribution of traffic even if one AZ is down. Of course, it can be argued that we are just avoiding an eventuality. Yes, but during an outage the primary goal of anyone would be to sustain operations (even at reduced capacity) during the outage and return to normalcy at the earliest.

Understand AWS Services that are multi-AZ
The following services from AWS are inherently multi-AZ and are highly available. During an outage, these services will be still available and they would have internally failed over to another AZ.
  • Route53 - DNS
  • Elastic Load Balancing - Load Balancer
  • CloudFront - Content Distribution Network
  • Amazon S3 - Storage Service
  • DynamoDB - NoSQL Database Service
  • SQS - Simple Queue Service
  • SES - Simple Email Service
RDS Multi-AZ supports multiple AZ's but you unlike the other services listed above, the application will notice the RDS failover. During an AZ outage, when RDS internally triggers a failover to the standby instance in another AZ, the application will lose connectivity to the RDS endpoint. Depending upon the volume of data and transactions that were happening at the time of failure, the failover time could be around 3-5 minutes.

But even Multi AZ RDS Instances were affected
Yes, and that again seems to be because of bugs in the automatic failover software. RDS relies on EBS for database storage and this outage started with problems in EBS Volumes (which was a result of improper internal DNS propagation). If the EBS of primary RDS Instance is stuck it would result in an automatic failover to the standby immediately. The failover didn't happen immediately because the software didn't anticipate the type of EBS I/O issue that happened during this outage. But for some of the RDS Instances, looks like the primary lost connectivity to the standby and the primary was still accepting transactions. This should have immediately triggered a failover. Because the standby is not receiving updates and can result in data inconsistency. But again, there seems to be a software bug which prevented such a failover. This has to be addressed immediately because RDS is a service where AWS has developed synchronous replication to the standby which isn't available in native MySQL. And that needs to definitely work perfectly all the time.

Perceived Availability
Different applications need different levels of availability. And it is easier to achieve 99.99% of availability. But pushing things forward, every extra nine comes with additional complexity in the architecture and cost. As application architects, it is also required to understand how the application end users perceive availability of a system. For example, let's consider a photo sharing website built on top of an open source CMS. If you break down the different areas in the architecture, you will be looking at the following
  • A DNS service
  • Content Distribution Network - serving static assets from the nearest edge location
  • A front end web load balancer
  • A page cache - to cache the HTML
  • The web server - storing templates, etc...
  • Object Cache - to cache the content, store user sessions
  • Database - storing user data and other transactional data
AWS Services and High Availability
AWS Services and High Availability

With such kind of integration of services that Amazon offers, you have certain portions of your website still available during an outage. For example, your users can still view photos (through CDN) but might not be able to upload new photos (at normal throughput) during outage. Users will still be able to see data from cache which is not up-to-date. Likewise, each area in the application needs to be looked at from a failure point of view and addressed accordingly.

As architects, one needs to continuously ask questions on failures in each area/tier in the architecture. What if I pull out that web server behind the ELB, will my users sessions be lost? What if that EBS volume becomes unresponsive? What if my RDS Instance is flooded with database connections? Such probing can lead in to small improvements in the architecture which can increase the overall site availability.

So why did Reddit and Heroku go down?
I am sure the engineers at Reddit and Heroku are well aware of all these and would have designed an architecture that scales and tries to keep the service up and running all the time. While, there is no information on what went really wrong, we all know that software systems are not perfect at all the time. No one designs systems that anticipates all probabilities of failure and has a fool proof mechanism to address them. Only when the system is put to use, pushed to boundaries and scales it opens up new flaws in the architecture that once we thought was perfect. I am sure Reddit is learning from areas in their architecture that went down during the outage and will make it better. Just like how Netflix wrote an article on what lessons they learnt from the previous outage.

Wednesday, September 26, 2012

Few more AWS updates

RDS Provisioned IOPS
Just as I mentioned in previous article that Provisioned IOPS (PIOPS) will be naturally extended to RDS as well, AWS just released PIOPS feature for RDS. You can read the full update here. Couple of points to be noted
  • This is currently available only for newly provisioned RDS Instances. If you create a new RDS Instance you will be able to specify the required IOPS
  • At the time of writing this article, specifying PIOPS is not yet available in Singapore region. It is available in US-East, US-West and EU-West regions
  • As the article notes, AWS will release an automate way to migrate existing RDS Instances to PIOPS RDS. I believe it will come with its own down time. I am wondering if AWS will make it without an endpoint change so that there will not be any change in the application side
  • For now, if you want to migrate the RDS Instance to PIOPS based, you can follow the below steps
    • Take an application downtime and stop all transactions hitting the RDS Instance
    • Create a new DB Snapshot from your existing RDS Instance
    • Create a new RDS Instance from the Snapshot and mention the required PIOPS
    • The new RDS Instance will come with its own endpoint which needs to specified in your application
Non BGP device support in VPC
AWS Virtual Private Cloud (VPC) is a great way for enterprises to extend their data center to AWS and use AWS for burst capacities. Workloads can be setup in public and private subnets within VPC with a secured IPSec tunnel to existing data centers. For a VPN connectivity there are two sides of the VPN tunnel - one at AWS side and the other at the Customer side. The AWS side of the tunnel is automatically taken care by AWS whenever we create a VPC. But on the customer side, one needs to have a compatible device that can talk to AWS VPC. Till now, these customer devices needs to support BGP protocol to successfully establish a VPN connection with VPC. Not anymore. AWS has released a new static routing feature that can create VPN connections from VPC to customer devices which do not support BGP protocol.
  • Most of the enterprises will have a device which support BGP protocol. But in our experience, we have seen that the network team sometimes find it hard to configure. Especially when the enterprise has existing VPN connections to other data centers. A lot of care should be taken while adding another VPN connectivity and advertising the right set of IP ranges. Otherwise, it might result in loops where traffic can be diverted elsewhere bringing critical systems down
  • With Static routing, testing a VPC setup becomes very easy. Most of the offices and corporate branches might have devices which support IPSec but not BGP. In such cases, this comes pretty handy to test out a VPC setup. It would be worthwhile to quote a real life example that we faced. As a practice, we generally test out all architecture recommendations before we make the recommendations to our customer. For one of the large enterprise customer, the architecture involved VPC with VPN connectivity to their data center. We were able to successfully create the VPC architecture in AWS and configure all the routing. But to test it out, we needed to have a device at our office that supports BGP. We had a simple device that supported IPSec but not BGP. We had to procure a new device and continue with the testing. This will not be the case anymore
  • Here's the complete list of devices that AWS VPC supports - both BGP (dynamic) and non BGP (static) ones
SQL Server RDS in VPC
RDS for SQL Server is now available within VPC. This has been long pending from AWS. VPC is the way any typical enterprise would like to move their workloads into AWS and use AWS for their burst capacities. With enterprises, SQL server is still the widely used database and non-availability of SQL Server RDS within VPC has been a long pending issue. Not anymore. We can now provision a RDS SQL Server within a VPC and let AWS manage the database such as patching, upgrades and automated backups.
  • Multi-AZ deployments are not yet available. Unlike RDS MySQL, RDS SQL Server still does not support Multi-AZ deployments. So in case of any primary failure, we will have to recover the DB Instance from the latest backup (latest automated snapshot)
  • Read Replicas for RDS SQL Server are not yet available.
  • Scaling Storage - MySQL RDS allows us to scale the storage space on demand. If we start with 100GB MySQL RDS and once we near the 20% available free space, we can always increase the storage space to, say, 200GB. RDS will automatically increase the underlying storage space without causing any database downtime. This is currently not available for RDS SQL Server
  • RDS SQL Server currently supports only SQL Server authentication. Windows authentication is not yet supported

Monday, August 27, 2012

AWS updates that makes architects happy

AWS keeps adding a lot of features to its existing services and also keeps launching new services. With the pace of innovation from AWS and the rate at which new services and updates are released, one may lose track of them pretty easily. As architects it is essential to know all of these all the time so that one proposes the right solution for a given requirement. Here are some of the features and services that has made the lives of architects easy and can help in simpler infrastructure design addressing specific needs. There is no order to this list - I am just listing them down. And here's a summary before we go detail in to each of them

High I/O EC2 Instance Type - This is a new family of Instance types from AWS in addition to the existing High Memory and High CPU Instance types. The first one in this family is hi1.4xlarge which comes with 60GB of memory and 8 virtual cores. The best part of this Instance type is the whopping 2 TB of SSB attached to them. These SSDs are the ephemeral disks that come as part of the Instance and are of course non-persistent. AWS promises around 120K random read IOPS and between 10K and 85K write IOPS. This Instance type is suitable for I/O intensive applications and running NoSQL databases. Here are couple of pointers about this Instance type

  • SSD disks are ephemeral - you will lose data on Instance termination (of course if you stop and start the Instance)
  • Data will be persistent between reboot like any other EC2 Instance
  • You will see two 1TB volumes on the Instance. Can be striped to double the throughput
  • If you are launching a Windows Instance, choose the "Microsoft Windows 2008 R2 64-bit for Cluster Instances" AMI
  • Currently this Instance type is available only in US-East and EU-West regions
  • And its priced at $3.10 an hour :)
I am sure AWS will be releasing lower Instance types (such as High I/O large) since not everyone will be looking to use an 4xlarge Instance. There are already some benchmarks that you can refer here and specifically netflix has performed a detailed benchmark of this Instance type.

Provisioned IOPS for EBS - EBS is great for storing persistent data and the ability to attach and detach it to/from any Instance. But from day zero the performance and throughput has been an issue with EBS. The IO throughput that one gets out of EBS is very poor and all the more it is not consistent. If the network load during that time is not heavy then one might see large throughput from EBS. If not the throughput will hover around 100 IOPS. Fast forward to now and with provisioned IOPS we can set the IOPS that we expect from EBS and AWS will guarantee the IOPS at all times. This is great news for hosting databases on EC2 (if you are not using RDS). Here are couple of pointers on this:
  • Currently the max IOPS that can be set is 1000. AWS will soon increase this limit
  • We can of course drive more throughput by attaching multiple EBS volumes and striping them
  • Costs little extra than a normal EBS Volume - 0.125/GB against 0.10/GB. Of course the benefits will be far more than the cost difference. Guess everyone will eventually move to provisioned IOPS. Nice way to increase AWS revenue :)
I am assuming that provisioned IOPS will be extended to RDS as well in near future. Just like DynamoDB where one can specify the required throughput, AWS should let the user specify the throughput required from RDS. RDS already uses EBS behind the scenes for storing the data and hence this should definitely be in the pipeline. At least I wish so.

EBS-Optimized EC2 Instances - One thing with EBS that everyone understands is that it is network attached. Unlike local ephemeral storage which is directly attached to the EC2 Instance, there is a pipe which transfers all the data that will be written to EBS. So, even if we use provisioned IOPS to increase the throughput from an EBS, the network throughput can be significantly low that we may not utilize the increased EBS throughput; especially when the network is shared between many EC2 Instances. AWS has introduced EBS-Optimized EC2 Instances which have dedicated network throughput to the EBS that are attached to it. An EBS-Optimized m1.large Instance will have about 500 Mbits/s of guaranteed network throughput to EBS.
  • Currently only m1.large, m1.xlarge and m2.4xlarge are available as EBS-Optimized Instances
  • The guaranteed throughput is 500 Mbits/s for m1.large and 1000 Mbits/s for m1.xlarge and m2.4xlarge
  • This should not be confused with the regular network throughput of the EC2 Instance - for example talking to another EC2 Instance. This guaranteed throughput is only for the EBS related traffic. All other traffic is through the other network pipe given to the EC2 Instance (which is not guaranteed and differs by Instance)
  • Comes with additional hourly cost - Optimized m1.large will cost $0.345/hr against a normal on demand m1.large costing $0.320/hr
  • Always use EBS-Optimized Instance when using Provisioned IOPS
Elastic Network Interface (ENI) - This feature is available for Instances launched within Virtual Private Cloud (VPC). When you launch Instances within VPC you normally specify the subnet to launch it in and also specify the private IP address of that EC2 Instance. If the Instance is in a public subnet, you will attach and Elastic IP so that you can access the Instance over the internet. With ENI, you no longer specify a subnet while launching the Instance. You create an ENI (which is attached to a subnet) and attach the ENI to the Instance. So what are the benefits of using ENI?
  • You can attach multiple ENI (belonging to different subnets) to an EC2 Instance
  • One ENI can be configured to have a public IP (Elastic IP) and private IP and the other ENI can be configured to have just a private IP
  • With such a setup, you can allow traffic on port 80 through the first ENI and SSH traffic (port 22) through the other ENI
  • This way you can further secure your VPN architecture by allowing SSH traffic only from your corporate datacenter

Image Courtesy - AWS
Multiple IP Addresses per Instance - with this feature release, AWS addressed the long waited requested for hosting multiple SSL websites on a given Instance. This feature is an extension of  the ENI feature, where each ENI is now allowed to have multiple public and private IPs. We can create an ENI and create multiple Elastic IPs and attach them to the ENI. The ENI will be then attached to the EC2 Instance (during launch or hot attach during running) and the Instance gets multiple IP addresses.
  • The multiple public IP address and private IP addresses are attached to the ENI. And to the Instance when we attach the ENI to the Instance. The difference is that, if we terminate an Instance and relaunch a new one with the same ENI, all the mappings will remain and the new Instance will also get all the public IPs and their private IP mappings
  • Available only within VPC currently
  • Additional cost for each of the additional public IPs that we attach. No cost for the additional private IPs attached
ELB Security Group - When you use an Elastic Load Balancer (ELB), you will obviously place EC2 Instances behind it to accept incoming traffic. For quite a while, on the EC2 Instances Security Group, you will have to open the incoming port (to which ELB will be sending traffic) to all IPs ( This was because of the fact that ELB will scale itself and AWS will not know at a given point of time how many ELB Instances will be running. But now, ELB's come with their own security groups. So on your EC2 Instances (that sit behind the ELB) Security Group, you will have to open up the port to only ELB's security group. This way you secure your EC2 Instances to accept traffic only from ELB.

Internal Load Balancer - You can launch ELB within VPC as an internal load balancer. By this fashion, the ELB's do not have a public IP addresses and will only have private IP addresses. Such a load balancer can be used for internal load balancing requirements between different tiers in the architecture. I wrote a separate article explaining how it can be used.

Custom Security Group of ELB - If you launch ELBs within VPC (either internal or public facing) you have an option of specifying a custom Security Group of your own. This way custom rules can be configured on the ELB's Security Group instead of using the ELB's default Security Group (which cannot be edited).
ELB Custom Security Group

S3 Object Expiration
This feature is great for storing logs and scheduling them to be deleted after a definite period of time. When you have multiple web or application servers you will definitely consider pushing the logs generated on them to S3 for centralized storage. Even if you are running a minimal setup, you certainly do not want run out of space on your Instance. But you do not want to store the logs perpetually - one month old logs will be good enough for most of the applications. Instead of writing custom scripts that periodically deletes the logs stored in S3, use this feature so that Amazon will automatically delete them. Saves some cost.

Route 53 Alias record

Route53 is a scalable DNS service from AWS. With pay-as-you-go pricing, there are whole lot of benefits (in terms of flexibility) with Route53. It is definitely not sophisticated like other DNS service providers in terms of feature set. Be it latency based routing or weighted round robin, the service is updated with features. One feature that will be of interest is "Alias Record". Let's say you have a website called "". And you are hosting the infrastructure in AWS. If your load balancer layer uses AWS Elastic Load Balancing, then you will be provided with a CNAME for the ELB. ELB internally may employ "n" number of servers to scale itself and hence it may have more than one IP. Hence AWS provides the CNAME for an ELB which may resolve to one or more IPs at any time. With that background,

  • You want to create a DNS record for ""
  • You also need to create a DNS record for ""; the naked domain or top level domain. Most of the users will use only the naked domain
By design, DNS does not support adding a CNAME entry for the naked domain - simply put you cannot point "" to an ELB but you can point "" to an ELB CNAME. The naked domain can be pointed to only an A-record. Having only A-records for the naked domain will introduce complexity in HA and Scalability for the load balancing layer. And if your load balancing layer employs ELB, there is no way you will know the IPs before hand. One way to address this constraint is to put a separate Apache server which can do the translation. But this will be a serious bottleneck in the architecture since most of the users will use only the naked domain and soon the infrastructure will crumble even though we have build HA and Scalability in all other layers. With the "Alias" feature

  • First create a Route53 record set for "" and point it to the ELB CNAME
  • Create another record set for "" and set the type as "Alias". Point the record value to the record set created above. Or directly point it to the ELB CNAME
  • With the "Alias" option you will be able to point the record value to either an ELB CNAME or another record set that is available in Route53.
This way, you take care of both naked domain and "www" with both pointing to the load balancing layer in the AWS setup.

Of course there are a whole lot of features and updates that AWS keeps pushing across its services. The above are some of the features that helps architects while designing the infrastructure architecture. These have personally helped us build better architectures. Prior to these features, we always had to look for alternatives such has tuning, custom solutions or sacrifice on certain parameters to achieve others. I am sure there are many more such updates that we can uncover and will be coming from AWS in future. That will be for another post here.

Monday, August 20, 2012

The AdWantageS

Every customer has a reason to move in to the cloud. Be it cost, scalability, on demand provisioning, there are plenty of reasons why one moves into the Cloud. The latest whitepaper "The Total Cost of (Non) Ownership of Web Applications in the Cloud" by Jinesh Varia, Technical evangelist of Amazon Web Services provides a good insight between hosting an infrastructure in-house and in the Cloud (AWS). There are plenty of pricing models available with AWS currently which can provide cost benefits ranging from 30% to 80% when compared to hosting the servers in-house.

On-Demand Instances - this is where every one starts with. You simply add your credit card to your AWS account and start spinning up Instances. You provision them on demand and pay for how long you run them. You of course have the option of stopping and starting them whenever needed. You are charged for every hour of running an Instance. For example, a Large Instance (2 CPU, 7.5GB Memory, Linux) you will pay $0.32/hr (US-East).

Reserved Instances - let's say after you migrate/host your web application to AWS and run multiple web servers and DB servers in AWS. After couple of months, you may notice that some of your servers will run 24 hours/day. You may spin up additional web servers during peak load but will at least run 2 of them always; plus a DB server. For such cases, where you know that you will always run the Instances, AWS provides an option for reducing your cost - Reserved Instances. This is purely a pricing model, where you purchase the required Instance type (say Large/X-Large) in a region for a minimum amount. You will then be charged substantially less for your on-demand charge of that Instance. This way there is a potential savings of 30% over a period of one year when compared to on-demand Instances. The following provides an illustration of the cost savings for purchasing an m1.large Reserved Instance against using it on demand through an year
Cost comparison between On-Demand and Reserved Instance for m1.large Linux Instance in US-East

Be careful that,
  • Reserved Instances are purchased against an Instance type. If you purchase an m1.large Reserved Instance, at the end of the month when your bill is generated, only m1.large usage will be billed at the reduced Instance hourly charge. Hence, on that given month if you figure out m1.large is not sufficient and move up to m1.xlarge, you will not be billed at the reduced hourly charge. In such a case, you may end up paying more to AWS on an yearly basis. So, examine your usage pattern, fix your Instance type and purchase a Reserved Instance.
  • Reserved Instances are for a region - if you buy one in the US-East region and later decide to move your Instances to US-West, the cost reduction will not be applicable for Instances running out of US-West region
Of course, you have the benefits of,
  • Reduced cost - a minimum of 30-40% cost savings which increases if you purchase a 3-year term
  • Guaranteed Capacity - AWS will always provide you the number of Reserved Instances you have purchased (you will not get an error saying "Insufficient Capacity")
Spot Instances - in this model, you will bid against the market price of an Instance and if your's is the highest bid then you will get an Instance at your bid price. The Spot Market price is available through an API which can be queried regularly. You can write code that will check for the Spot Market price and keep placing bids mentioning the maximum price that you are willing to pay against the current Spot Market price. If your bid exceeds then you will get an Instance provisioned. The spot price will be substantially low than on demand pricing. For example, at the time of writing this article, the spot instance pricing for a m1.large Linux Instance in US-East was $0.026/hr as against $0.32/hr on demand pricing. This provides about 90% cost reduction on an hourly basis. But the catch is,
  • The Spot Market price will keep changing as other users place their bids and AWS has additional excess capacity
  • If your maximum bid price falls below the Spot Market price, then AWS may terminate your Instance. Abruptly
  • Hence you may loose your data or your code may terminate unfinished.
  • Jobs that you anticipate to be completed in one hour may take few more hours to complete
Hence Spot Instances are not suitable for all kind of work loads. Certain work loads like log processing, encoding can exploit Spot Instances but requires writing careful algorithms and deeper understanding. Here are some of the use cases for using Spot Instances.

Now, with that basic understanding, let's examine the whitepaper little carefully not just from cost point of view. Web applications can be classified in to three types based on traffic nature - Steady Traffic, Periodic Burst and Always Unpredictable. Here is a summary of the comparison of benefits of hosting them in AWS
AWS benefits for different type of web applications

Steady Traffic
The website has steady traffic. You are running couple of servers on-premise and consider moving it to AWS. Or you are hosting a new web application on AWS. Here's the cost comparison from the whitepaper

Source: AWS Whitepaper quoted above

  • You will most likely start with spinning up On-Demand Instances. You will be running couple of them for web servers and couple of them for your database (for HA)
  • Over long run (3 years) if you only use On-Demand Instances, you may end up paying more than hosting it on-premise. Do NOT just run On-Demand Instances if you your usage is steady
  • If you are on AWS for about couple of months and are OK with the performance from your setup, you should definitely consider purchasing Reserved Instances. You will end up with a minimum of 40% savings against on-premise infrastructure and about 30% against running On-Demand Instances
  • You will still benefit from spinning up infrastructure on demand. Unlike on-premise, where you need to plan and purchase ahead, here you have the option of provisioning on demand; just in time
  • And in case, you grow and your traffic increases, you have the option to add more capacity to your fleet and remove it. You can change server sizes on demand. And pay only for what you use. This will go a long way in terms of business continuity, user experience and more sales
  • You can always mix and match Reserved and On-Demand Instances and reduce your cost whenever required. Reserved Instances can be purchased anytime
Periodic Burst
In this scenario, you have some constant traffic to your website but periodically there are some spikes in the traffic. For example, every quarter there can be some sales promotion. Or during thanks giving or Christmas you will have more traffic to the website. During other months, the traffic to the website will be low. Here's the cost comparison for such a scenario
Source: AWS Whitepaper quoted above

  • You will spin up On-Demand Instances to start with. You will run couple of them for web servers and couple of them for the database
  • During the burst period, you will need additional capacity to meet the burst in traffic. You need to spin up additional Instances for your web tier and application tier to meet the demand
  • Here is where you will enjoy the benefits of on demand provisioning. This is something that is not possible in on-premise hosting. In on-premise hosting, you will purchase the excess capacity required well ahead and keep running them even though the traffic is less. With on demand provisioning, you will only provision them during the burst period. Once the promotion is over, you can terminate those extra capacity
  • For the capacity that you will always run as the baseline, you can purchase Reserved Instances and reduce the cost up to 75%
  • Even if you do not purchase Reserved Instances, you can run On-Demand Instances and save around 40% against on-premise infrastructure. Because, for the periodic burst requirement, you can purchase only during the burst period and turn off later. This is not possible in an on-premise setup where you will anyways purchase this ahead of time
Always Unpredictable
In this case, you have an application where you cannot predict the traffic all the time. For example, a social application that is in experimental stage and you expect it to go viral. If it goes viral and gains popularity you will need to expand the infrastructure quickly. If it doesn't, then you do not want to risk a heavy cap-ex. Here's the cost comparison for such a scenario
Source: AWS Whitepaper quoted above

  • You will spin up On-Demand Instances and scale them according to the traffic
  • You will use automation tools such as AutoScaling to scale the infrastructure on demand. You can align your infrastructure setup according to the traffic
  • Over a 3 year period, there will be some initial steady growth of the application. As the application goes viral you will need to add capacity. And beyond its lifetime of say, 18 months to 2 years the traffic may start to fall
  • Through monitoring tools such as CloudWatch you can constantly tweak the infrastructure and arrive at a baseline infrastructure. You will figure out that during the initial growth and "viral" period you will need certain baseline servers. You can go ahead and purchase Reserved Instances for them and mix them with On-Demand Instances when you scale them. You will enjoy a cost saving benefit of around 70% against on-premise setup
  • It is not advisable to plan for full capacity and run at full capacity. Or purchase full Reserved Instances for the full capacity. If the application doesn't go well as anticipated, you may end up paying more to AWS than the actual
As you can see, whether you have a steady state application or trying out a new idea, AWS proves advantageous from different perspectives for different requirements. Cost, on-demand provisioning, scalability, flexibility, automation tools are things that even a startup can think off and get on board quickly. One question that you need to ask yourself is "Why am I moving in to AWS?". Ask this question during the early stages and spend considerable time in design and architecture for the infrastructure setup. Otherwise, you may end up asking yourself "Why did I come into AWS?".

Tuesday, July 3, 2012

Yet another post on High Availability

Looks like High Availability, Scalability are becoming the most abused words / jargons these days. And Cloud Computing has helped these words reaching the common man. I was really annoyed at an article on High Availability on CloudAve. The article was posted immediately after the latest AWS outage and talks about how companies like Netflix and Pinterest have all got it wrong and how to "program" for failover. Or is that failure? Here are the main items that the article talks about

  1. Programming for failover / failure. Anyone who doesn't is a fool
  2. Doing an S3 sync can help you avoid outages
  3. Elastic Load Balancing can ensure regions are healthy

1. Agreed. Any large scale system should definitely be designed for failure. And I am sure the folks at Netflix have not ignored this fact. Netflix is at a scale where they are probably one of the largest customer of AWS and they indeed work very closely with AWS. And to imagine what their scale is, they made their own custom AWS management solution called Asgard  and open sourced it. They were not happy with the feature set of AWS management console and built their own. And the news also appeared in AWS blog. Now, I am at loss of words if Netflix hasn't thought about using multiple AZs, load balancers, etc...That is like a bare minimum for their scale.

2. Syncing between S3 buckets. Here we are talking about Syncing large amount of objects between S3 buckets located across continents. Over the Internet. Now, this looks all the more simple task to do in paper. Anyone who has thought about the actual implementation will fairly appreciate the complexity and cost involved in doing so. Of course that may outweigh the benefits of being available during an outage. But before we get to that there are many implementation bottlenecks to be addressed. You just cannot imagine putting a Single EC2 Instance (oh yeah we are talking about HA. lets make it one Instance in each AZ) and writing a custom script to perform the sync. Bandwidth availability from a single EC2, availability of that setup during an outage and inter region internet bandwidth charges should definitely be considered to go for such a design. And in large scale architecture, companies heavily rely on Content Distribution Network (CDN) such as Akamai / AWS CloudFront to deliver the media assets. If the content is stored in S3, it is most likely served through a CDN. In such case, even if the origin server is not available (in this case an outage at AWS), the content is always available at the edge location. Apart from the media assets, companies on the likes/scale of Netflix will heavily use other technologies like NoSQL (Mongo, Cassandra), Relational Databases (RDS) and Cache Clusters. And they are probably running 100s in each. Replicating them, building failover is a slow and complex process.

3. Elastic Load Balancing is a load balancing service from AWS. It can be used to bring HA in to your architecture - within a region. You cannot use ELB to ensure regions are healthy. ELB is a service that is restricted to a specific region and operates within it. To detect multiple region health status, one needs a solution at the DNS level. Such as going for a managed DNS solution which can perform health checks between multiple data centers (geo distributed) and failover. AWS Route53 can do that. Route53 Latency Based Routing (LBR) can detect multi region latency and route traffic accordingly.

Building a Highly Available solution is not about just utilizing multiple data centers or syncing data. It involves very deep understanding of the application and the infrastructure. And how they work together. Availability is perceived by the user. Hence it varies from application to application. For example, a photo sharing website might have three major tiers - photo viewer, photo upload, commenting. If the website is completely hosted on AWS, during at outage (depending upon which AWS services are affected), some of the tiers can still be perceived to be working. Photo viewer might continue to serve the pictures from a CDN while users might not be able to add new comments. Or, new comments might be working for the users who are posting but might be delayed for the ones who are viewing (achieved through queues and cache clusters). Architects can think of various ways to build such systems and that's how one increases the overall availability of the system. It is not an one time activity but a continuous process requiring patience, careful examination and planning. Guess such architectures will quietly work behind the scenes and let others abuse words like High Availability, Scalability for eternity.