IOPS (Input Output Per Second) is a common benchmark for storage performance. IOPS basically is a measurement of how many reads or writes can be performed in a second. Storage performance is very important when it comes to how well your application will perform. Whether its a web server or a database server, good storage performance is critical to overall application performance. For example, with a vanilla WordPress LAMP stack, 500 IOPS can mean a site that takes 4 seconds to load. Compare that to a server with 2000 IOPS and the same website will take less than 1 second to load. We perform a Cloud IOPS Comparison to see where we get the best performance.
- 7,200 RPM SATA Drive: 75-100 IOPS
- 10k RPM SATA Drive: 125-150 IOPS
- Corsair Force Series GT SSD: 85k IOPS
- EMC DSSD D5: 10 Million IOPS
As we move most of our server hosting to Infrastructure as a Service cloud providers, we need to look closer at the performance they offer. Cloud providers usually provide fairly clear details about CPU Cores, Memory, and Storage Sizes. But they don’t provide clear details about storage performance. I’m going to benchmark 3 popular Cloud IaaS providers, Amazon AWS, Microsoft Azure, and Digital Ocean. Which one of these will provide better IOPS on their Virtual Machines?
I performed two types of tests to benchmark storage performance. First a set of IOPS performance tests to measure IOPS for each virtual servers using FIO 2.0.9. I first ran 3 sets of Combined Random Read/Write tests, they were configured for 75%/25% Read/Write.
./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
Then I used FIO to run just a Random Read test with the following command:
./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randread
After that I ran a Random Write test with this command:
./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
Lastly I measured storage latency using IOPing.
./ioping -c 10 .
For all of the tests, I tried to spread them over 24 hours to account for any peak utilization times. I referenced the excellent article at Binarylane, for how to setup FIO and IOPing. All of these were on Ubuntu 15 servers with as similar CPU and Memory configurations as possible.
I used an AWS EC2 t2.micro (1 CPU core, 1GB of memory) with a GP2 volume attached to it. Amazon describes the GP2 as “General Purpose SSD”, that offers cost-effective storage that is ideal for a broad range of workloads. They claim a GP2 volume has single digit millisecond latencies and the ability to burst to 3,000 IOPS for extended periods of time. Their claimed IOPS is between 100 IOPS and 10,000 IOPS. The monthly cost for this Virtual Machine is $17.64.
On Azure I used a Standard A1 (1 core, 1.75GB memory) Virtual Machine. The attached disk is their Standard disk, with a claimed Max IOPS of 500, Throughput Limit of 60 gb/s. The Standard A1 Virtual Machines costs $44.64 a month.
I also setup a 2nd Virtual Machine, a Standard DS1 (1 core, 3.5GB memory). Azure claims the Max Disk IOPS on the Standard DS1 is 3,200. The price tag on the Standard DS1 is $57.29.
In Digital Ocean I used their entry level $5 Virtual Machine (1 core, 512MB, 20GB of storage). All Digital Ocean servers have SSD disks. Digital Ocean does not report any details around storage performance. Digital Ocean has 5 different sizes of offers the same level of storage for each VM, only the size increases. Their highest end VM has 4 Cores, 8GB of memory, and 80GB of storage, costing $80/month.
Cloud IOPS Comparison Results
As the results show Digital Ocean, was blazingly fast. On the straight random read test Digital Ocean almost hit 100,000 IOPS. Each test took about 12 – 15 seconds to run on the Digital Ocean VM. Azure A1 was very slow. The IOPS tests took about 90 minutes each to run on the Azure A1. The Azure DS1 was noticeably faster than the A1 instance, consistently going over 1000 IOPS. The DS1 instance took about 30 minutes to run each test. The AWS t2.micro easily pulled ahead of the Azure instances and also had very consistent results from test to test.
|AWS t2.micro||Azure A1||Azure DS1||Digital Ocean|
|IOPS Test 1 Combined R/W – Read||2,250||231||1,114||44,035|
|IOPS Test 1 Combined R/W – Write||750||77||371||14,632|
|IOPS Test 2 Combined R/W – Read||2,246||190||1,117||37,041|
|IOPS Test 2 Combined R/W – Write||752||63||371||12,322|
|IOPS Test 3 Combined R/W – Read||2,251||191||1,114||43,949|
|IOPS Test 3 Combined R/W – Write||748||63||371||14,653|
|IOPS Read Test||2,918||298||4,080||98,959|
|IOPS Write Test||2,929||185||654||16,135|
The Azure DS1 had the best results in the IOPing test, and not surprisingly the Azure A1 instance had the worst performance. Digital Ocean had the 2nd best results here, and the AWS t2.micro was just slightly better than the Azure A1 results. In general all the results were fairly close, although the latency of the Azure Premium Disks seems to be a stand out performer.
|AWS t2.micro||Azure A1||Azure DS1||Digital Ocean|
|IO Ping Min||0.3||0.6||0.1||0.1|
|IO Ping Max||0.8||0.9||0.2||0.4|
|IO Ping Average||0.6||0.7||0.2||0.3|
|IO Ping Std Dev||0.2||0.1||0||0.1|
AWS’s performance basically matches the claims they make, and for roughly $17/month they offer a great product. For raw power, at a low price, Digital Ocean really steals the show. For $5/month you get R/W IOPS over 40,000, it’s a pretty great deal.
Azure’s performance was very underwhelming. Considering that Azure charges a premium price for their IaaS, the performance certainly does not fit the price tag. In all fairness, Azure does offer Virtual Machines with higher performance. The GS series of Virtual Machines has a claimed max IOPS of 5,000 for the GS1, going all the way to 80,000 IOPS with the GS5. Of course the monthly price for the GS1 is $453 and the GS5 is $6,456.36.
Both Azure and AWS offer much more than IaaS, each of these vendors has built and ecosystem of interconnected components to allow for developers to build complex applications and for enterprises to shift entire datacenters to the cloud. Digital Ocean on the other hand has a more focused offering. They offer high performance virtual machines in the cloud. Their automation capabilities are on par with AWS and Azure, but the performance of their servers is top tier. If you are looking for pure server performance, Digital Ocean is the winner here. In an AWS vs. Azure comparison, AWS seems to be the winner, with a better performing product at a lower price point.