IOPS: Benchmarking Disk I/O – AWS vs DigitalOcean

IOPS Consideration

Consider that you are extending your operations to cloud or planning to change your hosting cloud provider or buying a new VPS. How do you decide the VPS provided by a cloud provider will suit your requirement ?

There are many aspects that you need to try and test before choosing a provider. One such aspect is how fast is your server based on disk performance (IOPS – Input Output per second) and how do you measure its performance ?

In this article, we will see how to do this and we will compare the performance between two different providers.

How to Do It?

Many of you might have used dd command (device drivers) to check the speed of the drive. But, it has quite a number of problems:

• It is a single-threaded, sequential-write test. If you are running the typical web+database server on your VPS, the number is meaningless because typical services do not do long-running sequential writes.
• There’s no read performance testing at all.

We will be using ‘FIO’ (flexible I/O tester) to do the benchmarking to overcome all the above said drawbacks of dd command.

From the man page:

fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. The typical use of fio is to write a job file matching the I/O load one wants to simulate.

Install FIO

I’ll be doing the test in Ubuntu 14.04. To install FIO in ubuntu, run

apt-get install fio

Let’s test it

I’ll be running the test in VPS running in AWS and Digital Ocean and do the comparison. In AWS, I’ll be using the t2.small instance. The instance details between the providers are below,

• AWS

Instance type – t2.small (2GB RAM, 1 core), General Purpose SSD for storage – 50GiB

• Digital Ocean

Instance type – (2GB RAM, 2 core), SSD for storage – 60GB

Note: The baseline CPU performance of t2.small is 20% of Intel Xeon family processor. However, it’ll earn 12 CPU credits per hour. 1CPU credit = 100% CPU performance for 1 minute. The t2 instance can burst above the base level, provided it has enough CPU credits. More details on that here.

Random Read/Write Performance

Run the following command

fio –randrepeat=1 –ioengine=libaio –direct=1 –gtod_reduce=1 –name=test –filename=test –bs=4k –iodepth=64 –size=4G –readwrite=randrw –rwmixread=75

This will create a 4 GB file, and perform 4KB reads and writes using a 75%/25% (ie 3 reads are performed for every 1 write) split within the file, with 64 operations running at a time. The 3:1 ratio is a rough approximation of your typical database. You can change it as per your need.

Random read/write – Digital Ocean

# fio –randrepeat=1 –ioengine=libaio –direct=1 –gtod_reduce=1 –name=test –filename=test –bs=4k –iodepth=64 –size=4G –readwrite=randrw –rwmixread=75
test: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio 1.59
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m] [100.0% done] [142.2M/49469K /s] [35.6K/12.8K iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=22329
read : io=3070.5MB, bw=132862KB/s, iops=33215 , runt= 23665msec
write: io=1025.6MB, bw=44375KB/s, iops=11093 , runt= 23665msec
cpu : usr=14.62%, sys=45.22%, ctx=30449, majf=0, minf=17
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued r/w/d: total=786042/262534/0, short=0/0/0

Run status group 0 (all jobs):
READ: io=3070.5MB, aggrb=132861KB/s, minb=136050KB/s, maxb=136050KB/s, mint=23665msec, maxt=23665msec
WRITE: io=1025.6MB, aggrb=44375KB/s, minb=45440KB/s, maxb=45440KB/s, mint=23665msec, maxt=23665msec

Disk stats (read/write):
vda: ios=779743/260397, merge=0/6, ticks=400068/754896, in_queue=1154500, util=99.55%

Random Read/Write – AWS

/home/ubuntu# fio –randrepeat=1 –ioengine=libaio –direct=1 –gtod_reduce=1 –name=test –filename=test –bs=4k –iodepth=64 –size=4G –readwrite=randrw –rwmixread=75
test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.1.3
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m] [100.0% done] [11136KB/3640KB/0KB /s] [2784/910/0 iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=13241: Mon Jun 13 09:47:55 2016
read : io=3071.7MB, bw=9028.2KB/s, iops=2257, runt=348403msec
write: io=1024.4MB, bw=3010.7KB/s, iops=752, runt=348403msec
cpu : usr=1.28%, sys=2.36%, ctx=1014229, majf=0, minf=22
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=786347/w=262229/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
READ: io=3071.7MB, aggrb=9028KB/s, minb=9028KB/s, maxb=9028KB/s, mint=348403msec, maxt=348403msec
WRITE: io=1024.4MB, aggrb=3010KB/s, minb=3010KB/s, maxb=3010KB/s, mint=348403msec, maxt=348403msec

Disk stats (read/write):
xvda: ios=783129/261925, merge=3166/439, ticks=16287332/5908212, in_queue=22196660, util=100.00%

Result: (Random Read / Write)

Digital Ocean

• Digital Ocean’s SSD performing ‘33215’ read operations per second and ‘11093’ write operations per second
• The test got completed in ~30 seconds

AWS

• AWS’ SSD performing ‘2257’ read operations per second and ‘752’ write operations per second
• The test got completed in ~5 mins

Random Read Performance

Run the below command to test random read performance.

fio –randrepeat=1 –ioengine=libaio –direct=1 –gtod_reduce=1 –name=test –filename=test –bs=4k –iodepth=64 –size=4G –readwrite=randread

Random Read – Digital Ocean

# fio –randrepeat=1 –ioengine=libaio –direct=1 –gtod_reduce=1 –name=test –filename=test –bs=4k –iodepth=64 –size=4G –readwrite=randread
test: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio 1.59
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [328.7M/0K /s] [82.2K/0 iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=22615
read : io=4096.0MB, bw=346522KB/s, iops=86630 , runt= 12104msec
cpu : usr=14.31%, sys=52.84%, ctx=12981, majf=0, minf=85
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued r/w/d: total=1048576/0/0, short=0/0/0

Run status group 0 (all jobs):
READ: io=4096.0MB, aggrb=346522KB/s, minb=354838KB/s, maxb=354838KB/s, mint=12104msec, maxt=12104msec

Disk stats (read/write):
vda: ios=1030890/3, merge=0/2, ticks=441920/0, in_queue=441828, util=95.57%

Random Read – AWS

/home/ubuntu# fio –randrepeat=1 –ioengine=libaio –direct=1 –gtod_reduce=1 –name=test –filename=test –bs=4k –iodepth=64 –size=4G –readwrite=randread
test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.1.3
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [15880KB/0KB/0KB /s] [3970/0/0 iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=13266: Mon Jun 13 10:03:45 2016
read : io=4096.0MB, bw=12065KB/s, iops=3016, runt=347640msec
cpu : usr=1.07%, sys=2.36%, ctx=1019315, majf=0, minf=88
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=1048576/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
READ: io=4096.0MB, aggrb=12065KB/s, minb=12065KB/s, maxb=12065KB/s, mint=347640msec, maxt=347640msec

Disk stats (read/write):
xvda: ios=1042450/8, merge=5210/2, ticks=22138824/132, in_queue=22139708, util=100.00%

Result: (Random read)

Digital Ocean

• Digital Ocean’s SSD performing ‘86630’ read operations per second.
• The test got completed in ~30 seconds

AWS

• AWS’ SSD performing ‘3016’ read operations per second.
• The test got completed in ~5 mins

Random Write Performance

Run the below command to test random write performance.

fio –randrepeat=1 –ioengine=libaio –direct=1 –gtod_reduce=1 –name=test –filename=test –bs=4k –iodepth=64 –size=4G –readwrite=randwrite

Random Write – Digital Ocean

# fio –randrepeat=1 –ioengine=libaio –direct=1 –gtod_reduce=1 –name=test –filename=test –bs=4k –iodepth=64 –size=4G –readwrite=randwrite
test: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio 1.59
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/39016K /s] [0 /9525 iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=22927
write: io=4096.0MB, bw=35404KB/s, iops=8850 , runt=118471msec
cpu : usr=3.41%, sys=25.19%, ctx=155102, majf=0, minf=17
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued r/w/d: total=0/1048576/0, short=0/0/0

Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=35403KB/s, minb=36253KB/s, maxb=36253KB/s, mint=118471msec, maxt=118471msec

Disk stats (read/write):
vda: ios=1/1048294, merge=0/67571, ticks=0/5491480, in_queue=5491644, util=99.87%

Random Write – AWS

/home/ubuntu# fio –randrepeat=1 –ioengine=libaio –direct=1 –gtod_reduce=1 –name=test –filename=test –bs=4k –iodepth=64 –size=4G –readwrite=randwrite
test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.1.3
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [w] [100.0% done] [0KB/16560KB/0KB /s] [0/4140/0 iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=13273: Mon Jun 13 10:10:59 2016
write: io=4096.0MB, bw=11774KB/s, iops=2943, runt=356237msec
cpu : usr=0.51%, sys=2.89%, ctx=1029874, majf=0, minf=22
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=11773KB/s, minb=11773KB/s, maxb=11773KB/s, mint=356237msec, maxt=356237msec

Disk stats (read/write):
xvda: ios=530/1051034, merge=390/143157, ticks=7336/22762076, in_queue=22770100, util=100.00%

Result: (Random write)

Digital Ocean

• Digital Ocean’s SSD performing ‘8850’ write operations per second.
• The test got completed in ~30 seconds

AWS

• AWS’ SSD performing ‘2943’ write operations per second.
• The test got completed in ~5 mins

IO Latency on Individual Request

We will be using ioping to measure the latency on individual request. Install ioping by running the below command.

apt-get install ioping

Run the below command to measure IO latency using ioping

ioping -c 10 .

IO Latency – DigitalOcean

# ioping -c 10 .
4096 bytes from . (ext4 /dev/disk/by-label/DOROOT): request=1 time=0.2 ms
4096 bytes from . (ext4 /dev/disk/by-label/DOROOT): request=2 time=0.3 ms
4096 bytes from . (ext4 /dev/disk/by-label/DOROOT): request=3 time=0.6 ms
4096 bytes from . (ext4 /dev/disk/by-label/DOROOT): request=4 time=0.3 ms
4096 bytes from . (ext4 /dev/disk/by-label/DOROOT): request=5 time=0.2 ms
4096 bytes from . (ext4 /dev/disk/by-label/DOROOT): request=6 time=0.2 ms
4096 bytes from . (ext4 /dev/disk/by-label/DOROOT): request=7 time=0.2 ms
4096 bytes from . (ext4 /dev/disk/by-label/DOROOT): request=8 time=0.4 ms
4096 bytes from . (ext4 /dev/disk/by-label/DOROOT): request=9 time=0.2 ms
4096 bytes from . (ext4 /dev/disk/by-label/DOROOT): request=10 time=0.2 ms

— . (ext4 /dev/disk/by-label/DOROOT) ioping statistics —
10 requests completed in 9005.3 ms, 3318 iops, 13.0 mb/s
min/avg/max/mdev = 0.2/0.3/0.6/0.1 ms

IO Latency – AWS

/home/ubuntu# ioping -c 10 .
4.0 KiB from . (ext4 /dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada): request=1 time=710 us
4.0 KiB from . (ext4 /dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada): request=2 time=347 us
4.0 KiB from . (ext4 /dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada): request=3 time=312 us
4.0 KiB from . (ext4 /dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada): request=4 time=333 us
4.0 KiB from . (ext4 /dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada): request=5 time=349 us
4.0 KiB from . (ext4 /dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada): request=6 time=294 us
4.0 KiB from . (ext4 /dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada): request=7 time=309 us
4.0 KiB from . (ext4 /dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada): request=8 time=363 us
4.0 KiB from . (ext4 /dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada): request=9 time=337 us
4.0 KiB from . (ext4 /dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada): request=10 time=305 us

— . (ext4 /dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada) ioping statistics —
10 requests completed in 9.0 s, 2.7 k iops, 10.7 MiB/s
min/avg/max/mdev = 294 us / 365 us / 710 us / 116 us

Result: (IO Latency)

Digital Ocean

• Digital Ocean’s IO latency is around 0.3ms on average

AWS

• AWS’ IO latency comes around 356 microseconds, which is around 0.3 ms on average

Analysis

How Does AWS General Purpose SSD Work

From AWS documentation:

I/O Credits and Burst Performance

General Purpose (SSD) storage performance is governed by volume size, which dictates the base performance level of the volume and how quickly it accumulates I/O credits. Larger volumes have higher base performance levels and accumulate I/O credits faster. I/O creditsrepresent the available bandwidth that your General Purpose (SSD) storage can use to burst large amounts of I/O when more than the base level of performance is needed. The more credits your storage has for I/O, the more time it can burst beyond its base performance level and the better it performs when more performance is needed.

If you go through the documentation on General purpose SSD EBS volume, you’ll come to know that AWS provide base performance of 3 IOPS per Gibibyte (GiB) and the ability to burst to 3,000 IOPS for General Purpose (SSD) volumes under 1000 GiB.

Note: I made a point that this is for volume size less than 1000GiB.

1 IOPS = 256 KiB ( for SSD volumes )

IOPS stands for Input Output operations per second

For General Purpose SSD volumes less than 1000GiB, you’ll get a base performance of 100 IOPS irrespective of the size of the volume. For example, if the size of your volume is 50 GB, then you’ll get 50 x 3 IOPS = 150 IOPS as baseline performance.

On the other hand, if the size of your volume is 10GB, you’ll still get 100 IOPS and not 10 x 3 = 30 IOPS

For those time where your server is not utilizing the base IOPS, then the excess IOPS will be stored in credits. The credits will be used when there is a need to burst more than the baseline value. It can burst up to a value of 3000 IOPS and that is what we see in the disk performance of AWS.

We saw values of 3009 for read/write, 3016 for read and 2943 for write. Though it has slightly exceeded the 3000 mark, the I/O for general purpose SSD stood at ~3000 max

On the other hand, the SSD from Digital Ocean has no such capping like AWS. It exploded the moment we ran the command and reached values of 44308 for read/write, 86630 for read and 8850 for write operations. It reacted like a jet.

If you prefer AWS for some other aspects based off of your requirement but you need more IOPS than the one provided by general purpose SSD, then you need to go for provisioned IOPS which comes at a higher price than the former or choose a larger volume size.

On the other hand, if all you need is a VPS then by now you should know which gives the real value for money

You can refer the below for General Purpose SSD volumes greater than 1000GiB

For General Purpose (SSD) volumes above 1000 GiB you get a baseline performance of 100 per GiB up to 10000 IOPS. Volumes larger than 1,000 GiB have a baseline performance that is equal or greater than the maximum burst performance, and their I/O credit balance never depletes.

Above statement means, that if you have 1000GiB volume, then you’ll get 1000 x 3 = 3000 IOPS by default as baseline performance. The IO credits will not play any role here. This IOPS can reach a max limit of 10,000 IOPS which is attained at volume size of 3,334 GiB. Above that irrespective of your volume size, you’ll only get a maximum of 10,000 IOPS (Max volume size you can create is 16 TiB for General purpose SSD).

And that’s it for this article !! Hope this might be of help.