Virtualization: Performance Comparison

Introduction

As you remember from my previous article, I have been interested in testing the performance levels of two virtual SAN configurations from different vendors. I got my results, but this experience prompted me to continue. Here, I’ve chosen to try another configuration for performance comparison, albeit with only a slightly different list of participants.

Since no one needs an introduction from VMware vSAN, I’d like to say a few words about its companion – Ceph. Basically, it is an object-based software storage platform. I know that doesn’t sound epic at all, but Ceph is also completely fault-tolerant, uses off-the-shelf hardware, and is extremely scalable. The most interesting thing is that some Ceph releases apply erasure-coded data pools so that it would be a less resource-hungry solution than traditional replicated pools. In practice, that means the following: when you store an object in a Ceph storage cluster, the algorithm divides this object into data and coding chunks, stored in different OSDs (that way, the system could lose an OSD without actually losing the data).

Now, that’s when I thought that theoretically, Ceph could make a good virtualization platform (proper configuration, of course), so I had to see whether it would be justified in terms of time and resources spent. Naturally, I hardly could have done it without a credible comparison, hence VMware vSAN (with a similar configuration, of course, otherwise it would make no sense).

So, shall we?

Time to Look Under the Hood!

So, today’s participants are VMware vSAN’s 4-node configuration (RAID5 – erasure coding) and Ceph (Erasure coding 3+1). Keep in mind, though, that my goal is rather to establish if Ceph is up to the task than to determine which configuration gives more performance.

Now it seems to be the time to take a look at the test environment:

Hardware:

Software:

Attention:

To create a vSAN datastore on each node, you’ll need a disk for caching for each node. The vSAN Storage Policy allows you to stripe an object across disks, so we will require testing parameter “Number of disk stripes per object = 1, 3, 6” as well, to determine the optimal configuration. VMware vSAN stripes data as follows:

Ceph:

Attention:

Each client VM would address a separate 256GB RBD disk. The exclusive-lock & object-map parameters were disabled. On OSD VM NVMe, Intel® Optane™ SSD DC P4800X has been partitioned to 6 50GB logical volumes with bluestore db and wal for Ceph OSD each.

Erasure Coding Pool Parameters:

Ceph Cluster configuration file: ceph.conf.

Testing methods:

VMware vSAN

Testing will be performed with the following configuration: “HCIBench + fio”.

HCIBench will create test VMs with zeroed 100GB disk and launch fio with the following parameters:

Random Read 4k:

Random Write 4k:

Random Read 64k:

Random Write 64k:

The amount of VMs will be increased by 4 each time to establish the highest performance possible.

As usual, HCIBench will keep on clearing the cache after each testing (Clear Read/Write Cache Before Each Testing is the option designed for vSAN users to flush the cache tier before each test case).

As you already know, testing time will differ because VMware vSAN has caching. So, it’ll take some time until it’s clear and ready to go (in the case with All-flash – only write operations).

Ceph

Testing will be performed with the following configuration: “CLI VM + fio”.

Random Read 4k:

Random Write 4k:

Random Read 64k:

Random Write 64k:

Results were as follows:

Test time:

vSAN – 3600 sec;

Ceph – 600 sec;

NVMe and SSD performance testing:

As declared by the manufacturer:

VMware vSAN 16VM with “Number of disk stripes per object = 1,3 and 6” (the amount of disks in each node) parameters performance testing results:

According to our test results, “Number of disk stripes per object = 1” is the optimal parameter for this configuration in terms of performance.

VMware vSAN testing results (I only count final results, since vSAN, as you remember, uses caching):

VMware vSAN test results:

Ceph testing results:

<thCPU OSD USAGECPU CLIENT USAGE

It becomes apparent that Ceph erasure coding operations require a lot of CPU resources, so that’s pretty much the only thing that counts on OSD VMs.

OSD VMs CPU  (random read 4k):

OSD VMs CPU  (random write 4k):

VMware vSAN vs Ceph:

Conclusions

Well, the results are definitive, but the conclusions are not as simple. VMware vSAN showed impressive performance of erasure coding operations, and even though Ceph manufacturers do give some explanation in that regard (look for “How fast is it?”), it still is slower.

However, since the speed is practically the only issue, whether to use Ceph or not is up to you. You got results and everything you need, and I only hope that my work was of any use.