Xen file-based vs LVM-based disk images (benchmarks)
I’ve been messing around a lot with Xen lately and have seen several different articles and forum posts debating the advantages over using file-based disk images, like /mnt/xen/VM01-disk.img, versus giving the VM direct access to a LVM partition. So, I ran a few simple tests on my own to determine what would be best for my machine.
Dom-0 is using four 1.5tb 7200rpm seagate drives in a software Raid-10, /dev/md2. Both Dom0 and DomU have 1gb of ram and are using ext3. Xen is using mainly default settings with the default scheduler.
From DomU with the DomU image being file-based on ext3 Dom-0 fs
DomU has 1gb ram
[root@DomU]# dd if=/dev/zero of=tmpfile.bin bs=1024k count=10k 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 192.556 seconds, 55.8 MB/s |
From Dom0 on the same filesystem where above DomU’s image is stored.
Dom0 has 1gb ram
[root@Dom0]# dd if=/dev/zero of=tmpfile.bin bs=1024k count=10k 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 72.0743 seconds, 149 MB/s |
I was really surprised to see there was this much of a difference between Dom0 and DomU disk access. Several tests were run, and the results didn’t vary much at all.
I also ran a bonnie++ benchmark on both Dom0 and DomU.
Dom0:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP razer.justynshul 2G 419 98 144092 21 68102 2 882 97 183977 2 699.3 0 Latency 60846us 1136ms 381ms 53632us 98757us 371ms Version 1.96 ------Sequential Create------ --------Random Create-------- razer.justynshull.c -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 87us 447us 486us 694us 34us 41us 1.96,1.96,razer.justynshull.com,1,1304536826,2G,,419,98,144092,21,68102,2,882,97,183977,2,699.3,0,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,60846us,1136ms,381ms,53632us,98757us,371ms,87us,447us,486us,694us,34us,41us |
DomU:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP razer2.justynshu 2G 446 99 94601 12 62259 1 1006 99 130897 1 182.1 0 Latency 64057us 1771ms 2969ms 19222us 1887ms 348ms Version 1.96 ------Sequential Create------ --------Random Create-------- razer2.justynshull. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 44083us 395us 410us 59us 10us 66us 1.96,1.96,razer2.justynshull.com,1,1304538856,2G,,446,99,94601,12,62259,1,1006,99,130897,1,182.1,0,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,64057us,1771ms,2969ms,19222us,1887ms,348ms,44083us,395us,410us,59us,10us,66us |
Round 2 – LVM-based disk on DomU
Here are the results of the same tests, except this time DomU is using LVM as the root disk instead of a file-based image. It is still on the same exact hardware and drives/raid set-up.
[root@DomU]# dd if=/dev/zero of=tmpfile.bin bs=1024k count=10k 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 123.34 seconds, 87.1 MB/s |
Bonnie++ test:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP CentOS 2G 444 99 78784 6 48588 1 957 97 190338 1 612.1 0 Latency 18285us 433ms 306ms 33926us 48079us 456ms Version 1.96 ------Sequential Create------ --------Random Create-------- CentOS -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 6908us 386us 402us 433us 23us 33us 1.96,1.96,CentOS,1,1304828315,2G,,444,99,78784,6,48588,1,957,97,190338,1,612.1,0,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,18285us,433ms,306ms,33926us,48079us,456ms,6908us,386us,402us,433us,23us,33us |
I should re-run the bonnie tests with a large test size since these may be inaccurate with only 2g. Going purely off of the dd results, using LVM is the way to go.
The Sequential Output from bonnie++ is higher for the DomU using a file-based disk image, but that doesn’t match up to any of the dd tests on the same DomU. When I have time, I’ll re-run the bonnie tests with larger file sizes to see if the results stay the same or not.
Tags: CentOS, lvm, virtualization, Xen
May 31st, 2011 at 4:04 pm
Hey I found this information really useful for I doing a research with Xen as well and I was debating over how to create my VMs in DomU. Have you run other tests as you planned to. I would like to see more results. And can you give a me good link follow to create a LVM for DomU. Thanks
June 15th, 2011 at 6:02 pm
I haven’t had the time yet to run any more tests since this post, but I still plan to. Hopefully sometime soon.
As for creating a LVM for a DomU, it’s very simple. Create the lvm like normal:
lvcreate -L30G -n testvm1 XenVMAnd then for the disk in the DomU xen configuration, you’d use something like this for the disk:
disk = ['phy:/dev/XenVM/testvm1,sda1,w']August 3rd, 2011 at 12:25 pm
Try this url, helped me a lot
http://www.redhatlinux.info/2010/11/lvm-logical-volume-manager.html