In short it's demonstrating that NPT or FV alone provide useful boosts but still land you with an inefficient VM. Whereas the combination provides a huge boost and makes the VM come close to native performance!
I think it's important that all the benchmarks should show native machine performance where possible. eg run workload 1 on server X. Run workload 2 on server X. Now run workload 1 and/or 2 on server X in a VM.
The OLTP test ("Oracle Swingbench") is not limited by the harddisk performance. We test with six 15000 rpm seagate SAS 300 GB disks (more or less the fastest harddisk you can get) in RAID-0 and we have performed the same test with twelve disks. The extra disks gave a 1% boost, so our disk system is not the bottleneck.
You might want to consider 6x Non-RAID or 3x RAID-0 for database benchmarking. Databases are smart and the parallelism is achieved inside the database and I/O has to be parallel as well. One big 6x RAID0 is slow.
If you maxed out the parallism (by as many independent drives as possible), you may use RAID0 or RAID10 for at most two drives just to be sure to have always a bandwidth that's >1Gb/s networking performance over the entire platters.
Excellent Suggestion. Although in this setup we are sure that we are definitely not disk (DQL checked and no disk wait cycles) limited, we'll redo the more disk intensive tests.
This is interesting from a kid system point of view but I would be seriously interested in comparing this virtualization to system z with PR/SM && LPAR technologies or z/VM and z/Linux.
Its a shame that the techie hardware sites don't talk about the grown up systems more.
The reasoning is simple.
System/Z plays a different league, period.
You have allready mentioned it Yourself, the whole x86 virtualization is just kids play compared to mainframes.
But hell, the mainframes are also a different league from cost POW - while not so expensive anymore, for casual mid-sized company that may consider this site it is usually just an overkill.
Also, a mainframe (even the smallest one) is not somethig You go out and buy to test ...
In the article it is stated:
"It offers massive tagged TLBs that can support the memory management of the different guest OS and the hypervisor. Intel's Nehalem will use this technique too."
Obviously the TLBs (standard and nested-extended) are of finite size. Couldnt excessive context switching of processes within an OS result in thrashing the TLB cache (exceeding the address space tag maximum for instance)?
Similarily, are there limitations to the NPT-EPT with regards to the maximum number of hosted VMs, or the context switch rate between processes and VMs? In the future it would be nice to see how this technology scales with VM count and activity. I sincerely doubt businesses intend to run a single Win2k3 virtual machine at a time.
Also, now that this is done in hardware, what kind of memory savings can be expected compared to previous software implementations?
Just to show what hardware can do for stability, my recent horror story.
I have 3 Sun X2200 M2 two CPU dual Core fitted with 16Gb and one with 12Gb and setting up a Xen based environment (Virtual Iron 4.3.8)
Things seemed to work smoothly until I noticed that once in a while the node itself, not the VM, seemed to crash. This started an investigation which led to the following conclusion:
All 3 nodes were upgraded to the then latest BIOS/BMC (tools CD version 1.6), mainly because of security fixes and functionality in the eLOM. I can consistently crash a node by stress testing a XP VM using a HDBenchmark together with a CPU benchmark (HD Speed and 7-Zip). Once I degrade a node to a BIOS/BMC version of the 1.3 Tools CD (2.70 3B13 BMC/BIOS) that VM runs perfectly well while under stress.
Another pecularity of the higher BIOS/BMC version than 1.3 is that the crashes occur more or less at the same time, for example when it first crahes around 10:30 then it will crash again around xx:30 whether that is 1 hour later or 10 hours, on average it runs 2-3 hours.
So be careful to state stability statements without reasoning behind it.
Those servers are very stable with VI as long as I stay on the 1.3 Tools release but sadly that looks like an impossiblity in the future.
Any of you techies having an explanation for this, how weird it may seem I'm in for any explaination.
1) I REALLY appreciate your effort to making a reasonable performance benchmark. AFAIK no such 3rd party benchmark with reasonable value is available for free.
2) Please be careful talking about Xen "hypervisor (in)stability".
XEN is as a name You should sorely reserve for Citrix Xen Server or the deep-technical talk about hypervisors.
The stability AND performance of an virtualized solution is mostly affected by the I/O. I/O is NOT part of the hypervisor as occurs in so-called dom0/service-partition (or whatever PR guys rename it) that is essentially na I/O router for all the VM's on the machine.
Also, as You are well aware, there are many virtualization solution utilizing the Xen hypervisor.
In brief, in order of maturity:
Virtual Iron
Citrix's Xen Server (former XenSource XenServer)
Novel's; Red Hat's; and other "Xen-based virtualization" solution from Linux distribution vendors.
Please, do not reproduce the same mistake I see all over the web - using the hypervisor name for describing a virtualization solution characteristics.
(I know Citrix does it intentionally in their PR, but we are technologists not Avareage Joe's to be misled)
3) What I would like to see is have benchmarked the real solution that compete for the market i.e. ESX vs. Xen Server vs. Virtual Iron vs. Hyper-V.
However You take it, the solutions from the Linux OS Vendors have a very limited market penetration. Microsoft too has not market, but everybody knows it will get some so it makes sense to include Hyper-V.
Comparison of Red Hat's Xen solution against their KVM solution may be interesting but the real value for IT people out there is to compare the existing solutions one is going to buy for his possible project.
I appreciate someone looking into such benchmarks, it's about time.
When comparing enterprise level solutions such as VMware ESX and Hyper-V it is probably more directly comparable to the enterprise level Xen put out by Citrix, XenServer, than the free version. The embedded product that ships on select HP systems may be of particular interest as it has the ability to display up to 9 VM's in a grid locally using the servers display driver instead of being required to remote in (also still an option). There is something special about being able to view 4, 6, or 9 VM's at once when you're sand-boxing a QA build, etc.
Just a thought as my experience with CTX has always been a positive one.
This is my first read into this area. I also read the three articles posted at the bottom of this one. Information is the key to understanding and these articles help with this. Like any assignment at university, it will lead you to further studies for a better understanding under different conditions.
This article and the comments, have given me more insight into this area.
You mention KVM yourself Johan, and I'd be very interested if you had a look on it. Any reasons not to include it in the upcomming article? Any timeline on when we might see benchmarks on virtualization including it?
While I understand most current use for Virtualization is in Servers. Would they one day come to gaming / graphics. Being able to use WindowsXP inside Virtual Machine with FULL Graphics power.
That is coming with IOMMU by AMD (the Intel version escapes me). It will probably NOT enable DX10, but it will allow better memory management and throughput for graphics. SO you won't have an emulated S3 graphics card.
You may want to consider fixing the RSS Feed link to container based OS virtualization, as it has not worked for me since day 1. The link from here however does work.
I would also like to add that Xen *IS* free, not cheap, as you can download a binary, or the source from xen.org HVM(hardware virtual machine) have also been possible for over a year, and is supported on all core 2 duo up CPUs, and AMD AM2 on up CPUs. You guys should really check our irc.freenode.org ##xen(if you have not already) and speak to the vetrans who have been running xen para vitual, or hardware virtual for years now.
*shakes head at running MySQL/php on a Win2k3 box*
In all fairness, this test is not even close to being realistic.
1. How many web servers does an organization run?
2. How many SQL servers does an organization run?
3. XEN? Yah XEN is free, the free version.
I Have to tip my hat to VMware. YAH it costs..
BUTTTTT there are things i can do on a vmware server that XEN, rather the freebe, Citrix XEN, even SUN XEN.. They are close, but not even within the same league.
I actually have VMware VIRTUAL servers that out- preform Physical servers.
NEXT time you do something like this, how about give more details. The test that you preformed, is amateur at best.
there are 3 main reasons to even think about virtualization.
1. Legacy server consilidation
2. Disaster scenerios
3. Virtual Infrastructure
Seeing how 4 web servers preform on 1 physical server doesn't make sense. Even Microsoft put out a benchmark on Exchange 2007 on VMware ESX. 10,000 users, on 2 VMware ESX servers. now that my friend is a benchmark.
NEXT demo/test, use the actual parts. You have failed to show actual cpu/memmory utilization. that is a XEN drawback. CPU outstanding, but Memmory utilization horrible.
The ability to do accurate CPU sharing (Resource sharing) is HORRIBLE. Network configurations and throughput, HORRIBLE
It is a blogpost, not a full article with full benchmark configuration disclosure. That is mentioned more than once in this post.
Tests were performed based on the input of real companies. Yes, in some cases, just putting a lot of mailservers together might be more relevant. But as you indicate yourself, microsoft and VMWare produce more than enough exchange benchmarks. So if we can chose, we chose a benchmark scenario that is realistic, but not beaten to dead.
Virtual servers can only outperform physical servers if your application scales badly over more cores. You load balance your load over several virtual machines, and compare this with with one application that has to scale over many CPUs. But otherwise virtual servers can not outperform physical ones.
I am not saying Xen is better than VMWare. I am just saying we found out that Xen is finally ready to run Windows. I have seen a lot of claims that it should work on Xen 3.0.x, but we found that it was impossible to run any kind of Windows on that version. But with Xen 3.2, it seems to be very stable.
We run on ESX Server 3.0.2 and I have been resisting attempts to move some of the heavier applications such as SQL Server 2005, Exchange Server 2003 and other LoB applications due to concerns about performance. I am hoping that the combination of upgrading ESX Server 3.5 plus NPT plus IOMMU would provide the necessary throughput for these applications.
So please include the latest version of ESX Server in your benchmarks. Also, even though the vendors have thier own whitepapers on SQL Server and Exchange Server I would still like to see some independent benchmarks. It would be interesting to make comparisons with and without NPT, and IOMMU when it comes out.
It's INSANELY difficult to find any good reviews/benchmarks/non vendor-speak about virtualization platforms; looking forward to this article! Even though my virtualization process doesn't involve any heavy duty webservers, seeing how it performs under that load will be cool.
Hopefully you'll cover a bit on the install/setup process to; I only mention this because from my experiences, both VMWare's ESX, and Xen's products could both use a bit of spit-shine to the end user, techy or not.
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
20 Comments
Back to Article
andy987s - Tuesday, August 19, 2008 - link
Red Hat magazine have an interesting benchmark showing the combination of NPT hardware support plus Para-Virtualised drivers makes a huge difference. See http://www.redhatmagazine.com/2008/07/03/red-hat-e...">http://www.redhatmagazine.com/2008/07/0...na-with-...In short it's demonstrating that NPT or FV alone provide useful boosts but still land you with an inefficient VM. Whereas the combination provides a huge boost and makes the VM come close to native performance!
I think it's important that all the benchmarks should show native machine performance where possible. eg run workload 1 on server X. Run workload 2 on server X. Now run workload 1 and/or 2 on server X in a VM.
7oby - Friday, August 15, 2008 - link
You wrote:The OLTP test ("Oracle Swingbench") is not limited by the harddisk performance. We test with six 15000 rpm seagate SAS 300 GB disks (more or less the fastest harddisk you can get) in RAID-0 and we have performed the same test with twelve disks. The extra disks gave a 1% boost, so our disk system is not the bottleneck.
You might want to consider 6x Non-RAID or 3x RAID-0 for database benchmarking. Databases are smart and the parallelism is achieved inside the database and I/O has to be parallel as well. One big 6x RAID0 is slow.
If you look at what VMMark benchmarks are typically performed:
http://www.vmware.com/products/vmmark/results.html">http://www.vmware.com/products/vmmark/results.html
They use a RAID10 setup with 7 independet drives.
Here's another article about the superiour performance of a RAID1 setup compared to a RAID10 in db benchmarking due to the additional parallesim:
http://blogs.zdnet.com/Ou/?p=484&page=2">http://blogs.zdnet.com/Ou/?p=484&page=2
If you maxed out the parallism (by as many independent drives as possible), you may use RAID0 or RAID10 for at most two drives just to be sure to have always a bandwidth that's >1Gb/s networking performance over the entire platters.
JohanAnandtech - Monday, August 18, 2008 - link
Excellent Suggestion. Although in this setup we are sure that we are definitely not disk (DQL checked and no disk wait cycles) limited, we'll redo the more disk intensive tests.gtrider - Wednesday, July 23, 2008 - link
This is interesting from a kid system point of view but I would be seriously interested in comparing this virtualization to system z with PR/SM && LPAR technologies or z/VM and z/Linux.Its a shame that the techie hardware sites don't talk about the grown up systems more.
mino - Thursday, July 24, 2008 - link
The reasoning is simple.System/Z plays a different league, period.
You have allready mentioned it Yourself, the whole x86 virtualization is just kids play compared to mainframes.
But hell, the mainframes are also a different league from cost POW - while not so expensive anymore, for casual mid-sized company that may consider this site it is usually just an overkill.
Also, a mainframe (even the smallest one) is not somethig You go out and buy to test ...
mabellon - Tuesday, July 22, 2008 - link
In the article it is stated:"It offers massive tagged TLBs that can support the memory management of the different guest OS and the hypervisor. Intel's Nehalem will use this technique too."
Obviously the TLBs (standard and nested-extended) are of finite size. Couldnt excessive context switching of processes within an OS result in thrashing the TLB cache (exceeding the address space tag maximum for instance)?
Similarily, are there limitations to the NPT-EPT with regards to the maximum number of hosted VMs, or the context switch rate between processes and VMs? In the future it would be nice to see how this technology scales with VM count and activity. I sincerely doubt businesses intend to run a single Win2k3 virtual machine at a time.
Also, now that this is done in hardware, what kind of memory savings can be expected compared to previous software implementations?
Thanks, Interesting read.
JvdW - Wednesday, July 16, 2008 - link
Just to show what hardware can do for stability, my recent horror story.I have 3 Sun X2200 M2 two CPU dual Core fitted with 16Gb and one with 12Gb and setting up a Xen based environment (Virtual Iron 4.3.8)
Things seemed to work smoothly until I noticed that once in a while the node itself, not the VM, seemed to crash. This started an investigation which led to the following conclusion:
All 3 nodes were upgraded to the then latest BIOS/BMC (tools CD version 1.6), mainly because of security fixes and functionality in the eLOM. I can consistently crash a node by stress testing a XP VM using a HDBenchmark together with a CPU benchmark (HD Speed and 7-Zip). Once I degrade a node to a BIOS/BMC version of the 1.3 Tools CD (2.70 3B13 BMC/BIOS) that VM runs perfectly well while under stress.
Another pecularity of the higher BIOS/BMC version than 1.3 is that the crashes occur more or less at the same time, for example when it first crahes around 10:30 then it will crash again around xx:30 whether that is 1 hour later or 10 hours, on average it runs 2-3 hours.
So be careful to state stability statements without reasoning behind it.
Those servers are very stable with VI as long as I stay on the 1.3 Tools release but sadly that looks like an impossiblity in the future.
Any of you techies having an explanation for this, how weird it may seem I'm in for any explaination.
Greetings,
Joop
mino - Tuesday, July 15, 2008 - link
1) I REALLY appreciate your effort to making a reasonable performance benchmark. AFAIK no such 3rd party benchmark with reasonable value is available for free.2) Please be careful talking about Xen "hypervisor (in)stability".
XEN is as a name You should sorely reserve for Citrix Xen Server or the deep-technical talk about hypervisors.
The stability AND performance of an virtualized solution is mostly affected by the I/O. I/O is NOT part of the hypervisor as occurs in so-called dom0/service-partition (or whatever PR guys rename it) that is essentially na I/O router for all the VM's on the machine.
Also, as You are well aware, there are many virtualization solution utilizing the Xen hypervisor.
In brief, in order of maturity:
Virtual Iron
Citrix's Xen Server (former XenSource XenServer)
Novel's; Red Hat's; and other "Xen-based virtualization" solution from Linux distribution vendors.
Please, do not reproduce the same mistake I see all over the web - using the hypervisor name for describing a virtualization solution characteristics.
(I know Citrix does it intentionally in their PR, but we are technologists not Avareage Joe's to be misled)
3) What I would like to see is have benchmarked the real solution that compete for the market i.e. ESX vs. Xen Server vs. Virtual Iron vs. Hyper-V.
However You take it, the solutions from the Linux OS Vendors have a very limited market penetration. Microsoft too has not market, but everybody knows it will get some so it makes sense to include Hyper-V.
Comparison of Red Hat's Xen solution against their KVM solution may be interesting but the real value for IT people out there is to compare the existing solutions one is going to buy for his possible project.
Perkida - Tuesday, July 15, 2008 - link
I appreciate someone looking into such benchmarks, it's about time.When comparing enterprise level solutions such as VMware ESX and Hyper-V it is probably more directly comparable to the enterprise level Xen put out by Citrix, XenServer, than the free version. The embedded product that ships on select HP systems may be of particular interest as it has the ability to display up to 9 VM's in a grid locally using the servers display driver instead of being required to remote in (also still an option). There is something special about being able to view 4, 6, or 9 VM's at once when you're sand-boxing a QA build, etc.
Just a thought as my experience with CTX has always been a positive one.
Take it away Brian Madden.
mino - Tuesday, July 15, 2008 - link
Second that, did not see You post comming ;)computerfarmer - Saturday, July 12, 2008 - link
This is my first read into this area. I also read the three articles posted at the bottom of this one. Information is the key to understanding and these articles help with this. Like any assignment at university, it will lead you to further studies for a better understanding under different conditions.This article and the comments, have given me more insight into this area.
Thank you all
sht - Friday, July 11, 2008 - link
You mention KVM yourself Johan, and I'd be very interested if you had a look on it. Any reasons not to include it in the upcomming article? Any timeline on when we might see benchmarks on virtualization including it?Thanks.
iwodo - Thursday, July 10, 2008 - link
While I understand most current use for Virtualization is in Servers. Would they one day come to gaming / graphics. Being able to use WindowsXP inside Virtual Machine with FULL Graphics power.i.e GPU Virtualization. Possible?
BaronMatrix - Friday, July 11, 2008 - link
That is coming with IOMMU by AMD (the Intel version escapes me). It will probably NOT enable DX10, but it will allow better memory management and throughput for graphics. SO you won't have an emulated S3 graphics card.yyrkoon - Thursday, July 10, 2008 - link
You may want to consider fixing the RSS Feed link to container based OS virtualization, as it has not worked for me since day 1. The link from here however does work.I would also like to add that Xen *IS* free, not cheap, as you can download a binary, or the source from xen.org HVM(hardware virtual machine) have also been possible for over a year, and is supported on all core 2 duo up CPUs, and AMD AM2 on up CPUs. You guys should really check our irc.freenode.org ##xen(if you have not already) and speak to the vetrans who have been running xen para vitual, or hardware virtual for years now.
*shakes head at running MySQL/php on a Win2k3 box*
rippleyalienshakd - Thursday, July 10, 2008 - link
In all fairness, this test is not even close to being realistic.1. How many web servers does an organization run?
2. How many SQL servers does an organization run?
3. XEN? Yah XEN is free, the free version.
I Have to tip my hat to VMware. YAH it costs..
BUTTTTT there are things i can do on a vmware server that XEN, rather the freebe, Citrix XEN, even SUN XEN.. They are close, but not even within the same league.
I actually have VMware VIRTUAL servers that out- preform Physical servers.
NEXT time you do something like this, how about give more details. The test that you preformed, is amateur at best.
there are 3 main reasons to even think about virtualization.
1. Legacy server consilidation
2. Disaster scenerios
3. Virtual Infrastructure
Seeing how 4 web servers preform on 1 physical server doesn't make sense. Even Microsoft put out a benchmark on Exchange 2007 on VMware ESX. 10,000 users, on 2 VMware ESX servers. now that my friend is a benchmark.
NEXT demo/test, use the actual parts. You have failed to show actual cpu/memmory utilization. that is a XEN drawback. CPU outstanding, but Memmory utilization horrible.
The ability to do accurate CPU sharing (Resource sharing) is HORRIBLE. Network configurations and throughput, HORRIBLE
JohanAnandtech - Friday, July 11, 2008 - link
It is a blogpost, not a full article with full benchmark configuration disclosure. That is mentioned more than once in this post.Tests were performed based on the input of real companies. Yes, in some cases, just putting a lot of mailservers together might be more relevant. But as you indicate yourself, microsoft and VMWare produce more than enough exchange benchmarks. So if we can chose, we chose a benchmark scenario that is realistic, but not beaten to dead.
Virtual servers can only outperform physical servers if your application scales badly over more cores. You load balance your load over several virtual machines, and compare this with with one application that has to scale over many CPUs. But otherwise virtual servers can not outperform physical ones.
I am not saying Xen is better than VMWare. I am just saying we found out that Xen is finally ready to run Windows. I have seen a lot of claims that it should work on Xen 3.0.x, but we found that it was impossible to run any kind of Windows on that version. But with Xen 3.2, it seems to be very stable.
aenagy - Monday, July 14, 2008 - link
We run on ESX Server 3.0.2 and I have been resisting attempts to move some of the heavier applications such as SQL Server 2005, Exchange Server 2003 and other LoB applications due to concerns about performance. I am hoping that the combination of upgrading ESX Server 3.5 plus NPT plus IOMMU would provide the necessary throughput for these applications.So please include the latest version of ESX Server in your benchmarks. Also, even though the vendors have thier own whitepapers on SQL Server and Exchange Server I would still like to see some independent benchmarks. It would be interesting to make comparisons with and without NPT, and IOMMU when it comes out.
HotdogIT - Thursday, July 10, 2008 - link
It's INSANELY difficult to find any good reviews/benchmarks/non vendor-speak about virtualization platforms; looking forward to this article! Even though my virtualization process doesn't involve any heavy duty webservers, seeing how it performs under that load will be cool.Hopefully you'll cover a bit on the install/setup process to; I only mention this because from my experiences, both VMWare's ESX, and Xen's products could both use a bit of spit-shine to the end user, techy or not.
Nehemoth - Friday, July 11, 2008 - link
Yes, And install/setup process of Xen and WMARE ESX would be very helpful.Thank you for the good job.