Samsung SM951-NVMe (256GB) PCIe SSD Review
by Kristian Vättö on June 25, 2015 9:40 AM ESTPerformance Consistency
We've been looking at performance consistency since the Intel SSD DC S3700 review in late 2012 and it has become one of the cornerstones of our SSD reviews. Back in the days many SSD vendors were only focusing on high peak performance, which unfortunately came at the cost of sustained performance. In other words, the drives would push high IOPS in certain synthetic scenarios to provide nice marketing numbers, but as soon as you pushed the drive for more than a few minutes you could easily run into hiccups caused by poor performance consistency.
Once we started exploring IO consistency, nearly all SSD manufacturers made a move to improve consistency and for the 2015 suite, I haven't made any significant changes to the methodology we use to test IO consistency. The biggest change is the move from VDBench to Iometer 1.1.0 as the benchmarking software and I've also extended the test from 2000 seconds to a full hour to ensure that all drives hit steady-state during the test.
For better readability, I now provide bar graphs with the first one being an average IOPS of the last 400 seconds and the second graph displaying the standard deviation during the same period. Average IOPS provides a quick look into overall performance, but it can easily hide bad consistency, so looking at standard deviation is necessary for a complete look into consistency.
I'm still providing the same scatter graphs too, of course. However, I decided to dump the logarithmic graphs and go linear-only since logarithmic graphs aren't as accurate and can be hard to interpret for those who aren't familiar with them. I provide two graphs: one that includes the whole duration of the test and another that focuses on the last 400 seconds of the test to get a better scope into steady-state performance.
Steady-state performance isn't mind blowing, although it's a little unfair to compare the 256GB SM951-NVMe against a 512GB SM951-AHCI because available NAND throughput can play a major role in steady-state performance with a properly designed controller and firmware.
The mediocre steady-state performance is replaced by outstanding consistency, though. During the last 400 seconds, the SM951-NVMe has standard deviation of only 1.12, whereas the next drives are in the order of hundreds with many drives surpassing 1,000.
 |
|||||||||
| Default | |||||||||
As there's practically no variation in performance, the graph is just a straight line after the initial cliff. There are zero up and down swings, which is something I've yet to see even in an enterprise SSD. Samsung has always done well in IO consistency, but in all honesty the SM951-NVMe sets a new bar for consistency. On the downside, I would rather take some variation with higher performance because especially for client workloads such level of consistency isn't really needed, but increased performance can always help with intensive IO workloads. Nevertheless, I'm very happy with the direction Samsung is taking because we still see some controller vendors not paying enough attention to consistency, but it's clearly a high priority for Samsung.
Unfortunately I couldn't run any tests with added over-provisioning because the hdparm commands I use for limiting the capacity do not work with NVMe drives. There are similar commands for NVMe drives too, but for now there is no publicly available utility for issuing those commands.
 |
|||||||||
| Default | |||||||||
Facebook
Twitter
RSS
74 Comments
View All Comments
kspirit - Saturday, June 27, 2015 - link
Thank you, I appreciate the response. I'll check it out :)fastfood8891 - Thursday, June 25, 2015 - link
@Kristian VättöFor that one graph on page 3, why do you divide IOPS by the standard deviation? I understand you are showing that this drive is a lot more consistent than the others, I just wonder what the mathematical intuition behind this value is.
Kristian Vättö - Thursday, June 25, 2015 - link
It's just a key ratio. Standard deviation alone is a bad metric because it doesn't take the performance into account at all. It wouldn't be fair to compare stdev of a drive that does 15K IOPS against a drive that does 5K IOPS because if the two had equivalent stdev the impact would be much more severe on the slower one (e.g. frequent 50% drops in performance, whereas only 15% for the 15K IOPS drive).krumme - Friday, June 26, 2015 - link
Enjoy this man guys while we have him. This is competence at its finest.JellyRoll - Friday, July 10, 2015 - link
Well, if competency consists of him originally presenting test results wrong in several articles, and then a reader correcting him in the comments of the 750 article, and then him adopting the correct method...well, yeah. Then that is competency. (that is what happened)fastfood8891 - Saturday, June 27, 2015 - link
makes sense, thank you for the explanation.Samus - Thursday, June 25, 2015 - link
First gen. Give it time. Seems like a dedicated nvme controller (ala Intel) would improve random performance...it looks like this controller is already running near its capability in AHCI mode.But as with everything nand, mature firmware will help
willis936 - Thursday, June 25, 2015 - link
Exciting stuff. Does anyone know if current broadwell laptops can be retrofitted with nvme M.2 drives in the future? Specifically the xps 13 2015. It's a hot item with poor storage options.extide - Thursday, June 25, 2015 - link
That laptop, probably yes. Best place to check would be the notebookreview forums -- they should have a specific forum for that laptop and you can ask there, or see if someone has already swapped in a PCIe drive. -- You need to verify 1) That the m.2 slot is actually pcie and not just sata, and 2) that it supports nvme in the bios, but being broadwell, you can probably assume it does.lilmoe - Thursday, June 25, 2015 - link
Is there any tangible difference in CPU utilization using NVMe? Would be nice if there was a graph...