AnandTech 2015 Client SSD Suite

The core of our SSD test suite has remained unchanged for nearly four years now. While we have added new benchmarks, such as performance consistency and Storage Bench 2013, in response to the evolution of the SSD industry, we haven't done a major overhaul to take our testing to the next level. That all changes today with the introduction of our 2015 Client SSD Suite.

Just to be clear, there weren't any flaws in the way we did testing in the past -- there were simply some shortcoming that I've been wanting to fix for a while now, but like any big upgrade it's not done overnight. There are four key areas where I focused in the 2015 Suite and these are modernizing our testbed, depth of information, readability and power consumption.

Our old testbed was old, really old. We were using a Sandy Bridge based system with Intel Rapid Storage Technology 10.2 drivers from 2011, so it doesn't take a genius to figure out that our system was desperately in need of a refresh. The 2015 testbed is the latest of the latest with an Intel Haswell CPU and ASUS Z97 motherboard. For the operating system, we have upgraded from Windows 7 to Windows 8.1 with native NVMe driver, which ensures that our setup is fully prepared for the wave of PCIe NVMe SSDs arriving in the second half of 2015. We are also using the latest Intel Rapid Storage Technology drivers now, which should provide a healthy boost over the old ones we were using before. I've included the full specs of the new system below.

AnandTech 2015 SSD Test System
CPU Intel Core i7-4770K running at 3.5GHz (Turbo & EIST enabled, C-states disabled)
Motherboard ASUS Z97 Deluxe (BIOS 2205)
Chipset Intel Z97
Chipset Drivers Intel 10.0.24+ Intel RST 13.2.4.1000
Memory Corsair Vengeance DDR3-1866 2x8GB (9-10-9-27 2T)
Graphics Intel HD Graphics 4600
Graphics Drivers 15.33.8.64.3345
Desktop Resolution 1920 x 1080
OS Windows 8.1 x64

The second improvement we have made is regarding the depth of information. Every now and then I found myself in a situation where I couldn't explain why one drive was faster than the other in our Storage Bench tests, so the 2015 Suite includes additional Iometer tests at various queue depths to help us understand the drive and its performance better. I'm also reporting more data from the Storage Bench traces to better characterize the drive and providing new metrics that I think are more relevant to client usage than some of the metrics we have used in the past. The goal of the 2015 Suite is to leave no stone unturned when it comes to explaining performance and I'm confident that the new Suite does an excellent job at that.

However, the increase in depth of information creates a readability problem. I know some of you prefer to have easy and quick to read graphs, but it's hard to present a mountain of data in a format that's convenient to read. To give you the best of both worlds, I'm providing both the easy and quick to read graphs as well as the full data for those who want to dig in a bit deeper. That way the benchmarks will remain comfortable to skim through in case you don't have a lot of time on your hands, but alternatively you will get access to far more data than in the past.

Last but not least, I'm taking power testing to a whole new level in our 2015 Suite. In the past, power consumption was merely a few graphs near to the end of the article and to be honest the tests we ran didn't give the full scope of the drive's power behavior. In our 2015 Suite, power is just as important as performance is because I'm practically testing and reporting power consumption in every benchmark (though for now this is limited to SATA drives). In the end, the majority of SSDs are employed in laptops and power consumption can actually be far more critical than performance, so making power consumption testing a first class citizen makes perfect sense.

A Word About Storage Benches and Real World Tests

While I'm introducing numerous new benchmarks and performance metrics, our Storage Bench traces have remained unchanged. The truth is that workloads rarely undergo a dramatic change, so I had no reason to create a bunch of new traces that would ultimately be more or less the same that we have already used for years. That's why I also dropped the year nomenclature from the Storage Benches because a trace from 2011 is still perfectly relevant today and keeping the year might have given some readers a picture that our testing is outdated. Basically, the three traces are now called The Destroyer, Heavy and Light with the first one being our old 2013 Storage Bench and the two latter ones being part of our 2011 Storage Bench. 

I know some of you have criticized our benchmarks due to the lack of real world application tests, but the unfortunate truth is that it's close to impossible to build a reliable test suite that can be executed in real time. Especially if you want to test something else than just boot and application launch times, there is simply too many tasks in the background that cannot be properly controlled to guarantee valid results. I think it has become common knowledge that any modern SSD is good enough for an average user and that the differences in basic web-centric workloads are negligible, so measuring the time it takes to launch Chrome isn't an exciting test to be honest.

In late 2013, I spent a tremendous amount of time trying to build a real world test suite with a heavier workload, but I kept hitting the same obstacle over and over again: multitasking. One of the most basic principles of benchmarking is reproducibility, meaning that the same test can be run over and over again without significant unexplainable fluctuation in the results. The issue I faced with multitasking was that once I started adding background operations, such as VMs, large downloads and backups like a heavier user would have in the background, my results were no longer explainable as I had lost the control of what was accessing the drive. The swings were significant enough that the results wouldn't hold any ground, which is why you never saw any fruit of my endeavors. 

As a result, I decided to drop off real world testing (at least for now) and go back to traces, which we have been using for years and know that they are reliable, although not a perfect way to measure performance. Unfortunately there is still no TRIM support in the playback and to speed up the trace playback we've cut the idle times to a maximum of 25 milliseconds. Despite the limitations, I do believe that traces are the best to measure meaningful real world performance because the IO trace is still straight from a real world workload, which cannot be properly replicated with any synthetic benchmark tool (like Iometer). 

Introduction & The Drive Performance Consistency
Comments Locked

128 Comments

View All Comments

  • iLovefloss - Tuesday, February 24, 2015 - link

    Samsung's first two TLC drives, the 840 and 840 EVO, has some firmware issues that cause month old data to be read slowly. The severity ranges from slower than a speedy HDD to as slow as a SATA2 SSD. Samsung's first patch didn't resolve the issue for all the 840 EVO SSDs suffering from the slowdowns or only temporarily resolved, so Samsung is in the process of making another patch.
  • kgh00007 - Wednesday, February 25, 2015 - link

    I have an 840 EVO and I applied the firmware fix in October last year and the reads have dropped again to below 50MB/s on older data, ie. my OS files and stuff that was installed when I first set the drive up.

    I will be waiting to see how Samsung handle this before I buy another SSD from them. Benchmarks and reviews mean nothing if an SSD drops below HDD read speeds after a few months of real world use.

    Cold boot now takes minutes, not seconds!!
  • 3DoubleD - Wednesday, February 25, 2015 - link

    Exactly. I have one drive that has sequential read minimums as low as 8.8MB/s and large portions averaging 50MB/s. Another drive is fine and operates at 300MB/s consistently (although I'm pretty sure that should be higher on SATA3, but day-to-day that is fast enough not to notice). They need to squash this bug if they plan on selling TLC drives in the future in any real volume. Enthusiasts will care, which is admittedly a small market, but I think some laptop vendors might begin to take notice and avoid Samsung TLC products as well, and that's a larger market.
  • Irish_adam - Tuesday, February 24, 2015 - link

    So when are they going to make a desktop version with a heatsink on it? It seems like everyone is so obsessed with portables these days that the desktop crowed is getting ignored but surely this kind of performance would mainly be used for a desktop machine than an ultra thin laptop. Its my main gripe with PCIe SSDs atm
  • dananski - Tuesday, February 24, 2015 - link

    Same occurred to me. Could probably get a substantial boost in long-running operations by attaching a heatsink. Should be easy enough to do yourself - thermal tape and some old vram heatsinks would probably do the trick without being so heavy as to break the pcie slot.

    I would like to see the rate of heat dissipation after heavy use (i.e. how that temperature graph looks after you stop writing to the disk). It starts throttling after roughly 180GB sequential, which is plenty for most scenarios, but how long does it take to cool back down again for your next big write? Does throttling occur under more mixed, sustained loads like a database server? Not exactly my kind of use cases, but I'd be interested to see.
  • DanNeely - Tuesday, February 24, 2015 - link

    "However, it's nowhere near the maximum bandwidth of the PCIe 3.0 x4 bus, though, which should be about 3.2GB/s (PCIe only has ~80% efficiency with overhead after the 128b/132b scheme used by PCIe 3.0)."

    Where's the 20% loss coming from? 128/132 bit encoding only has a 3% overhead, is this an incompletely updated copy/paste from a description of PCIe 2.0? The 8/10bit encoding used in the older version did have a 20% penalty.
  • Kristian Vättö - Tuesday, February 24, 2015 - link

    That's the overhead on top of the encoding scheme and is a rough figure based on our own testing with GPU memory bandwidth that will saturate the interface.

    It's the same in PCIe 2.0 too: the interface is good for 5GT/s per lane, which equals 500MB/s per lane once you take the 8b/10b encoding and bits to bytes translation into account. However, in real world the best bandwidths I've seen have been about 390MB/s per lane.
  • extide - Tuesday, February 24, 2015 - link

    Protocol overhead (NOT the 120/132b part) -- the commands and stuff, interrupt latency from the cpu and other devices, DMA latencies on read/write to main system memory, etc.
  • Hulk - Tuesday, February 24, 2015 - link

    Would it be possible to display the entire AS SSD results window?
  • Kristian Vättö - Tuesday, February 24, 2015 - link

    I only run the sequential test, but I can certainly switch to running the full test and publishing the results as a screenshot if that's preferred.

Log in

Don't have an account? Sign up now