There are lies, damn lies, and then there are storage vendor performance metrics. In the race to see who has the most IOPS, storage vendors often focus on a very small – and often unrealistic – range of performance values. Many storage benchmarks are performed using a 4K block size when, in reality, typical workloads use larger block sizes. The downside is that larger block sizes often carry with them much less impressive IOPS values, meaning the storage vendors wouldn’t be able to say things like, “Our storage delivers a million IOPS!” and would instead have to say, “Our storage delivers hundreds of thousands of IOPS.”
Pure Storage has done a good job of cutting through the 4K block size hype by providing some. So, do customers really need “millions of IOPS”? The answer probably comes as no surprise. It’s no.
By the way, according to Pure, only 3% of its customers are using a block size that falls between 1K and 4K. Their average customer block size is actually 38K… a far cry from 4K.
Millions of IOPS – heck, even hundreds of thousands of IOPS – sounds like a pretty impressive number, but given the disparity in benchmarks, it can be somewhat difficult to get a true apples-to-apples comparison between systems, particularly since there are a broad range of potential outcomes as the block size changes.
What are some more practical ways that you can measure differences between storage arrays from different vendors?
The first is latency, which is basically the amount of time it takes for a storage array to complete an operation. The higher the latency, the longer it takes for storage operations to be completed. That metric is generally consistent across different workloads and block sizes. At the end of the day, all other metrics eventually come down to latency. If you don’t have sufficient IOPS, latency will suffer. If you don’t have sufficient throughput on the storage fabric, latency will suffer. Latency is the end result of everything else put into the system.
Cost metrics also matter. Initial acquisition cost is, unfortunately, used by many customers as a proxy for understanding the full expense of a storage array. There are so many other items to consider. Rather than focusing on initial capital cost, customers should adopt a Total Cost of Ownership (TCO) economic model for determining how much a solution costs to run over time. Such TCO calculations should include:
- Support costs. Most systems carry annual service and support costs.
- Does the solution carry additional licensing cost to gain access to advanced capabilities or are these capabilities included in the base system?
- Rack space. Rack space isn’t free. If you can reduce the storage footprint, you can reduce the number of racks needed.
- Power/cooling. Especially when comparing against spinning disk, flash-based storage systems can dramatically reduce power and cooling costs, primarily due to the fact that flash storage devices have no moving parts.
There are also some other kinds of “metrics” to consider, such as flexibility when it comes to storage protocol choice. While this would generally be considered a feature and not a metric, it can be important to many people. In fact, according to a TechValidate survey , more than 50% of customers that purchased Tegile storage systems are using multiple storage protocols. Every protocol has strengths and weaknesses, so having the ability to, well, not have to choose can be compelling.
The product’s feature sets also make a big difference. Systems that don’t support high availability or deduplication have very different economic considerations that systems that include these capabilities. Other critical features include compression, thin provisioning, snapshots and replication. The latter two features can help organization improve on existing disaster recovery capabilities.
To compare solutions, TCO should be used in concert with performance outcomes. A solution may carry a slightly higher initial price tag, but the TCO may be far lower and the product may include features that amplify economic benefits. Don’t rely on IOPS comparisons as the end-all-be-all way to determine which product is better. Consider how additional features may positively impact the economic equation.
Finally, don’t overbuy! Most organization’s simply don’t need millions of IOPS! For some, it would be like buying a Ferrari for the three block commute down the street. A bike would do just fine.