IDC Report: AFA Performance Testing Framework

idc-tableThe storage performance requirements of 3rd Platform computing workloads demand more than what traditional hard disk drive (HDD)-based architectures can cost effectively deliver. As a result, flashbased storage is rapidly penetrating the datacenter. Flash operates very differently in terms of read/write performance than traditional HDDs, and this has major implications for how testing should be performed on flash-based arrays.

There are several types of flash-based arrays where these considerations apply: all-flash arrays (AFAs), all-flash configurations of hybrid flash arrays (HFAs), and some more flash-optimized HFAs. Although early HFAs exhibited storage performance characteristics more similar to that of HDD-based arrays, these offerings are becoming increasingly flash optimized, narrowing the performance differences between them and AFAs. End users that are testing flashbased arrays for use in their production datacenters need to understand how flash-based arrays and flash-based testing is different.

Findings include:

Preconditioning. Flash-based arrays must be transitioned from a fresh-out-of-the-box (FOB) state to a steady state before test results can start to accurately reflect the performance the array will deliver in production. Preconditioning to transition flash-based arrays to steady state operation is a requirement in all testing.

Data stream/data set definition. The workloads that are most often run on AFAs differ from classic workloads in significant ways. In the mixed virtual workloads where AFAs are most often deployed, both data streams and data sets exhibit a variety of read/write ratios, use a wide distribution of block sizes, are heavily skewed toward random I/O, have a high percentage of data that is reducible, and are characterized by I/O bands that drift over time. To understand how an AFA will perform on workloads that exhibit these characteristics, it must be tested with workloads that mimic them.

Read/write asymmetries. While flash-based arrays deliver average latencies and throughput that are an order of magnitude faster than HDD, flash does not exhibit the same type of performance consistency. In fact, there can be wide variability between read/write latencies in flash-based arrays, depending on what is going on in the array at the time. Test administrators need to understand these characteristics, the behavior they produce at a system level, and use a test plan designed for flash, not HDD.

Scale of workload generation. The largest HDD-based arrays are generally delivered on the order of tens of thousands of IOPS, but flash-based arrays can deliver hundreds of thousands to millions of IOPS. To exercise flash-based arrays up to and beyond their performance saturation points, testing equipment must be able to generation sufficient workload.

Read the full IDC report here…