to view or download a detailed introduction to Myce’s Enterprise Class Solid
State Storage Testing Methodology as a PDF.
All testing is performed on a world-class, OakGate
Technology test unit, please click here
for a Review of Myce’s OakGate test platform.
We perform two sets of Performance Tests:
A full set of the Storage Network Industry Association’s (‘SNIA’) tests
with mandatory parameters, as specified in their Solid State Storage
Performance Test Specification Enterprise V1.0.
A set of tests, known as the ‘Myce/OakGate Full Characterisation Test
Set’, that provides readers with a fuller characterisation of the solution.
We also test Power Efficiency and review other important
factors such as Data Reliability and Failover features.
Before we move on, let’s remind ourselves of some basics –
When reviewing the performance of a Solid Sate Storage
solution there are three basic metrics that we look at:
IOPS – the number of Input/Output Operations per Second
Bandwidth – the number of bytes transferred per second (usually measured
in Megabytes per second, ‘MB/s’)
Latency – the amount of time each IO request will take to complete
(usually, in the context of Solid State Storage solutions, measured in
Microseconds, which are millionths of a second).
It is true to say that IOPS and Bandwidth had all been
growing rapidly before the advent of Solid State Storage, but Latency can only
be significantly decreased by eliminating mechanical devices, and thus Latency
is the single most important aspect that Solid State solutions deliver to
Latency in a technical environment is synonymous with delay.
In the context of a Solid State solution it is the amount of time between an IO
request being made, and when the request is serviced.
Bandwidth, also commonly referred to as ‘Throughput’, is the
amount of data that can be transferred from a storage device to a host, in a
given amount of time. In the context of Solid State solutions it is typically
measured in Megabytes per second (MB/s).
A great enterprise Solid State solution offers an effective
balance of all three metrics. High IOPS and Bandwidth is simply not enough if
Latency (the delay in an IO operation) is too high. As we will see in the test
results presented below, as Latency increases IOPS will inevitably decrease.
Queue Depth is the average amount of IO requests
outstanding. If you are running an application and the Average Queue Depth is
one or higher and CPU utilisation is low, then the application’s performance is
most probably suffering from a ‘Storage Bottleneck’.
Another important factor to remember is that Solid State Storage
performance is influenced by previous workloads, not just the current workload,
and especially by what has previously been written to the drive. As specified
in the SNIA Solid State Storage PTS the goal of all good Enterprise level
testing is to provide consistent circumstances, so that results can be compared
fairly across different Solid State solutions – it is for this reason that all
of our tests start with a purge of the drive, so that it starts in a ‘Fresh Out
of the Box’ (FOB) state. Most tests then have a pre-conditioning phase where
the drive is put into a ‘Steady State’ before the test phase begins. Put
briefly, a ‘Steady State’ is achieved when the performance of the drive no
longer varies over time and settles into a consistent level of performance for
the workload in hand. You can find a detailed explanation of ‘Steady State’ and
how it is determined in the SNIA tests in our Enterprise Testing Methodology
paper, which can be viewed or downloaded as a PDF by clicking here.
For interest, here are some
generally accepted assumptions that differentiate the use and therefore the
approach to testing Enterprise/Server and Consumer/Client Solid State
Enterprise/Server Solid State
The drive is always full
The drive is being accessed 100% of the time (i.e. the drive gets no
Failure is catastrophic for many users
The Enterprise market chooses solutions based on their performance in
steady state, and that steady state, full, and worst case are not the same
Consumer/Client Solid State Storage
The drive typically has less than 50% of its user space occupied
The drive is accessed around 8 hours per day, 5 days per week, and
typically data is written far less frequently
Failure is catastrophic for a single user
The consumer/client market generally chooses solutions based on their
performance in the FOB state
Now let’s head to the next page, to look at the results
of our SNIA IOPS (Input/Output Operations per Second) Test…..