to view or download a detailed introduction to Myce’s Enterprise Class Solid
State Storage (‘SSS’) Testing Methodology as a PDF.
All testing is performed on an OakGate Technology test unit
We perform two sets of Performance Tests:
A full set of the Storage Network Industry Association’s (‘SNIA’) tests
with mandatory parameters, as specified in their Solid State Storage
Performance Test Specification Enterprise V1.0.
A set of tests, known as the ‘Myce/OakGate Full Characterisation Test
Set’, that provides readers with a fuller characterisation of the solution.
Comprehensive power consumption testing is performed using
Quarch hardware as documented here.
We also review other important factors such as Data
Reliability and Failover features.
A word about SNIA testing – before striking a partnership
with OakGate Technology I spent some time researching how I might implement
SNIA testing using freely available tools such as IOMeter and FIO. I arrived
at the conclusion that whilst it was theoretically possible it was
impractical. The reason for this is as without the automation offered by a
test bench, such as the OakGate Unit, the only way to meet the SSS PTS
requirements is to run the maximum number of test cycles and then to manually
look back at the results to determine when/if steady state has been achieved in
the workload specific test cycle, and then harvest the data from the qualifying
Measurement Window. This means that the test runs would always take a maximum
elapsed time, and there would be a great deal of human effort required to
review, gather, and report upon the data. I empathise with, acknowledge, and
respect the efforts of other reviewers who endeavour to meet the SNIA’s
principles in their testing – I am privileged and thankful to be able to use a
superb test bench which automates the whole process and allows me to meet the
SNIA’s specification in full.
Before we move on, let’s remind ourselves of some basics –
When reviewing the performance of an SSS solution there are
three basic metrics that we look at:
IOPS – the number of Input/Output Operations per Second
Bandwidth – the number of bytes transferred per second (usually measured
in Megabytes per second, ‘MB/s’)
Latency – the amount of time each IO request will take to complete
(usually, in the context of SSS solutions, measured in Microseconds, which are
millionths of a second).
It is true to say that IOPS and Bandwidth had all been
growing rapidly before the advent of SSS solutions, but Latency can only be significantly
decreased by eliminating mechanical devices, and thus Latency is the single
most important aspect that SSS solutions deliver to enhance performance.
Latency in a technical environment is synonymous with delay.
In the context of an SSS solution it is the amount of time between an IO
request being made, and when the request is serviced.
Bandwidth, also commonly referred to as ‘Throughput’, is the
amount of data that can be transferred from a storage device to a host, in a
given amount of time. In the context of SSS solutions it is typically measured
in Megabytes per second (MB/s).
A great enterprise SSS solution offers an effective balance
of all three metrics. High IOPS and Bandwidth is simply not enough if Latency
(the delay in an IO operation) is too high. As we will see in the test results
presented below, as Latency increases IOPS will inevitably decrease.
Queue Depth is the average amount of IO requests outstanding.
If you are running an application and the Average Queue Depth is one or higher
and CPU utilisation is low, then the application’s performance is most probably
suffering from a ‘Storage Bottleneck’.
Another important factor to remember is that SSS performance
is influenced by previous workloads, not just the current workload, and
especially by what has previously been written to the drive. As specified in
the SNIA SSS PTS the goal of all good Enterprise level testing is to provide
consistent circumstances, so that results can be compared fairly across
different SSS solutions – it is for this reason that all of our tests start
with a purge of the drive, so that it starts in a ‘Fresh Out of the Box’ (FOB)
state. Most tests then have a pre-conditioning phase where the drive is put
into a ‘Steady State’ before the test phase begins. Put briefly, a ‘Steady
State’ is achieved when the performance of the drive no longer varies over time
and settles into a consistent level of performance for the workload in hand. You
can find a detailed explanation of ‘Steady State’ and how it is determined in
the SNIA tests in our Enterprise Testing Methodology paper, which can be viewed
or downloaded as a PDF by clicking here.
For interest, here are some
generally accepted assumptions that differentiate the use and therefore the
approach to testing Enterprise/Server and Consumer/Client SSS solutions:
The drive is always full
The drive is being accessed 100% of the time (i.e. the drive gets no
Failure is catastrophic for many users
The Enterprise market chooses SSS solutions based on their performance
in steady state, and that steady state, full, and worst case are not the same
The drive typically has less than 50% of its user space occupied
The drive is accessed around 8 hours per day, 5 days per week, and typically
data is written far less frequently
Failure is catastrophic for a single user
The consumer/client market generally chooses SSS solutions based on
their performance in the FOB state
Esther Spanjer, Director, Enterprise
Business Development EMEA at Sandisk, said, ‘I am happy to commend Myce for
their high level of professionalism and cooperation during the review process’,
Ms. Spanjer added, ‘I wish them every success in their partnership with OakGate
Technology and their initiative to provide authoritative performance reviews
for the Enterprise Solid State Storage market’
Now let’s head to the next page, to look at the results
of our SNIA IOPS (Input/Output Operations per Second) Test…..