Myce/OakGate 4K Read and Write Latency Tests and Quality of Service
These tests steadily increase the random 4K IO demand in
terms of IOPS, and report the drive’s response in terms of Average IOPS, Average
Latency and Maximum Latency. It is designed to show a drive’s maximum IOPS
capability and report the all important Latency numbers for each level of IOPS
demanded. The Maximum latency numbers give us an insight into the occurrence
of Latency peaks that could cause an unexpected response from time to time.
Firstly, let’s have a look at a standard Pre-Conditioning
step (4K Random Writes) –
Please note that the 4K Random Writes were preceded by a
pre-fill step, which performed 128K sequential writes to twice the drive’s
capacity (to facilitate the achievement of a steady state).
Myce/OakGate 4K Read and Write Latency Tests
You can see that the drive hit a write cliff at around 1,800
seconds, followed by a gradual recovery of performance to around 6,600
seconds. This was followed by a smaller write cliff and a slide into a Steady
State around 38,000 IOPS.
4K Latency Read Test
Myce/OakGate 4K Read Latency Test
You can see that in this test the drive tops out at an
impressive 83,000 IOPS, which exceeds Toshiba’s specification of 75,000 for
Random 4K Reads.
We can see that read latency remains beneath 120
Microseconds until a level of 55,000 IOPS, which is impressive.
You can see a couple of Maximum Latency spikes.
Let’s have a close look at the distribution of the Latency
results at one of these spikes, the 67,000 IOPS level –
As this is the first time, in this review, that we are
looking at a High Resolution Latency Histogram, here’s an explanation – The X
axis to the left is the count of the IOs in the observation period (in a Round)
that had a Latency of the value along the Y axis (please note that the X axis
is logarithmic to allow the low order counts of the huge number of IOs that
have been measured to be visible); the Y axis is the Latency value measured in
Microseconds; The X axis to the right is the % of the Total IOs observed that
have a Latency <= to a given Latency value; the rate of getting to 100% is
highlighted by the red graph line.
We can see that 99.9% of the Latency values are <= 330 Microseconds
and there are very few outliers – the Quality of Service as measured in this
test is excellent.
4K Latency Write Test
Myce/OakGate 4K Write Latency Test
This result appears – up to a level of demand of 17,000 IOPS
the drive responds linearly and meets the demand. After this, the drive starts
to fail in meeting the increase in demand but then bounces back again (e.g.
fails to meet 18,000 but then appears to meet a demand of 20,000).
Average Write Latency bounces up and down.
… and the Maximum Write Latency Values appear to be generally
bad, also bouncing around.
So, what’s going on here and how can a drive appear to fail
to meet one level of demand but then meet a higher level? To see if we can
throw some light on what’s going on let’s have a close look at the ‘4 minute
period, detailed performance over time, graphs’ from which the plots in the
above graphs have been taken.
Here are the performance over time graphs (measures in IOPS)
for the plots in the range of 20,000 IOPS demand through 25,000 –
You can see that the drive mostly meets the demand but falls
off towards the end of the 4 minute period.
You can see that a demand of 21,000 IOPS is not being met
You can see that a demand of 22,000 is then not being met
and that the variations in performance grow towards the end of the 4 minute
Similarly, a demand of 23,000 is not being met.
At the 24,000 level of demand we can see that the demand
continues to not be met until around 182 seconds when the drive suddenly has a
spurt in performance up to 34,000 IOPS (which is above the level of the IO cap
of 24,000 that we have applied during this 4 minutes period) before it begins
to fall away.
You can see that a demand of 25,000 IOPS is now largely
In attempt to clarify what is going on I decided to rerun
the test, but with the plots being taken over far longer periods of time. Here
are the results of the test when it is rerun with 1 hour observation periods,
where the demand in IOPS is increasing in increments of 5,000 –
You can see that we now have a much more linear result.
Let’s have a look at each of the detailed performance over
time graphs from which the values in the previous graph have been taken.
You can see that at 5,000 IOPS the demand is being met.
At a demand of 10,000 IOPS you can see the demand is very
largely being met, but that there are a few small dips.
At 15,000 you can see the demand is largely being met, but
that there are a few, more noticeable, dips.
You can see that a demand of 20,000 IOPS is not being met
after 210 seconds.
You can see that in the 25,000 IOPS period demand is at
first not being met but that there is an increase at around 250 seconds
followed by a spurt of performance at 400 seconds, after which a demand of
25,000 is largely met.
You can see that the 30,000 IOPS period of demand shows a
similar pattern of behaviour as for the 25,000 period.
Again, you can see a similar pattern of behaviour.
In the 40,000 IOPS period of demand we see a few spurts of
In the 45,000 IOPS period of demand we see a spurt of
performance but generally the demand is not being met.
So what is
causing the performance behaviour? I can’t be certain without having an
intimate understanding of the drive’s firmware but here’s what I think may be
I think the drive’s firmware may have two ways in which it
can clean blocks to accommodate new writes. Firstly a ‘foreground’ method
which, if needed, cleans blocks on the fly immediately before a new write is
processed. Secondly, a ‘background’ garbage collection process which
periodically makes a tranche of clean blocks available to accommodate new
writes. When a tranche of clean blocks is made available the drive is able to
write at a significantly faster speed than is possible when blocks must first
be cleaned on the fly. I believe the apparent spurts in performance (above the
level of the IOPS cap that is in force) occur when there is a backlog of Write IO
requests in the IO stack that can then suddenly be cleared following the
release of a tranche of fresh blocks.
I imagine that one of the objectives for a firmware
developer is to balance how much time is given to a background process so that
it does not impact unduly upon foreground performance.
Whatever, in the context of this test, the firmware’s behaviour
does impact upon the drive’s performance behaviour and the Quality of Service
for 4K Random Writes, as presented in ‘Quality of Service’ below. Whilst, I
can state that the conditions under which our test is performed are the same
for every drive we review, I cannot argue is that the test reflects any real
world scenario. So, for example when the drive is used in a mixed work load
environment (especially where the majority of IOs are reads), or in an
environment where it’s peak of Random Write performance is not approached, then
the apparent adverse impact upon the Random Write Quality of Service may well never
To illustrate here is the result of a 4K Random Write
preconditioning step run with an IOPS cap of 25,000 in force from the outset (which
followed a standard sequential pre-fill step) –
You can see that the demand of
25,000 IOPS is consistently met with only an occasional dip.
… and here’s the Latency
distribution for this 25,000 IOPS level preconditioning step –
You can see that the 99.9% percentile Latency Value is an
outstanding 30 Microseconds which is significantly better than the result
extracted from our extended standard test, as presented below.
Quality of Service
Let’s have a closer look at the Quality of Service delivered
by the Toshiba THNSN81Q60CSE.
I have extracted the 95th, 99th, 99.9th
and 99.999th percentile Latency values for the range of 5,000
through 85,000 IOPS, in increments of 5,000 IOPs, and charted them below. The
Latency Value (percentile rank) for the 95th percentile is the Latency
that 95% of all IOs fall within (and similarly for the 99th, 99.9th
and 99.999th percentiles). Here are the results for Random 4K Reads
The Quality of Service at the given IOPS levels can then be
compared to that achieved by other drives. For example, here are the values
for the outstanding Samsung SM863 480GB –
You can see that the results for the Toshiba drive go up to 85,000
IOPS (as this is the highest 5,000 increment that its 4K Random Read IOPS
performance accommodates). You can also see that the Samsung SM863 is
consistently better than the Toshiba THNSN81Q60CSE and in particular there is a
huge gap in the 99.999th percentile. Having said this, it is fair
to say that the Toshiba THNSN81Q60CSE has excellent results in the 99.9th,
99th, and 95th percentiles.
Now, let’s look at the results for Random 4K Writes (where
the values for the Toshiba THNSN81Q60CSE have been taken from the second
extended test as mentioned above) –
Here you can see that the results for the Toshiba
THNSN81Q60CSE are not great.
Now, let’s compare them to the results for the Samsung SM863
Here you can see that the results for the Samsung SM863 are
I have the feeling that in the context of this test the
Toshiba’s Random Write QoS may be adversely affected by the firmware giving too
much time to its background garbage collection.
Now let’s head to the next page, to look at the results
for the Myce/Oakgate Reads and Writes Tests…..