[H]ard Forum Storage Showoff Thread

[Deadjasper]

Speaking of solid state data retention, the very first flash drive I ever bought was a 512mb Simpletech. It's been sitting in a drawer for god knows how many years. I recently came across it and plugged it in. Yep, data was still there. I opened a few files and everything appeared normal. Given it's a flash drive I would have thought Everything would be gone, I know it's been at least 10 years since it was last plugged into a computer. Don't know if today's flash drives would fair as well but I do believe the tech exists, and has always existed, to make long term solid state storage feasible.

 

[robbiekhan]

If it was a real world issue then can you imagine all those USB flash drives out there that would be empty on a daily basis because most people don't use them all the time?! He'll even some retail drives that come with bundled software on the storage sit on a shelf or warehouse for months or years before being sold and still have the bundled apps still on there.

 

[TeeJayHoward]

Speaking of solid state data retention, the very first flash drive I ever bought was a 512mb Simpletech. It's been sitting in a drawer for god knows how many years. I recently came across it and plugged it in. Yep, data was still there. I opened a few files and everything appeared normal. Given it's a flash drive I would have thought Everything would be gone, I know it's been at least 10 years since it was last plugged into a computer. Don't know if today's flash drives would fair as well but I do believe the tech exists, and has always existed, to make long term solid state storage feasible.

I've got a 4GB and a 2GB drive from college like 15 years ago. I just plugged them in after reading this.

Sure enough, I can access my homework. Groovy.

 

[Zarathustra[H]]

Speaking of solid state data retention, the very first flash drive I ever bought was a 512mb Simpletech. It's been sitting in a drawer for god knows how many years. I recently came across it and plugged it in. Yep, data was still there. I opened a few files and everything appeared normal. Given it's a flash drive I would have thought Everything would be gone, I know it's been at least 10 years since it was last plugged into a computer. Don't know if today's flash drives would fair as well but I do believe the tech exists, and has always existed, to make long term solid state storage feasible.

Huh. Interesting. I would have expected it to be gone based on everything I have read as well, or at least hopelessly corrupted.

Maybe it was some sort of enterprise model with an unusually capable battery built into it or something?

Or - more likely - it was an SLC drive. Since each cell only has one state (on or off) the effects of cell discharge are much less likely to flip a bit than they are in MLC, TLC or QLC drives. But still, 10 years is quite a long time...

Or maybe a little bit of both...

 

[robbiekhan]

Speaking of SSD long term reliability, Puget have now released their rundown of what they do each year, Samsung SSDs come out top for reliability, basically 0% failure rate.

https://www.tomshardware.com/news/hardware-reliability-puget-systems-2021

 

[Zarathustra[H]]

Speaking of SSD long term reliability, Puget have now released their rundown of what they do each year, Samsung SSDs come out top for reliability, basically 0% failure rate.

https://www.tomshardware.com/news/hardware-reliability-puget-systems-2021

Interesting. Their Intel CPU failure rates are off the charts.

I've always considered CPU's to be close to bulletproof unless you do something really stupid.


The Tech Reports SSD Endurance Test is pretty old at this point, but that is what finally convinced me that we no longer needed to worry about SSD endurance, and that was 9 years ago now.

They torture tested SSD's from different manufacturers with constant massive writes and monitored how long it took for them to fail.

https://techreport.com/review/24841/introducing-the-ssd-endurance-experiment/

The winner was the 256GB Samsung 840 Pro with 2.4PB, that is 2400TB.

Also consider this was with Planar NAND, and Samsung introduced 3D Nand in the 850 series after this model, which DRASTICALLY increased SSD life by as much as 10x, so it is not unreasonable to expect that if they tested an 256GB 850 pro instead it would have hit 25PB of writes. (they speced the 850 Pro for 150TB of lifetime writes, but that was clearly extremely conservative)

Also consider that write endurance goes up linearly with drive size, as there simply are more cells to write to. I'm thinking the smallest drive most of us are putting in our systems these days is 1TB., so multiply those numbers above by 4. A 1TB Samsung 850 Pro should theoretically be good for 100PB of writes.

This is what has allowed manufacturers to move from MLC to TLC and QLC which sacrifice write endurance and write performance.

I haven't seen any torture tests on newer TLC or QLC drives yet, but my expectation is that anything modern SLC (if you can find it), MLC or TLC is probably fine. You'll likely never use up the write cycles even in extreme use cases. QLC is probably fine for typical client workloads, but I wouldn't want to use it for any heavy write or caching type workloads.

I recently overhauled the SSD drives in my server pulling all the old SATA drives (except the boot drives, because it is an old Supermicro drive and can't boot from NVMe). and replaced them with various NVMe drivers.

My two old 100GB Intel S3700 SATA drives I used in a mirror as LOG/ZIL/SLOG drives were replaced by a couple of Intel 280GB Optane 900p drives. They absolutely excel at this task. I hooked them up using a cheap 8x PCIe to 2x U.2 adapter and my Supermicro boards 8x -> 4x bifurcation capability.

After some reading, I determined their performance was great enough to split them into 3 partitions each, so that I could use them as LOG/ZIL/SLOG drives for 3 different ZFS pools. This is not recommended, but it works perfectly, and I've never noticed a write slowdown, even when multiple pools write at the same time.

Then I picked up 3x Asus 16x PCIe to 4 m.2 slot boards and used those to replace the rest of the SSD's in the system. To save some money I decided to go with Microcenters excellent Inland Premium MLC drives, and configured them as follows:

1.) I added a "Special VDEV" for metadata and small files, which I didn't have before by mirroring three 2TB m.2 drives
2.) I replaced my two striped 512GB 850 Pro drives serving as read cache with two 1TB m.2 drives, similarly striped (data loss doesn't matter in the cache, so striping is fine)
3.) I had a mirror with two 1TB Samsung 850 EVO drives dedicated TV recordings using MythTV. That got replaced with two 1TB M.2 drives in a mirror.
4.) I had an old 128GB Samsung 850 Pro serving as a swap device. I replaced that with an MADM mirror using two 256GB m.2 drives
5.) I moved my VM/LXC datastore off of my SATA boot drives (the only SATA SSD's that remain in the system) and created them their own mirror with two 256GB M.2 drives
6.) And finally I had a lone 128GB Samsung 850 Pro used for a MythTV LiveTV ring-buffer which I replaced with one one lone 256GB m.2 drive.

This was a fun little project, especially since Supermicros documentation and labeling of the bifurcation settings in the BIOS was wrong, so it took some trial and error.

I finished swapping these in back in September and thus far, despite heavy writes, there is next to no wear on them as of yet. Like, not enough to register as a percentage unit change yet. I think these will do quite nicely and last a very long time. 10+ years.


I did have one of my spinning hard drives start giving me read and write errors and dropping out of the pool. Just in case it was a fluke, I cleared the errors and re-added it a couple of times, but they came back, so I guess I have a bad Seagate drive. I tried to do Seagates advanced RMA replacement on the drive, but they are unable to do that due to staffing shortages, only offering me a vague promise they weould get a drive out to me at some point after returning my defective drive when it was in stock. I was a little concerned by this, as I don't want to go long with my pool in a degraded state, so I bought a new 16GB Exos x18 drive on Amazon to replace it with in the interim.

It is a little busy, but here is essentially my current state when it comes to ZFS.

The MADM mirror for swap space and the standalone LiveTV ring buffer drive are not included here.

I decided to do MADM for Swap instead of ZFS as I suspected there might be a RAM problem here. If RAM ever gets low enough that I need SWAP (which it shouldn't) if the swap space were a ZFS mirror, you could have the ZFS memory use go up as the ARC increases in size, causing even more swap to be necessary, and causing a problem. I decided I didn't want that, so I went with MADM here instead as not being a Copy on Write file system it does not ahve the same RAM impact. I'll probably never need it, but better safe than sorry.


I think I'm going to go ahead and buy a 16GB Seagate drive every month or so and gradually swap it in to grow the pool. Once I've received my RMA for the current defective 10TB drive, and have all of my other 10TB drives sitting around, I'll ahve to decide if I want to add them back in for a third hard drive data VDEV. I haven't decided if I want to do this yet, or swap the old drives into my remote backup server instead. It currently has 16x old 4TB WD Reds in it. It's not low on storage yet, but it might be some day...


Anyway, that was a weird meandering post that started being an SSD endurance commentary, yet somehow wound up being a description of th ecurrent state of my storage server...

 

[robbiekhan]

That is quite a lot to take in! Interesting though and yeah since those days of talks about SSD endurance being a thing, things have moved on drastically for the better where as you say, it's a total non issue now and SSD endurance is often outpacing HDD endurance as evidenced by Puget's findings too.

Interesting that NVMe failures are higher than SATA SSD though. I guess it';s the higher temps they run at. My 970 Evo Plus has a heatsink attached but the pad is only covering the controller chip not the flash chips. Idle it's around 45-55 for the controller whilst the memory chips are pretty constant around 55 or above depending on use. Under load the controller hits 78 whilst memory chips are 80+. The SATA SSDs I've had barely get above 35 degrees by comparison.

 

[Zarathustra[H]]

That is quite a lot to take in! Interesting though and yeah since those days of talks about SSD endurance being a thing, things have moved on drastically for the better where as you say, it's a total non issue now and SSD endurance is often outpacing HDD endurance as evidenced by Puget's findings too.

Yep. The way I think of it, the drives I'm popping in today are likely going to be good for at least 10 years in the worst of cases.

In 10 years, do I really want to still be using these drives, when something way fancier, faster and larger is on the market?

Interesting that NVMe failures are higher than SATA SSD though. I guess it';s the higher temps they run at. My 970 Evo Plus has a heatsink attached but the pad is only covering the controller chip not the flash chips. Idle it's around 45-55 for the controller whilst the memory chips are pretty constant around 55 or above depending on use. Under load the controller hits 78 whilst memory chips are 80+. The SATA SSDs I've had barely get above 35 degrees by comparison.

That's an interesting thought.

Temperature might have something to do with it. I mean, generally temperatures are pretty forgiving when it comes to silicon, and the fact that it is solid state, and not moving parts should mean that as long as you keep it within the operational TJMax specs you should have next to no impact on longevity from temperature, just like with CPU's and GPU's, but Flash NAND is weird in the way it is designed to hold charge over time, and who knows how that might interact with temperature.

Because I was curious, I just used smartcl to grab the temps of the SSD's in my server. The two Optane drives are relatively hot at 44 and 43 celsius respectively, but all of the Inland Premium m.2's are in the high 20's. Low of 27C, high of 29C.

Not sure how much of this is due to differences in design between those m.2 sticks and the u.2 Optane's, and how much is due to the added airflow the fan in those 16x PCIe adapters provides.

I'm guessing it's a little bit of each.

The ambient temp in the server room is also a little warm at 76F (24.5C), as I am still working on the temperature vent.

So, actually the temp of those m.2's really isn't bad, sitting at 2.5C to 4.5C above ambient.

 

[robbiekhan]

From my understanding in recent times since upgrading my OS drive from a SATA Intel 730 series 480GB Skulltrail SSD to this Samsung 970 Evo Plus 1TB is that NVMe SSDs are designed to run hot for the memory chips whilst the controller should not his its thermal throttle limit else the IC will throttle the transfer speeds down.

I note that I have been using Samsung Magician since getting this drive and between a major update before and after Samsung adjusted what the app classed as "normal". for the temps of the NVMe. Before the update it was saying that the 65 degrees I was seeing was High. But after the update anything below 80 degrees is classed as Normal:

(Ignore the write speed there, Windows 11 still has the slow write speed issue to NVMe/SSD drives for the OS disk)

From what I read the memory should not be cooled because for data longevity the memory chips should be in a hot state, not tjmax hot, but hot, so in the 50s etc. Keeping them passively or actively cooled too low could result in data integrity errors as flash chips are designed to be optimal when hot whereas the controller chip should be cool to avoid thermal throttling. Hence why I cut away the thermal pad on my mobo's NVMe heatsink so it only covers the controller leaving the memory chips without contact on the heatsink.

 

[Zarathustra[H]]

From what I read the memory should not be cooled because for data longevity the memory chips should be in a hot state, not tjmax hot, but hot, so in the 50s etc. Keeping them passively or actively cooled too low could result in data integrity errors as flash chips are designed to be optimal when hot whereas the controller chip should be cool to avoid thermal throttling. Hence why I cut away the thermal pad on my mobo's NVMe heatsink so it only covers the controller leaving the memory chips without contact on the heatsink.

That is interesting. I had never heard that before. I'll have to do some reading.

I presume the temperature readings come from the controller, so who knows what the memory chips themselves are at.

 

[robbiekhan]

That is interesting. I had never heard that before. I'll have to do some reading.

I presume the temperature readings come from the controller, so who knows what the memory chips themselves are at.

The funny thing is the Samsung Magician app reads only one temp, so you'd assume it was the controller but in actual fact it's the memory chips if checking side by side with an app that reads more sensor data like HWINFO64. Here's mine currently as an example:

 

[Zarathustra[H]]

The funny thing is the Samsung Magician app reads only one temp, so you'd assume it was the controller but in actual fact it's the memory chips if checking side by side with an app that reads more sensor data like HWINFO64. Here's mine currently as an example:

View attachment 431766

Hmm. Interesting.

I'm guessingf this might differ from manufacturer to manufacturer.

I use smartctl from the command line to pull the data from the SSD's on my server.

I don't have Samsung SSD's in my Server, but I do have a 970 EVO in my desktop as a secondary drive. If I use smartctl on that, this is what I get:

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 970 EVO 1TB
Serial Number:                     
Firmware Version:                   1B2QEXE7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 1,000,204,886,016 [1.00 TB]
Unallocated NVM Capacity:           0
Controller ID:                      4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          1,000,204,886,016 [1.00 TB]
Namespace 1 Utilization:            951,523,602,432 [951 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 5581b417d8
Local Time is:                      Thu Jan 13 19:24:19 2022 EST
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     85 Celsius
Critical Comp. Temp. Threshold:     85 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     6.20W       -        -    0  0  0  0        0       0
 1 +     4.30W       -        -    1  1  1  1        0       0
 2 +     2.10W       -        -    2  2  2  2        0       0
 3 -   0.0400W       -        -    3  3  3  3      210    1200
 4 -   0.0050W       -        -    4  4  4  4     2000    8000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        21 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    17,648,786 [9.03 TB]
Data Units Written:                 32,653,802 [16.7 TB]
Host Read Commands:                 62,279,576
Host Write Commands:                45,927,353
Controller Busy Time:               374
Power Cycles:                       837
Power On Hours:                     548
Unsafe Shutdowns:                   99
Media and Data Integrity Errors:    2
Error Information Log Entries:      2,591
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               21 Celsius
Temperature Sensor 2:               42 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0       2591     0  0x1006  0x4004      -            0     0     -

So, three temps, two at 21C (are they the same sensor? Who knows) the other at 42C, no indication which is which. Interestingly my SSD has never seen Samsung Magician, so it has never received any firmware updates, but it lists both the warning and critical temp threshold at 85C.


My main SSD in the desktop is a Sabrent Rocket 4.0. (It was the only PCIe gen4 SSD I could find when I bought my Threadripper)

=== START OF INFORMATION SECTION ===
Model Number:                       Sabrent Rocket 4.0 2TB
Serial Number:                     
Firmware Version:                   RKT401.1
PCI Vendor/Subsystem ID:            0x1987
IEEE OUI Identifier:                0x6479a7
Total NVM Capacity:                 2,000,398,934,016 [2.00 TB]
Unallocated NVM Capacity:           0
Controller ID:                      1
Number of Namespaces:               1
Namespace 1 Size/Capacity:          2,000,398,934,016 [2.00 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            6479a7 2aa0933336
Local Time is:                      Thu Jan 13 19:27:09 2022 EST
Firmware Updates (0x12):            1 Slot, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005d):     Comp DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     90 Celsius
Critical Comp. Temp. Threshold:     95 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.78W       -        -    0  0  0  0        0       0
 1 +     6.75W       -        -    1  1  1  1        0       0
 2 +     5.23W       -        -    2  2  2  2        0       0
 3 -   0.0490W       -        -    3  3  3  3     2000    2000
 4 -   0.0018W       -        -    4  4  4  4    25000   25000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        46 Celsius
Available Spare:                    100%
Available Spare Threshold:          5%
Percentage Used:                    0%
Data Units Read:                    22,320,758 [11.4 TB]
Data Units Written:                 38,426,068 [19.6 TB]
Host Read Commands:                 127,189,541
Host Write Commands:                122,497,992
Controller Busy Time:               614
Power Cycles:                       370
Power On Hours:                     4,085
Unsafe Shutdowns:                   55
Media and Data Integrity Errors:    0
Error Information Log Entries:      2,185
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, max 63 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0       2185     0  0x100d  0x4004  0x028            0     0     -

Only one temperature reading on this one, 46C. Interestingly warning and critical temps are higher than Samsung, at 90 and 95C respectively.

And then I also have my old 400GB Intel SSD750 in a PCIe slot:

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPEDMW400G4
Serial Number:                     
Firmware Version:                   8EV10174
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          400,088,457,216 [400 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Thu Jan 13 19:30:28 2022 EST
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -     512       8         2
 2 -     512      16         2
 3 -    4096       0         0
 4 -    4096       8         0
 5 -    4096      64         0
 6 -    4096     128         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        35 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    40%
Data Units Read:                    68,335,724 [34.9 TB]
Data Units Written:                 39,357,156 [20.1 TB]
Host Read Commands:                 953,138,854
Host Write Commands:                691,668,327
Controller Busy Time:               42
Power Cycles:                       2,112
Power On Hours:                     17,002
Unsafe Shutdowns:                   306
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged

Again, only one temp reading, lower at 35C (probably due to larger enclosure)

No listed warning temps either.

I found it interesting that the Samsung and Sabrent SMART data will tattle on you if you have spent any time above the temp limits. I'm guessing this might impact warranty claims.

 

[Zarathustra[H]]

The funny thing is the Samsung Magician app reads only one temp, so you'd assume it was the controller but in actual fact it's the memory chips if checking side by side with an app that reads more sensor data like HWINFO64. Here's mine currently as an example:

View attachment 431766

Comparing that to the server (I'll do one of each type)

The Optane drives are similarly sparse as the SSD 750 in my desktop is:

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPE21D280GA
Serial Number:                      
Firmware Version:                   E2010435
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          280,065,171,456 [280 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Thu Jan 13 19:33:43 2022 EST
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    18.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        43 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    6,987 [3.57 GB]
Data Units Written:                 6,430,275 [3.29 TB]
Host Read Commands:                 85,848
Host Write Commands:                80,661,507
Controller Busy Time:               28
Power Cycles:                       60
Power On Hours:                     4,267
Unsafe Shutdowns:                   2
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

Single temperature (43C) no warning or critical temps listed.

Next, the Inland Premium:

=== START OF INFORMATION SECTION ===
Model Number:                       PCIe SSD
Serial Number:                      
Firmware Version:                   ECFM13.3
PCI Vendor/Subsystem ID:            0x1987
IEEE OUI Identifier:                0x6479a7
Total NVM Capacity:                 2,048,408,248,320 [2.04 TB]
Unallocated NVM Capacity:           0
Controller ID:                      1
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          2,048,408,248,320 [2.04 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            6479a7 4d0020202c
Local Time is:                      Thu Jan 13 19:37:01 2022 EST
Firmware Updates (0x12):            1 Slot, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005d):     Comp DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x08):         Telmtry_Lg
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     75 Celsius
Critical Comp. Temp. Threshold:     80 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.51W       -        -    0  0  0  0        0       0
 1 +     6.47W       -        -    1  1  1  1        0       0
 2 +     4.96W       -        -    2  2  2  2        0       0
 3 -   0.0490W       -        -    3  3  3  3     2000    2000
 4 -   0.0018W       -        -    4  4  4  4    25000   25000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        28 Celsius
Available Spare:                    100%
Available Spare Threshold:          5%
Percentage Used:                    0%
Data Units Read:                    67,117 [34.3 GB]
Data Units Written:                 3,783,159 [1.93 TB]
Host Read Commands:                 805,400
Host Write Commands:                163,152,416
Controller Busy Time:               61
Power Cycles:                       19
Power On Hours:                     2,802
Unsafe Shutdowns:                   8
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged

It also just lists the one temperature (28C). Interestingly it has the lowest warning and critical temps to date, at 75C and 80C respectively.

 

[Deadjasper]

A while back I picked up a little Moderro box and I'm using it as an HTPC. It's totally solid state. No moving parts at all and of course it's dead silent. It has an nvme SSD in it. It has one hugh 6" x 6" heatsink that serves it all. The nvme has a thermal blanket that fills the space between it and the heatsink. I haven't checked temps, it's been flawless so far. Maybe I better not check them. I might be tempted to put a fan in it.

 

[EnderW]

Huh. Interesting. I would have expected it to be gone based on everything I have read as well, or at least hopelessly corrupted.

Maybe it was some sort of enterprise model with an unusually capable battery built into it or something?

Or - more likely - it was an SLC drive. Since each cell only has one state (on or off) the effects of cell discharge are much less likely to flip a bit than they are in MLC, TLC or QLC drives. But still, 10 years is quite a long time...

Or maybe a little bit of both...

I still have an OCZ Rally2 Turbo 4GB that I believe is a confirmed SLC drive.

 

[VoodooRufus]

I got a steal on this Optane 380GB M.2. Always wanted to pick one up for a fun drive or boot drive but they were always stupid expensive. The stock Intel "Thermal Solution" is pretty much a joke. On basic drive speed tests it would throttle before the end of the test. Picked up a Thermalright 22110 M.2 heatsink and the throttling problem was solved.

I also have a SATA MLC SSD that it just killer for a SATA drive, too.

Planning on seeing if I can use it for my Z390 boot drive soon.

 

[Zarathustra[H]]

I got a steal on this Optane 380GB M.2. Always wanted to pick one up for a fun drive or boot drive but they were always stupid expensive. The stock Intel "Thermal Solution" is pretty much a joke. On basic drive speed tests it would throttle before the end of the test. Picked up a Thermalright 22110 M.2 heatsink and the throttling problem was solved.

I also have a SATA MLC SSD that it just killer for a SATA drive, too.

Planning on seeing if I can use it for my Z390 boot drive soon.

3D Xpoint (used to make the Optane drives) was an amazing technology technology. It's really sad that Intel decided to shut down the line, though I understand that it wasn't exactly profitable for them.

Picking up used ones is a viable alternative as they have nearly inexhaustible write cycles, so it will likely be good.

Only downside with them is that they are Gen 3 drives.

Random small size reads and writes and latency are all still top notch, better than anything else on the market except RAM, but sequential reads and writes are not going to keep up with the latest NVME drives. It's really just a matter of what your workload looks like, but I still think for the majority of workloads an Optane will feel snappier.

I can't help but wonder what the go-to Log/ZIL/SLOG drives for ZFS will be going forward, now that the rockstar Optane's are gone.

 

[daglesj]

Only downside with them is that they are Gen 3 drives.

I saw a YouTuber doing a review of a H20 optane drive today (was from last year). He said that the speeds were only 3300MBps but somehow he could run a script and load 20 apps almost instantly, which was amazing for the "low speed".

Thing is why wouldnt it be fast when the average app is maybe 200MB in payload? This is why you get the law of diminishing returns for day to day usage when you go above 500MBps.

 

[VoodooRufus]

I compared the Optane to my HP EX920 and even though the sequentials are still in the high 2000MB/s range, the Q1D1 test was 3-4x faster than the flash based drive. Looking at Gen4 drives the Optane still is better by a large margin in that game.

Once I migrate to a Gen 5/DDR5 based rig, the Optane likely won't go into it, but it is basically peak Gen 3 performance for my Z390 machine.

 

[daglesj]

I like using the little optanes for pagefile cache drives. Set them to 12GB -12GB (leave 1GB for some extra OP just in case) and forget about them. Cheap as chips now for the 16GB ones. It was interesting tech. I don't think we've seen the last of it though...

 

[Vengance_01]

I got my M10 32GB drive for 30$ on ebay.

 

[Zarathustra[H]]

OOp.

Forgot to update the latest upgrade.

Over the last year, I have swapped in one 16TB drive to replace a 10TB drive every month. Finally done

NAME                                                   STATE     READ WRITE CKSUM
 poolname                                             ONLINE       0     0     0
 raidz2-0                                             ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
 raidz2-1                                             ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
   ata-ST16000NM000J-2TW103                           ONLINE       0     0     0
special
 mirror-4                                             ONLINE       0     0     0
   nvme-PCIe_SSD_2TB                                  ONLINE       0     0     0
   nvme-PCIe_SSD_2TB                                  ONLINE       0     0     0
   nvme-PCIe_SSD_2TB                                  ONLINE       0     0     0
logs
 mirror-3                                             ONLINE       0     0     0
   nvme-INTEL_SSDPE21D280GA                           ONLINE       0     0     0
   nvme-INTEL_SSDPE21D280GA                           ONLINE       0     0     0
cache
 nvme-PCIe_SSD_2TB                                    ONLINE       0     0     0
 nvme-PCIe_SSD_2TB                                    ONLINE       0     0     0

NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
poolname  176T  74.9T   101T        -         -     5%    42%  1.00x    ONLINE  -

 

[Welsper]

175tb of NVME storage is pretty sexy.

 

[robbiekhan]

Just upgraded from a 970 Evo Plus 1TB to a 990 Pro 2TB. Got it for £194 which is much cheaper than even a 980 Pro 2TB thanks to a 15% code Samsung sent on my birthday, and then a further £10 cashback because I paid via Samsung Pay lol.

So my storage currently looks like

Samsung 2TB 990 Pro NVMe (OS, games, apps)
Samsung 8TB 870 QVO SATA (Documents, Photos, working backups)
Western Digital 5TB My Passport USB-C HDD (backups archive)

I will be looking to get an 8TB NVMe or another 8TB SATA once prices start to get cheaper as they haven't shifted since I bought the 870 QVO and replace the 5TB WD which will rid my life completely of any mechanical drives.

I no longer archive movies or TV shows like I used to because 1Gbps internet, can just stream or download anything I need on the fly within 10-20 mins flat for 4K HDR.

 

[Zarathustra[H]]

175tb of NVME storage is pretty sexy.

I don't know if you are talking about mine, but mine is mostly spinning rust.

I do have 14 NVME drives in it for various purposes (cache, ZIL, metadata and small files, etc.) but the mass storage is all on good old fashioned spinning rust.

 

[Welsper]

I don't know if you are talking about mine, but mine is mostly spinning rust.

I do have 14 NVME drives in it for various purposes (cache, ZIL, metadata and small files, etc.) but the mass storage is all on good old fashioned spinning rust.

Yeah I read it as 175tb of NVME, didn't even see the spinning rust, lmao.

 

[Vengance_01]

root@truenas:~# zpool status
  pool: Docker-VM Data
 state: ONLINE
config:

        NAME                                    STATE     READ WRITE CKSUM
        Docker-VM Data                          ONLINE       0     0     0
          2074eb75-ec71-48c9-aac6-97e786ede3a9  ONLINE       0     0     0

pool: Main-Pool
 state: ONLINE
  config:

        NAME                                      STATE     READ WRITE CKSUM
        Main-Pool                                 ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            4e25a577-6577-11e9-bef8-001b21a4368a  ONLINE       0     0     0
            598b0aa3-6577-11e9-bef8-001b21a4368a  ONLINE       0     0     0
            63898031-6577-11e9-bef8-001b21a4368a  ONLINE       0     0     0
            6c6f2f48-6577-11e9-bef8-001b21a4368a  ONLINE       0     0     0
            74f4f5bf-6577-11e9-bef8-001b21a4368a  ONLINE       0     0     0
        cache
          9523253a-ab44-497c-9884-5bd9654fa8e9    ONLINE       0     0     0

pool: boot-pool
 state: ONLINE

         NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sdg3      ONLINE       0     0     0

Main Pool Config is 5 4TB 5400RPM drives (still with 50% free) with a Intel Optane 32GB nvme drive as cache
Boot Pool is a 128GB Old Samsung 830 SSD booting from USB 3.0
Docker VM Pool is a sata III Inland 512GB Value SSD (Running Plex, and a few other containers)
Server is a 5600g, Asus B450ITX, 32GB Ram, Nvidia P400 for Plex Transcoding, and a PCI-e nvme stick to 5 SATA ports(no raid at all).

Super solid, dead silent. My little build is tiny compared to some of you guys

 

[Zarathustra[H]]

root@truenas:~# zpool status
  pool: Docker-VM Data
 state: ONLINE
config:

        NAME                                    STATE     READ WRITE CKSUM
        Docker-VM Data                          ONLINE       0     0     0
          2074eb75-ec71-48c9-aac6-97e786ede3a9  ONLINE       0     0     0

pool: Main-Pool
 state: ONLINE
  config:

        NAME                                      STATE     READ WRITE CKSUM
        Main-Pool                                 ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            4e25a577-6577-11e9-bef8-001b21a4368a  ONLINE       0     0     0
            598b0aa3-6577-11e9-bef8-001b21a4368a  ONLINE       0     0     0
            63898031-6577-11e9-bef8-001b21a4368a  ONLINE       0     0     0
            6c6f2f48-6577-11e9-bef8-001b21a4368a  ONLINE       0     0     0
            74f4f5bf-6577-11e9-bef8-001b21a4368a  ONLINE       0     0     0
        cache
          9523253a-ab44-497c-9884-5bd9654fa8e9    ONLINE       0     0     0

pool: boot-pool
 state: ONLINE

         NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sdg3      ONLINE       0     0     0

Main Pool Config is 5 4TB 5400RPM drives (still with 50% free) with a Intel Optane 32GB nvme drive as cache
Boot Pool is a 128GB Old Samsung 830 SSD booting from USB 3.0
Docker VM Pool is a sata III Inland 512GB Value SSD (Running Plex, and a few other containers)
Server is a 5600g, Asus B450ITX, 32GB Ram, Nvidia P400 for Plex Transcoding, and a PCI-e nvme stick to 5 SATA ports(no raid at all).

Super solid, dead silent. My little build is tiny compared to some of you guys

Very nice. Please tell me you use ECC RAM?

Also, would be pretty cheap to mirror that Docker drive to protect against losing your VM configurations.

 

[robbiekhan]

So after the Samsung 990 Pro issues, my drive got returned for refund and I was contacted by Sabrent offering to send me some storage.

I now have:
2TB Rocket 4 Plus-G (OS/apps/games)
8TB Rocket 4 Plus (data)
8TB Samsung 879 QVO in a UGREEN USB 3.1 enclosure (backups of the above)

The old 1TB 970 Evo Plus is going to sit in the cupboard as a backup hot spare incase something happens to the 2TB Sabrent one day.

2TB Rocket 4 Plus-G:

8TB Rocket 4 Plus:

 

[Zarathustra[H]]

Speaking of solid state data retention, the very first flash drive I ever bought was a 512mb Simpletech. It's been sitting in a drawer for god knows how many years. I recently came across it and plugged it in. Yep, data was still there. I opened a few files and everything appeared normal. Given it's a flash drive I would have thought Everything would be gone, I know it's been at least 10 years since it was last plugged into a computer. Don't know if today's flash drives would fair as well but I do believe the tech exists, and has always existed, to make long term solid state storage feasible.

That is interesting. I wonder if it had some long lasting batteries on board. I would have expected it to run into problems if unplugged for 10 years as well!

That said, if it was old enough to be SLC, SLC does naturally survive these things much better than more modern multi-level designs (MLC, TLC, QLC, etc.) do simply because a 1 is "charge", and a 0 is "no charge", so if the voltage starts dropping over time, it is much less likely to reach absolute zero and be considered "no charge" than a multi-level design is, to reach a certain lower voltage level that flips the bit.

Same with MLC drives, being more resilient than TLC drives, and so on down the line.

 

[Zarathustra[H]]

So after the Samsung 990 Pro issues, my drive got returned for refund and I was contacted by Sabrent offering to send me some storage.

I now have:
2TB Rocket 4 Plus-G (OS/apps/games)
8TB Rocket 4 Plus (data)
8TB Samsung 879 QVO in a UGREEN USB 3.1 enclosure (backups of the above)

The old 1TB 970 Evo Plus is going to sit in the cupboard as a backup hot spare incase something happens to the 2TB Sabrent one day.

2TB Rocket 4 Plus-G:
View attachment 551009

8TB Rocket 4 Plus:
View attachment 551010

I would have kept the 990 and flashed it (or RMA'd it if the wear was already bad) and kept the Samsung drive.

Sabrent's Rocket 4 drives have had a spate of sudden deaths. These are not a "kind" failure like the Samsung 990 which wore out and went read only, but rather - as happened to me - shut down the computer one night with the drive reporting something like 95% remaining life, and the next morning having it be a brick when you try to power it on again, with nothing that can be done to restore data.

This happened to my 2TB Rocket 4.

Its not a rare incident. Just google it, and you'll find countless occurrences. Like this one business that had 7 out of 36 drives suddenly fail.

I bought a Samsung 980 Pro same day at Microcenter to keep going (this was before the 990 launched) and RMA'd the Sabrent (which was a nightmare, they will try revery trick to deny your RMA, if you want an easier time of it, register your drives within 3 weeks (I think it was 3 weeks) of buying them otherwise they will give you lots of shit).

I popped the Sabrent drive back in when I got a replacement back, but I no longer trust it. I now use it for only my steam library, because I have a gigabit internet connection, so if I lose the drive I can always just quickly download the games again, nothing will be lost. I tend to max out steam at ~118MB/s on large downloads. The drive is nowhere near full, but if it were and I lost 2TB of games, I could re-download them all in under 5 hours.

No big loss. Download the first game you want to play in 15 minutes. Get the rest overnight and be back to normal next day.

I'll keep using the Sabrent drive for this purpose as long as it keeps working, but I don't think I'll buy a non-Samsung drive any time soon. They may have stumbled with the 990 and the initial firmware on the 980, but prior to that they were just about flawless since the 840 EVO issues 8 years ago. I'd call that a pretty good track record.

Ever since I had bad experiences with OCZ back in the day, I have considered Intel and Samsung to be my go-to brands. And I don't trust the gamer brand rebrand drives. Intel is no longer around having sold their SSD business to SK Hynix, so I guess that leaves Samsung. I might consider WD too, as they have their own production and controllers rather than relying on badge engineering. I only bought my Sabrent Rocket 4.0 in 2019 when I upgraded to my Threadripper, because it was the only drive I could find that actually supported Gen4.

In retrospect this was a mistake, as it had reliability issues, and the 4KRND speeds (which is what really matters in overall responsiveness) were actually lower than the 980 Pro. I should have just kept buying Samsung, even if that meant a Gen3 drive. It would have perfomed better. Heck even my old Gen3 1TB Samsung 970 EVO (not plus) performs better in 4KRND than my 2TB Sabrent Rocket 4.0 does.

 

[Deadjasper]

While we're at it, I recent bought a Patriot 1.92TB SSD. Put it in a newly built Linux box and immediately knew something wasn't right. It kept freezing with continuous drive activity. I pulled it out and put in a Windows box for testing and it was dead as a door nail. Thank god I bought it from Amazon, easy peasy return.

EDIT - Also like to add it was light as a feather, no doubt an all plastic case. I'll forever wonder what was or wasn't inside.

 

[EnderW]

That is interesting. I wonder if it had some long lasting batteries on board. I would have expected it to run into problems if unplugged for 10 years as well!

That said, if it was old enough to be SLC, SLC does naturally survive these things much better than more modern multi-level designs (MLC, TLC, QLC, etc.) do simply because a 1 is "charge", and a 0 is "no charge", so if the voltage starts dropping over time, it is much less likely to reach absolute zero and be considered "no charge" than a multi-level design is, to reach a certain lower voltage level that flips the bit.

Same with MLC drives, being more resilient than TLC drives, and so on down the line.

I would pay a premium for some small capacity SLC drives but have never seen anything for sale. Still have and use my OCZ Rally2 Turbo 4GB for important things like BIOS updates.

 

[Zarathustra[H]]

I would pay a premium for some small capacity SLC drives but have never seen anything for sale. Still have and use my OCZ Rally2 Turbo 4GB for important things like BIOS updates.

Yeah, I haven't seen them in over a decade, and back when they were available, they were crazy expensive enterprise parts, and weren't particularly fast by modern standards.

That, and to be fair, due to improvements in process technology (3D-VNAND) process maturity/yield, software logic to decrease write amplifcation and many other issues modern TLC drives probably have better write endurance than the SLC drives of that era.

My take is that these days SSD's are relatively cheap. They are cheap enough to buy them, beat them to crap, and then replace them before they fail.

I actually put some TLC drives in my server in high write configurations. At the rate they are going even the cache drives are going to last 8-10 years, at which time I will likely have itched for an upgrade two times over already

Real data wise, my inland premium ZFS L2Arc (read cache) drives now have an uptime of 12,500 hours (1.42 years) and SMART is only listing them as 3% used.

They are going to last for a while. (at this rate, 47 years)

 

[robbiekhan]

Personally I've not had any SSD failures since the original Crucial MX series when I rolled those 120GB models out to basically every machine at work cloning over from HDDs. A bunch died about a year later. None of my personal SSDs have ever died so am holding confident that these Sabrents will also be fine, all firmware updated etc too. Whatever happens though I have an 8TB backups drive I FreeFileSync to every other day and do a semi regular Windows System Image of the OS drive so everything is basically backed up anyway if the worst happens.

My machine for ref is never turned off, always has a constant thermal range unless I am gaming and the 3080 Ti comes alive - The NVMe drives are all heatsinked up as well and temps are within the norm.

The Rocket 4 Plus-G now has over 6TBW, the 8TB R4 Plus has just over 4TBW and is also my active downloads drive so that will get more TBW than the main Plus-G. In a year of ownership, the SATA 8TB Samsung SSD got over 21TBW for reference, so am expecting similar for the Sabrent too as the use case is the same, with the same data.

Edit*
Worth pointing out too that every brand has a failure rate, some drives/devices etc will need to be replaced, it is what it is. Those 7 out of 32 Sabrents were in laptops, cramped environments and used in a Raid setup, also they were on Dell laptops so the warranty from Dell is void until they are put back to stock since Dell won't touch them otherwise. Also Sabrent's response seems fair there. Consumer drives in such a business use case environment? That will result in issues.

I'd rather stick with a brand that does respond quickly rather than send you your faulty drive back saying no issue found (this is what Samsung did with my 990) and only offer to fix things once you write about it publicly on a large tech website.

Edit2*
So I did some digging and found that the guy who claimed 7 out of 32 drives had died in this work environment, there are no RMA tickets raised for these and the guy has only had one support ticket which was for cloning failures. Seems like a hit-piece post more than anything else.

 

[ae00711]

The old 1TB 970 Evo Plus is going to sit in the cupboard as a backup hot spare incase something happens to the 2TB Sabrent one day.

if it ain't plugged in & ready to go, it's a cold spare, not a hot spare

 

[Vengance_01]

Very nice. Please tell me you use ECC RAM?

Also, would be pretty cheap to mirror that Docker drive to protect against losing your VM configurations.

So sadly I had to switch back to my old 2U Trusty rack mount supermicro Micro ATX board, E3-1265L v3, and 32GB ECC Ram. Same pool and drive config. After the latest upgrade to bluefin, my AMD ITX build was going bonkers booting reliably and giving me all kinds of fits..... So yes I have ECC ram . IPMI is a god send. Thank goodness I had a backup of my Scale Config.

 

[w1retap]

Current status.. getting a little full. Before the end of the year, I'm going to upgrade the array and figure out which route I want to take. I still want to have bitrot protection because my data spans over 25 years of hoarding. Currently running SnapRAID w/ StableBit which has been great. It would be easy to upgrade the drives since I can just pull a drive, clone it over to the new one and resync. I've also been looking at ZFS, BTRFS, and ReFS w/ Storage Spaces. We'll see. Everything is virtualized, so starting with some other storage array type shouldn't be too bad. Transferring all that data is going to take some time though. I think the easiest way would be to pull the bottom row of drives, then throw in a few 22TB disks so I could still have the SnapRAID pool up and running concurrent with the new pool.

*** SMART warnings on the drives are just for head park event threshold. Green drives that I forgot to update the firmware on to prevent head parking.

 

[w1retap]

Well, I bought some drives. First I ordered (4) 20TB Seagate Exos from Newegg during their sale. Sent them all back because 3 out of 4 were DoA. Tested them all externally as well just to confirm it wasn't my backplane or something.

Fast forward 3 weeks and I hopped on the WD Red Pro sale from Western Digital's website. Tested all these, and 3 OUT OF 4 ARE DEAD. Honestly, wtf. One screeches like a dolphin, two of them just go brrr brrr brrr, and one works fine. All packaging was pristine OEM, no visible damage or prior use.

Currently trying to RMA these and the WD keeps saying they are "Out of Region" when I try to register them for RMA. Then I call their customer support and all I get is India and they keep sending me to some foreign support line that doesn't speak English. I finally got an Indian person who spoke good enough English to understand my order number and validated they are indeed under warranty. They said they'd send me an RMA via email right after the call, yet I get nothing. So sick of this.

 

[Spartacus09]

Currently trying to RMA these and the WD keeps saying they are "Out of Region" when I try to register them for RMA. Then I call their customer support and all I get is India and they keep sending me to some foreign support line that doesn't speak English. I finally got an Indian person who spoke good enough English to understand my order number and validated they are indeed under warranty. They said they'd send me an RMA via email right after the call, yet I get nothing. So sick of this.

WD has been like that for years, you have to contact WD support on 1/3 the stuff you buy in the US just to get the warranty to list right, its dumb but seems like they've made no effort to fix it.