[H]ard Forum Storage Showoff Thread

[TeeJayHoward]

Nice how that raid0 setup could max out even 100Gb.

Yeah. I thought about using that config. (Heck, I also priced out a 100GbE network. $750 or so, not that bad... House is already fiber'd up too... Hrm.) Ended up deciding to have "some" resilience built in. So, I played around with ZFS. By setting the ashift to 9, I could get better speeds and more capacity than the previous 13 I was "strongly recommended to use for flash":

 

[EnderW]

Definitely not the most practical or cost efficient setup but you already knew that. The 10Gbe NIC is a bummer but like you said, upgrading means a faster switch. And a fast NIC on the other end.
Props for doing something unique and cool. I always liked those Mac Pros but could never justify the cost and even less so now with M series CPUs.

 

[uOpt]

Very interested in your ZFS parameter tests, TeeJayHoward

 

[TeeJayHoward]

Very interested in your ZFS parameter tests, TeeJayHoward

I ran ashift=9, 12, 13, 14, 15, 16. I screenshotted the DiskMark run after each setup, but didn't note how much capacity was lost. If I recall correctly, it ranged from 54TB available w/ ashift=9 to... 22TB available w/ ashift=16? Something like that. I'll try and update this post with all the screenshots after work.

edit: Apparently I didn't screenshot ashift=12. Damnit. From my notes, the SEQ1M QD8 was 8.5GB/s read, 850MB/s write.
ashift=9

ashift=13

ashift=14

ashift=15

ashift=16

I always liked those Mac Pros but could never justify the cost and even less so now with M series CPUs.

Kinda how I feel about them. I can't recommend them. When I bought this, my plan was to throw Windows on it and dual-boot. Game and do CAD under Windows, daily drive it under MacOS. It worked for a little while, but Apple is terrible at staying up to date with their systems. The 580X started aging. I bought a W5700X. That wasn't enough, so I... Realized there's nothing better I could realistically install. Oh, sure, I "could" put in the W6900X, but it was something like $6k? For what was essentially a thousand dollar 6900 XT. There's no drivers available for most other GPUs. If I wanted to toss an nVidia card in there for my Windows sessions, I'd have to unplug the HDMI cable and swap it over every time I swapped OSs. I ended up spending so much time in Windows that there was no point to having the Mac hardware. Then the attempt to upgrade to Windows 11 happened, and... Well, it wasn't pretty. I spent hours tricking the system into getting Windows 11 installed. Then some minor thing happened - a driver wasn't available, or a Windows update borked it, I can't recall. I threw up my hands and built a Windows rig. The Mac Pro got put in the basement and forgotten about.

At this point, I feel like a Mac Mini connected via NoMachine would serve me so much better than a big rig like the Mac Pro. The 2023 model has fewer PCIe lanes than the 2019 model, and Apple no longer supports ANY GPUs. Half the PCIe cards that work with the 2019 model don't work with the 2023 model. I just can't see myself ever buying one. I do still love MacOS, despite it's ever-growing list of irksome qualities. For example, SMB. If can read locally from an array at 20GB/s, why can I only read over the network at something like 250MB/s? Especially when I can write at over 1GB/s and iperf3 shows line speeds? Why do you gimp VNC when it's the protocol behind Screen Sharing, which works incredibly (but only Mac->Mac)? Why no IPMI? Why can't you make a modern monitor turn on before you're done booting all the way to the OS login screen? Why do you have RAID-0 and RAID-1 but no RAID-5? Will you please just kiss and make up with nVidia?

 

[EnderW]

I thought there was a way to install off the shelf AMD GPUs in the Mac Pro but maybe I’m thinking of the older model? Guess it would work but no PCI-E power connectors?

 

[TeeJayHoward]

I thought there was a way to install off the shelf AMD GPUs in the Mac Pro but maybe I’m thinking of the older model? Guess it would work but no PCI-E power connectors?

Apple removed the AMD GPU drivers for the Mac Pro in the Apple Silicon models. They work in the 2019 model to a degree. Apple's power setup is both super cool and frustrating. There's modular PSU connectors under the cards in both the 2019 and 2023 Mac Pro models... But they have a 300W limit shared for the whole chassis, and no way to run another line to the PSU since it's so integrated. So if you're not pulling power from the MPX slot, you can't power high-end cards. And I don't know that there is (or was) an MPX->12V converter made to overcome that.

 

[Zarathustra[H]]

Yeah. I thought about using that config. (Heck, I also priced out a 100GbE network. $750 or so, not that bad...

I have never been able to get good performance out of anything over 10Gbit, at least to a single client.

I have 40 Gig now, and while it does a great job of satisfying loads from multiple clients at the same time, it never quite achieves the speeds you'd expect from a single client, despite the storage backing it having plenty of bandwidth.

And it's not just with storage. Even with iPerf I tend to max out at about 18-20Gbit for single connections. (which is higher than actual storage transfers, likely due to some storage protocol overhead or other inefficiencies)

Only way I get anywhere near 40Gbit/s is if I use a large number (like 20 or so) of parallel connections in iperf.

I suspect you'd have the same issue with 100Gbit. Great in theory, but your file transfers will still likely be limited to about 1.6 - 1.8 GB/s either way :/

In other words, I wouldn't waste your money.

Unless you have a server supporting a large number of clients simultaneously, the highest you are likely able to max out is going to be 10Gbit Ethernet. You'll continue to see performance increases if you go from 10gbit to 25 Gbit, but with limiting returns. Above 25Gbit it winds up being pointless.

 

[Zarathustra[H]]

By setting the ashift to 9, I could get better speeds and more capacity than the previous 13 I was "strongly recommended to use for flash":

These higher ashift recommendations have nothing to do with performance, and everything to do with write amplification.

SSD's usually have very large internal block sizes. The firmware just lies to the OS that it is 512B or 4k and lets the controller manage things. Unlike the Advanced Formate hard drives that were3 4k internally, but present themselves as 512B to the OS, this block size emulation actually works really well from a performance perspective in SSD's. After all, thats how all OS:es use them, and how they are benchmarked, so there is an incentive to make it work.

The downside is that this results in serious write amplification, and this is increased by how the ZFS copy on write system works.

Ashift=9 may work really well from a performance perspective, but I'd recommend keeping an eye on your SMART data, as you might find alarmingly accelerated consumption of write cycles.

In theory, if we set the ashift such that it matches the internal block size of the SSD's in use, we eliminate the write amplification problem, and get the same (or possibly even better) performance than at ashift=9 but the issue is that none of the drive manufacturers publish their internal block sizes, and these block sizes may not even line up with ZFS ashift settings.

Increasing ashift will still help reduce write amplification (which is the reason why everyone recommends it, whether they know it or not, they may just be passing on "wisdom" they don't understand) but unless you are lucky and hit the internal block size spot on, this will reduce performance.

I suppose one could try to get there by experimentation, creating and re-creating the pool at different ashift sizes, measuring the performance.

If one plots it, I'd expect to see high performance at ashift=9, and then see it drop with every increase in ashift, until there is a surprising peak in performance when the ashift randomly matches the internal block size.

But even then, this may never happen, as there is no guarantee the internal block size of the SSD aligns with the sizes offered by the ashift setting.

Example. Ashift=13 is 8K (2^13) Ashift=14 is 16k (2^14). What if your internal block size is 12k?

Also, even if you get the right internal block size, ZFS has no way of communicating with the drive internals and making sure those blocks align properly. There is no guarantee that the first whole block starts at 0 or at an even MB. It could be all over the place.

So in reality - unless you really luck out - you have a choice. Set ashift=9 and get pretty awesome read performance (but probably pretty bad write performance, due to the SSD controller having to write a full internal block for every ashift-9 512B (2^9) block.) and in the process wear out your drives rather quickly OR increase ashift resulting in a reduced read speed (but probably an increased write speed) and much reduced write amplification.

This is an unfortunate reality of using ZFS (or any other Copy on Write File system) on SSD's


As for your SSD's, just looking at your speed test, if I had to make an educated guess that your internal block size is probably ~16k, so Ashift=14 is probably close to optimal, but again, no guarantee this actually performs well, as the blocks may not be aligned, and may not be sized in perfect powers of 2.

 

[TeeJayHoward]

I have never been able to get good performance out of anything over 10Gbit, at least to a single client.

That makes me feel better about sticking to my 10GbE setup.

If one plots it, I'd expect to see high performance at ashift=9, and then see it drop with every increase in ashift, until there is a surprising peak in performance when the ashift randomly matches the internal block size.

I don't have access to my screenshots while I'm at work, but from what I recall of my performance testing, the performance order for ashift went 12 < 13 < 9 < 14 < 15 < 16. I did just like you said and recreated the pool at various ashift sizes. Increasing the ashift value reduced the available space, but also increased the performance. ashift=16 was almost as fast as RAID-0 if I recall correctly - but you lose out on something like 30TB of storage space. ashift=9 being faster than ashift=13 while also allowing for 4TB more space (I think?) was a very pleasant surprise. Write performance was ~30% faster than ashift=13. I think ashift=14 was where I wanted it to be - 1GB/s, but I couldn't lose that much space.

Thankfully, this is a NAS. 95% of the data on there will be WORM, so I shouldn't need to worry about wearing out the NVMe drives once I've got them loaded up.

 

[Zarathustra[H]]

Unless you have a server supporting a large number of clients simultaneously, the highest you are likely able to max out is going to be 10Gbit Ethernet. You'll continue to see performance increases if you go from 10gbit to 25 Gbit, but with limiting returns. Above 25Gbit it winds up being pointless.

Let me add to this.

I should qualify the above statement with "at least as it stands today" as it likely won't age well.

I remember back when I first got 10gig NIC's in 2014, and the same was true back then, but at lower speeds. I could max out gigabit, but I couldn't get more than 2.5-3.5 Gbit out of my 10gig adapters to a single client.

I'm not sure why this is. PCIe bus has plenty of bandwidth, CPU doesn't see significant load during file transfers. I've never been able to figure out where the bottleneck is. My best guess is that there are inefficiencies in the software that clients run, and they only get tested against typical client hardware. Back in 2014 gigabit NIC's were typically available, but 10gbit was a rare unicorn in a client.

These days, 10Gbit is not uncommon in a client, with many motherboards shipping with it ion board. Anything above 10Gbit is pretty rare in clients though.

I suppose that when new hardware becomes more common on the client side, someone complains, or a developer actually winds up using it and realizes it is slow, and sets out to optimize the code, but this doesn't happen until the hardware is commonly available on the client side.

Either way, my observations are my observations, and they pretty much align with others observations online. The discussion regarding why it happens is pure guesswork though.

It is worth noting that I have seen some suggestions that one can get more performance out of high bandwidth NIC's by manually performance tuning them thogh hand optimizing the latency though.

Someone linked me to a guide on how to do this a few months back, and I can't seem to find it now.

It had to do with measuring your actual observed latency in your application, doing some light math, and optimizing detailed network settings based on that.

I will look around and see if I can find it.

 

[Celcius]

Upgraded to the 4TB Samsung 9100 Pro today


View: https://imgur.com/BUH8MEz
My first gen 5 nvme drive

 

[TeleFragger]

I have never been able to get good performance out of anything over 10Gbit, at least to a single client.

I have 40 Gig now, and while it does a great job of satisfying loads from multiple clients at the same time, it never quite achieves the speeds you'd expect from a single client, despite the storage backing it having plenty of bandwidth.

And it's not just with storage. Even with iPerf I tend to max out at about 18-20Gbit for single connections. (which is higher than actual storage transfers, likely due to some storage protocol overhead or other inefficiencies)

Only way I get anywhere near 40Gbit/s is if I use a large number (like 20 or so) of parallel connections in iperf.

I suspect you'd have the same issue with 100Gbit. Great in theory, but your file transfers will still likely be limited to about 1.6 - 1.8 GB/s either way :/

In other words, I wouldn't waste your money.

Unless you have a server supporting a large number of clients simultaneously, the highest you are likely able to max out is going to be 10Gbit Ethernet. You'll continue to see performance increases if you go from 10gbit to 25 Gbit, but with limiting returns. Above 25Gbit it winds up being pointless.

I capped at a slow 9.4gb on Win10 with connectx-1 and 2 cards.. proprietary CX4 connectors... I have threads up here getting it going with help of members and showing speeds of 1gb and 10gb differences.

I have since moved to Brocade SFP+ 10gb, intel nic and mellanox connectx3 cards.. runnin 10gb on win10, win11, (plex, storage server, gaming rig and esxi 8)

 

[Zarathustra[H]]

Upgraded to the 4TB Samsung 9100 Pro today


View: https://imgur.com/BUH8MEz
My first gen 5 nvme drive

Why not upload image directly to the forums?

That would avoid this problem:

 

[TeleFragger]

I capped at a slow 9.4gb on Win10 with connectx-1 and 2 cards.. proprietary CX4 connectors... I have threads up here getting it going with help of members and showing speeds of 1gb and 10gb differences.

I have since moved to Brocade SFP+ 10gb, intel nic and mellanox connectx3 cards.. runnin 10gb on win10, win11, (plex, storage server, gaming rig and esxi 8)

and so people know I am not throwing internet warrior comments out there...
just downloaded iperf3.18 and here...
from my gaming rig to my plex server...

 

[TeleFragger]

Swapping from spinning rust to flash for the home NAS setup. While I wanted to build an EPYC/Threadripper setup for this, the budget simply didn't allow for it. So, I'm reusing an old workstation I had laying around instead. I figure the Xeon W-3223 in it has 64 lanes. They're 3.0, but that's fine. I'm limited by the NIC anyway. No Bifurcation, which means expensive cards, but still doable.

8x Crucial P3+ 4TB M.2 NVMe drives per Sonnet 8x4 card.
View attachment 695319

Heatsink attached:
View attachment 695320

Do it again so there's two maxed out cards:
View attachment 695321

Toss 'em in the MacPro:
View attachment 695323

And put the case back on:
View attachment 695324

Then clean up the mess left behind.
View attachment 695325

Verify they're all there.
View attachment 695326

Now, how to set it up? RaidZ?
View attachment 695327

Eek. That's quite the performance hit. Not to mention my data hoarding would necessitate the addition of another loaded card. Hows about simple RAID-0 and I keep the spinning rust for backup?
View attachment 695328

Maybe just a plain ol' JBOD?
View attachment 695329

The numbers are... Well, about what I'd expect for a single NVMe drive. But the network's only 10GbE so they should be just fine. Power draw at the wall is a little under 300W. More than I wanted, but I imagine much less than the 24-disk E3v2 I was using previously. I'd grab numbers to compare but the old and new NAS are syncing this week.

Now for the next issue - why does MacOS suck so badly at SMB? iperf3 shows line speed. I can write data to RAID-0 or JBOD at over 1000MB/s. RaidZ is about where I'd expect at 750MB/s...ish. But reading from it caps out at about 750MB/s for JBOD. (And on the raid setups, it's even worse - 300MB/s!) Most of my files are 8-50GB in size. I tried lots of different block sizes, it didn't make a difference. Copying from one NVMe drive to another goes quick as. It's just over the network that things get terrible. So maybe I'll end up building that Threadripper/EPYC box eventually anyway. Shame there's not a <100W TDP chip out there with >80 PCIe 4.0+ lanes (and the associated mobo with 5+ x16 slots). If I do end up building the new rig, it's gonna be mostly wasted. I don't need cores, damnit, I need PCIe lanes!

edit: Giving SoftRAID a try. I do NOT like the idea of a yearly subscription just to be able to create raid volumes, but if the performance is there...

View attachment 695331 View attachment 695332

edit2: It isn't. That is SO much worse than even RAIDZ. And people pay money for this!?

edit3: Went back to ZFS. Increasing ashift values gave me significant performance improvements. ashift=15 was twice what ashift=13 was... But it was also half my capacity gone. Oddly, moving down to ashift=9 resulted in a performance improvement AND increased capacity. 54TiB still ain't enough, but I guess I can get rid of some of the older junk I've got in storage.

wow!!!!!

I ended up putting a dell quad card with bifurcation in my current setup. small drives too cuz my lab hardware is all leftover garbage!!!!!! loving your setup!