ZFS New Performance/Capacity build

N

ndejong1980

n00b
Joined
Dec 6, 2011
Messages
15
I'm planning a new ZFS (Omnios/NAPP-it) storage server build; need some advice (if needed) on my planned setup.

Main goal is a MAIN storage server that delivers good IO/Throughput and maximum safety of data.
An all SSD system is not an option atm ($$ and storage capacity)

1 x Supermicro X10SRH-CF (LSI 3008 IT FW)
Supermicro | Products | Motherboards | Xeon® Boards | X10SRH-CF

8 x Kingston KVR21R15D4/16HA (128GB Total)
Memory Search | Kingston

1 x Supermicro 836BA-R920B (Direct Attached Backplane, no Expander)
Supermicro | Products | Chassis | 3U | SC836BA-R920B

4 x Supermicro CBL-SAST-0508-01
SFF-8643 > SFF-8087

1 x Intel E5-1620 v3
Intel® Xeon® Processor E5-1620 v3 (10M Cache, 3.50 GHz) Specificaties

1 x Supermicro AOC-S3008L-L8i (IT FW)
Super Micro Computer, Inc. - Products | Accessories | Add-on Cards | AOC-S3008L-L8i

1 x Noctua NH-D9DX i4 3U
1 x Intel X540-T2 (2x10Gbe)
16 x HGST Ultrastar 7K6000 4 TB 3.5" SAS 4k Native (HUS726040AL4210)
2 x Intel 750 400 GB NVME for SLOG/ZIL and L2ARC (Partition)
1 x Intel DC S3500 80 GB (Boot drive)

Setup VDEV (need maximum IO and throughput; 1st choice:3way mirror):
5 x (3xMirror) setup; with an HS
or
4 x (4xRAIDZ2)

2 x Intel 750 400GB setup for SLOG/ZIL & L2ARC
Both 750's partition as 5%(20GB) and 25%(100GB) > 70% free
So SLOG/ZIL will be mirrored and the L2ARC striped (200GB).
Will this work? Advised to use paritioning on this PCIE NVME device?

Is this a good setup for a main storage server? (backups will be made to another ZFS system)
 
You'll achieve best write IOps with 8x 2-way mirrors. What kind of loads are you going to put on the system? Is the reason for considering 3-way mirrors data safety? Does the backup not offer enough protection, i.e. what are your fault tolerance requirements and disaster recovery strategy?
Do not make 4-disk RAIDZ2 vdevs.
70% over-provisioning of SSDs is overkill.
 
Load on the system will be different VM's (Exchange/SQL servers, VDI desktops) around 40-50 VM's (Servers and VDI)
I'm considering a 8x 2-way mirrors, but fault-tolerance must be high so that's why we choose for now a 3-way mirror setup (5x 3-way) but again a 8x2-way is also an option.
70% OP of the NVME 750 400GB, because we don't want a big L2ARC (200GB should be enough); also what i read every 10GB of l2arc = 1Gb of memory. So basicly we have a ~100GB ARC and 200GB L2ARC

EDIT: Our current server has a 4 x 3-way setup of 7K4000 2TB drives (SATA) and for our current setup performance wise it's enough. With the new setup we have an extra VDEV and the newer 7K6000 SAS drives.
 
ndejong1980 said:
Load on the system will be different VM's (Exchange/SQL servers, VDI desktops) around 40-50 VM's (Servers and VDI)
Click to expand...
Is that as zvols exported via iSCSI?
also what i read every 10GB of l2arc = 1Gb of memory.
Click to expand...
How much overhead the L2ARC headers have depends on the average blocksize in the L2ARC. Glancing at one of our servers that has mostly 128k blocksize I see 70MB of headers only for 100GB L2ARC data.
 
Currently we are using NFS for storage protocol to our other VM servers or better choice ISCSI over NFS?
 
People have differing opinions about that. I can't really recommend one over the other because we are "converged", running the KVM VMs and LXC containers on the same machines where the pools reside, thus avoiding all the network overhead and hassle.
 
I would use 1 750 for SLOG, 1 for L2ARC.

I would make sure you over-provision as you had planned, but maybe leave 200GB for L2ARC, IMHO you could leave all 400 if you weren't worried about drive wear-out, you have enough RAM for this much and more L2ARC. The SLOG size 20GB is likely even more than enough, and should help the write-wear by OPing so much. I wouldn't use the SLOG device for anything but a SLOG (ie: not L2ARC) because you could max out writes with it as a SLOG, and when you max out write and throw in reads the mixed work load will slow the drive.

I would tune/adjust transaction time for the SLOG based on your network configuration/performance -- you likely won't be filling 20GB before each flush based on a single 10Gig connection.
 
SQL Server on iscsi requires volblocksize set to 4k, if you use L2ARC then it will consume a SHITLOAD of memory just for ARC mapping.
 
ToddW2 said:
I would use 1 750 for SLOG, 1 for L2ARC.

I would make sure you over-provision as you had planned, but maybe leave 200GB for L2ARC, IMHO you could leave all 400 if you weren't worried about drive wear-out, you have enough RAM for this much and more L2ARC. The SLOG size 20GB is likely even more than enough, and should help the write-wear by OPing so much. I wouldn't use the SLOG device for anything but a SLOG (ie: not L2ARC) because you could max out writes with it as a SLOG, and when you max out write and throw in reads the mixed work load will slow the drive.
Click to expand...

I don't know -- the 750 handles multiple streams of traffic very well, it is based on an enterprise controller after all, I like the mirror SLOG and stripe L2ARC idea myself.
 
In my list i've used the 4K (native) SAS drives from HGST 0F22794 HUS726040AL4210; or do i need the 512e version (0F22795 HUS726040AL5210)? As i've read 4K native is the best choice for ZFS?
 
512e is, 512 bytes emulated. The drive is still "native" 4k.

ZFS use dynamic block sizes, what is best depends on what you're storing. But if you want good compression a larger block size is preferred.
 
4kn is the way to go, that way you don't need to worry about alignment, etc.


olavgg said:
512e is, 512 bytes emulated. The drive is still "native" 4k.

ZFS use dynamic block sizes, what is best depends on what you're storing. But if you want good compression a larger block size is preferred.
Click to expand...

He is referring to the industry standard terms or 512e/n or 4kn -- the 512ke drive appear to have 512b sectors, but they are just 4k sectors divided by 8. I'd MUCH rather have 4kn drives because then everything is native, no emulation, no need to worry about aligning it, etc.
 
Thanks for the replies :)
An option that we also are looking at (MB/CPU platform):
Supermicro | Products | Motherboards | Xeon® Boards | X10SDV-7TP4F
Has onboard tons of features, but maybe not compatible (Like the 10GB card from the Xeon D platform?)
The onboard LSI controller is an older version then the previous setup (TS) AOC-S3008L-L8i but i think this onboard LSI 2116 has enough throughput for the harddrives.
Then the 2 x Intel 750 would be used as pure SLOG/ZIL (Mirror), and an extra M2 SSD for L2ARC (Vendor/type?).
Mainboard connected with an SFP+ TwinAX cable to an:
Cisco SG350XG-2F10 12-Port 10GBase-T Stackable Managed Switch
 
You must log in or register to reply here.
Back
Top