New (first major) ZFS Build

W

Willott

n00b
Joined
Jul 5, 2012
Messages
7
Just hoping someone can sanity check my spec. Looking to serve files (including video/audio media) to 300 clients, and NFS to (currently) 3 Xenserver hosts running 20 VMs across them (which is likely to increase) including an SCCM VM, and a web filtering box (which seems to do quite intensive database work).

Head:
Supermicro 6017R-N3RF4+ (includes X9DRW-3LN4F+ mobo)
2x E5-2620 Xeons
16x 16GB DDR3 1333MHz RAM (or 24x 8GB - which would be better)
1x LSI 9205-8e
1x Intel X520-DA2
2x 300GB 15KRPM drives

JBOD:
Supermicro 837E26-RJBOD1
21x Toshiba 1TB 7.2K RPM SAS drives (either 10 mirrored vdevs, striped with 1 hot spare, or several RaidZ2 vdevs with hotspares - ideas?)
2x ZeusRAM 8GB SAS (slog)
2x ZeusIOPS 200GB MLC SAS (L2ARC)

Various connecting cables

Idea would be to use either OpenIndiana or Illumian with Napp-it for now, possibly upgrading to Nexenta 4.0 if all parts fall on HSL (the motherboard and JBOD currently don't but there are other Supermicro JBODs on there, and there are E5 based motherboards going through cert currently). Next year would build a second head (lower spec - 1 CPU, 1/2 RAM) for failover, a couple of LSI SAS6160 switches and a second Cisco switch to stack (for extra 10GbE), and build some new virtual hosts.

I'm looking at a solution which will last at least 5 years (not counting drive replacement) and which allows expansion (both more space, and the ability to add replication for not much extra) and an increase in usage (quite possibly VDI or similar as we move forwards, more data access, more media in use on clients).

Any feedback or thoughts would be gratefully received.

Will
 
Your config seems quite ok for me

what I would do is replace the 15 K OS drives with smaller SSD's (40 GB+).
If you ask about RAM, 256GB is better than 192 GB, gives you more fast Cache,
maybee I would max out RAM without the L2ARC SSD's (ex 24x 16GB)

I would also create two pools. You cannot have best I/O and best capacity the same time.
Create one pool with best I/O and a mirrorred layout for VM's with one Zeus-RAM
(you do not need two unless you want to be save in case of a power-failure
causes a dead Zeus-RAM at the same time with the effect of about 5s data-loss)
Use as much 2x or 3x mirrored fast 10k/15k 2,5" spindels as needed.

The second pool used as a filer do not need a Log-device (no sync writes).
Optionally use one 24 x JBOD case for filer and a 24 x 2,5" case for fast 2,5" disks and the VM-pool.
Build a pool from two Z2 (each 10 disks) with 2 hotspare (if its fast enough for your workload)
or 2 x Raid-Z3 with 11 disks each for max. security + hotspare

If you need better read and I/O values for your filer, use bigger disks in a 2way or 3 way mirrored config
depending on security needs.
 
_Gea said:
what I would do is replace the 15 K OS drives with smaller SSD's (40 GB+).
Click to expand...

Thanks, I'll look at those - I should be able to get away with SATA for those as they're on separate controller, shouldn't I? Just need to find ones to slot into the drive sleds successfully (mostly 2.5" SSDs now).

_Gea said:
If you ask about RAM, 256GB is better than 192 GB, gives you more fast Cache,
maybee I would max out RAM without the L2ARC SSD's (ex 24x 16GB)
Click to expand...
I'll either go with 256GB or drop one L2ARC and up to 384GB RAM, dependent on pool config

_Gea said:
I would also create two pools. You cannot have best I/O and best capacity the same time.
Create one pool with best I/O and a mirrorred layout for VM's with one Zeus-RAM
(you do not need two unless you want to be save in case of a power-failure
causes a dead Zeus-RAM at the same time with the effect of about 5s data-loss)
Use as much 2x or 3x mirrored fast 10k/15k 2,5" spindels as needed.
Click to expand...

Was hoping that ZeusRAM and L2ARC would remove need for high speed spinning disk. I can see your point about splitting pools, but I was hoping that having all in one pool there would then be performance gains by the larger number of vdevs. Otherwise I'd split as 8 disks in 2x mirror (4TBish usable, 4 VDEVs instead of 10), 11 disks in RAIDZ3 (8TBish usable, 1 vdev) and have 2 hot spares. SLOG pair on the mirror pool, either 1x L2ARC on each pool, or a single L2ARC on one pool and nothing on other (well, ARC use).

I'm planning my capacity around about 5TB current use (approx 3TB user data, 1TB VMs), so don't expect expansion beyond 10TB (7TB User, 3TB VM), and hope that dedupe and compression would assist with both performance and space usage, hence happy to use mirrors - also trying for low cost - the 1TB SAS drives are 50% cheaper than 15K RPM 600GB disks... so I'd be looking at an extra ~£1k to put 15K disks in, giving me about 2.4TB space for VM storage.

Thanks for your help so far

Will
 
I would not use dedup, as it is quite problematic. But compression is fine to use.

And it would be better to max out RAM, that does a lot.
 
I thought dedupe was only really problematic if you have small amounts of RAM and little processing power... neither of which should be an issue here.

Will consider extra RAM and remove 1x L2ARC, any comments on second L2ARC and which pool it should go on, or pool structure considering staying with 7.2K RPM drives over 10K or faster?
 
Dedup is an issue of ram, when your block*320bytes exceeds your ram, and by default it will only be allowed to use 1/4 of your ram.

If you use pure 128k block writes only, your less likely to have dedup issues with ram, with 5tb of data, with 128k blocks, your looking at 12gigs of ram usage.

The issue is, dedup with 128k blocks is horrible, so you want to use smaller blocks, or things normally use smaller blocks anyways, so lets calc again using 8k block sizes.

with 8k blocks, and 5tb of data, you would need 186gigs of ram, just and only dedicated for the dedup tables.

and for 4k blocks, it would be 373gigs of ram.

This also doesn't touch the effects of using snapshots with dedup issues.
 
Thanks for the info patrockdk. Will leave Dedupe off in general, and maybe enable it on certain folders (like our student user space, which is likely to contain lots of duplicates and will have very regular, short time spaced snapshots, so snapshots should be quite small and removal shouldn't cause issues - unless there's something else I haven't yet got round to reading about dedupe!).

Any more opinions on pool setup? Do I need to bite it and go for 10K/15K drives for the VMs? Or just have 1 big mirrored pool and use the extra spanned vdevs to make up speed, or have a split pool using 7.2K RPM disks so VM load doesn't affect filer, and make full use of ZIL and ARC/L2ARC?

Cheers

Will
 
Not all clients will be doing audio/video at same time, most will be doc editing. I'd reckon between 30 and 60 on audio/video, talking MP4 format, not massive demands. Rest of access will mainly be office documents. In an educational setting, so budgets tight, and demands high :/ Just trying to find someone UK based who can do contract support in case of "Mythical IT Bus" situations (ie incase I get hit by a bus or similar), then this looks like a good project. Otherwise, I'm falling back to an EMC VNXe 3100 with 12x 300GB 15k disks and 6x 2TB disks.
 
Right, last post (think I've tied this down now to give me best cost/performance/space with expansion possibilities)

Go for 24x 26GB sticks in head (so 384GB RAM), do away with both L2ARC ZeusIOPs initially (see how ARC assists and whether normal working set spills over greatly), go for 11x 600GB 15K RPM SAS disks (Mirrored with 1 hot spare) for VM storage with the 2x ZeusRAMs as SLOG and 11x 1TB 7.2K RPM SAS disks (mirrored with 1 hot spare or RAIDZ3... will do some quick testing) for file storage. If further funds are available, I'll then start getting ZeusIOPs for L2ARC on both pools.

Any thoughts?

Also, anyone know UK based companies who will do Unix/OpenIndiana support on a call out basis?

Cheers

Will
 
Thanks for everyone's help, I've decided to order part of the system (partially as proof of concept, partially so it's a smaller initial outlay). Will be ordering 12x16GB sticks of RAM, 1x hex core CPU, 11x 1TB 7.2K RPM drives, 1x ZeusRAM. I'll be using as file storage, and doing some performance testing before putting file storage into production to see how well the 1TB drives copes with VM load with the ZeusRAM (and the various vdev layouts). If that's all good, I'll then be ordering another batch of drives (second ZeusRAM, either 11x 600GB 15K drives or 10x 7.2K 1TB drives, a shed load more RAM and a second CPU, and possibly a ZeusIOPS).

I'll be post back with feedback/performance once the first batch is here :)

Cheers

Will
 
You must log in or register to reply here.
Back
Top