• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZFS Build Check

I

impmonkey

n00b
Joined
Jun 19, 2012
Messages
17
I am planning on building a ZFS SAN for attachment to a small ESXi farm for my small business. About 20 virtuals including exchange.
This is what I have for a cart so far.
5BW41.png

I will be using the MLC 40gb ssds in Raid 1 for the OS (open + Nappit)
The 120gb Enterprise SSD's for the ZIL Log Files, and the 240gb Intels for the L2ARC cache.
Not positive how to go about mounting the ssds in the norco. Any Ideas?
Any positive feedback on other hardware choices? I am reading as much as possible and trying to make heads and tails of all the lingo.
*edit* Forgot to mention I would like to go with 10gbe for iscsi. Any recommendations?
Thanks!
 
Last edited:
Only 1 sff-8087 cable, or do you have 3 more cables allready?
Are you gonna connect your ssd to your motherboard, since you're only getting two sas-controllers, making you able to connect to 4 backplanes? Do you have the two reverse breakout cables needed to connect to your backplane?
 
Opps forgot to up the cable quantity. Thanks!

I do plan on attaching the ssds to the sas/sata connecters on the mobo.
If there is a way to get the ssds to fit in the bays I would use a reverse cable otherwise I will have to find a way to mount the ssds in the norco in another location.
 
16 drives in a 24 bay case, you could put the ssd's in the hotswap bays with 3.5 -> 2.5 converters. Or just fix them inside to the side of the case somehow. 2 SSDs for l2arc, you aren't mirroring them, right?
 
oh, and 120GB is way overkill for ZIL, I guess it might be ok to extend the life of the drive. ZIL rarely needs to be >4gb.
 
If you are using both of those SSDs, you have almost 500gb of l2arc, each l2arc entry requires an entry in ARC (aka RAM). I don't know the numbers right off hand, but I think you have too much l2arc given only 32gb of memory.

As for zil, if going ssd, I'd probably go intel 320, since it has the supercap. Partition it at 16gb (the max usable zil for 32gb of ram), leave the rest as over-provision should help increase it's lifespan. You could also look at something like the ddrdrive if you are really after performance.
 
I think SLC is preferred, but cheap SLC looks to be slow. The 320 has way better numbers than the 311. Don't forget you would be over provisioning these to a ridiculous degree, so it can do a lot of wear leveling, and you can replace them after 3 years or so.
 
I would look at skipping the Norco and PS and get a Supermicro 24 bay case with the redundant power supplies. Not only is the case better quality, you get redundant power. There is also a spot to mount two 2.5" drives inside the case and not use up drive bays. Plus is generally makes it easier to connect SM boards.
 
yeah ill source the 2 m1015's from ebay. Just had em in there to include them in the list of hardware.
 
Personally I'd not bother with any SSDs at first, probably not even for the OS (on a server you probably won't really benefit all that much from the OS being on SSD, apart from boot times)

Build the system first and test/benchmark performance - add SSDs once you have a good understanding of where/how/if they'll be of benefit.
You haven't mentioned how you'll be laying out the pool(s) - serving 20VMs concurrently may require IOPs performance (depending on what these VMs are doing of course) - you can test with both, but you may well find your performance is better with a multi vdev mirror pool rather than raidz2 or similar.
(put each mirror's member drives on a different controller).
There are trade-offs of course - there always are - but I'm assuming you are going for 16x 1TB spindles (rather than the cheaper option of 8x 2TB spindles) for performance reasons!

The case and power etc... up to you - obviously redundant power is nice for availability - Murphy's law states that a single PSU will blow once you have just taken off for your holiday in the sun :) (is there anyone else at your business who can change out a PSU?)
 
a vdev is a virtual device. Can be 1 drive, a mirror, or a raidz1, z2 or z3. If you are after performance and disk space isn't a main consideration, then your best performance would be with mirrors, as a vdev only performs as well as it's slowest member. With 16x drives, you'd have 8 mirror sets and the performance of 8 disks with 8TB usable. If you went with 2 raidz2 vdevs, you'd have more space (12TB), but only the performance of 2 disks, which is a big difference when hosting 20 vm's.

If you have 1 or 2 vm's (database or something similar) that really are performance hogs, also keep in mind a single mirror of 2 SSD's is going to outperform a 16 sata drive mirror set by a huge margin.
 
Last edited:
impmonkey said:
Anybody got any good links explaining vdevs and such?
Click to expand...


The easiest way to think of it is that pools are made from vdevs, and vdevs are made from physical disks. Each pool can have 1 or more vdevs.
Each vdev can be a single disk, mirror, raidz/z2/z3 - although ZFS by default wants all vdevs in a pool to be the same type (though you can override this).

Be aware that once you've added a vdev to a pool, you can't remove it without destroying the pool (though you can remove cache and log devices (aka L2ARC and ZIL devices).

Depending on performance and capacity considerations (though as I said earlier, I'm assuming you are going for 16x 1TB spindles for performance reasons) - you'd probably be best to start with a 16 drive pool made from 8 vdevs, where each vdev is a two disk mirror.
 
Last edited:
An vdev is a group of disks (or files, or partitions). Each group is configured as raidz1 (similar to raid-5) or raidz2 (raid-6) or raidz3 (allow three disks to fail) or mirrors. A zpool is made up of one or several vdevs. You can not remove a vdev from your zpool, but you can add a new vdev to your zpool.

There is more info on wikipedia.
 
Don't buy all your drives in one go, from the same mfr and same supplier.

16 drives, I would order 5 from 4 mfrs from 4 suppliers (I know that gives 20 but you'd have a couple of hot/cold spares). I would split each suppliers order between the drives so 2+2+1, 4 times.

I built my home server just under a year ago - got 15 2TB Hitachi's from the same supplier in one hit. So far replaced 7 of them! Not overly bothered as it's for a home media server and they were incredibly cheap (£44 for 7k2000's!!!), but i'd never do this for business critical stuff.
 
Ok after thinking this over and reading everything I could find I think I have a better understanding on this.
However I would like some input on what you guys would do.
Here is the current setup.
I currently have two esxi boxes with the following specs:
Norco 4u Rackmount
Asus d12/SAS http://pk.asus.com/Server_Workstation/Server_Motherboards/DSBFD12SAS/
Dual Intel Xeon E5410's
32GB of ram in one, 16GB in the other
4 1000 Nics in Each
Corsair CMPSU-620HX PSU
2 80gb drives in Raid 1 for the Hypervisor
ESXI 3.5 Free (I will be updating the esxi machine to probably 4.1 in the process)

SAN (Openfiler)
Norco 4020
Asus ASUS P5BV-E http://www.asus.com/Server_Workstation/Server_Motherboards/P5BVE/#specifications
Intel(R) Xeon(R) CPU X3320 @ 2.50GHz
4gb of Ram
Corsair CMPSU-750T PSU
Adaptec 31205
2 80gb drives in Raid 1 for the OS
12 1tb Samsung HD103UJ's running in Raid 10
3 2tb Luns shared over iscsi to the ESXi Boxes

Backup SAN
Same as above except:
Coolermaster Silent Pro M700 RS-700-AMBA-D3 PSU
Crappy Highpoint 2680
8 X 1 TB EcoGreen F2 HD103SI 5400 Drives running in raid 5
Split into 3 2tb smb shares attached to a VM used to backup other VM's

I currently have 7 VMs running one ESXi1 and 10 on ESXi2
ESXi1 hosts an exchange server, file and print server, DHCP, DNS, 2 DC's, an Ubuntu Knowledge Tree server and a couple other randoms
esxi2 hosts a ISA, IIS7, and a couple of xp workstations.
Here is a IO stat from the main SAN

[root@san ~]# iostat -m 5
Linux 2.6.24.7-0.2.6.smp.gcc3.4.x86.i686 (san.basiccom.local) 07/02/2012

avg-cpu: %user %nice %system %iowait %steal %idle
0.09 0.00 0.24 0.34 0.00 99.33

Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.00 0.00 0.00 2 1
sdb 2.43 0.00 0.02 92 448455
sdc 145.45 0.01 0.09 176345 1943561
dm-0 117.01 0.08 0.01 1795371 207223
dm-1 29.46 0.02 0.08 478122 1736337

avg-cpu: %user %nice %system %iowait %steal %idle
0.10 0.00 0.54 0.54 0.00 98.82

Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.00 0.00 0.00 0 0
sdb 2.18 0.00 0.03 0 0
sdc 410.69 4.14 0.28 20 1
dm-0 290.30 3.63 0.21 18 1
dm-1 121.58 0.51 0.07 2 0

avg-cpu: %user %nice %system %iowait %steal %idle
0.10 0.00 0.93 0.49 0.00 98.49

Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.00 0.00 0.00 0 0
sdb 4.15 0.00 0.03 0 0
sdc 703.36 6.60 0.43 33 2
dm-0 384.39 4.78 0.37 24 1
dm-1 319.76 1.82 0.05 9 0

avg-cpu: %user %nice %system %iowait %steal %idle
0.10 0.00 0.59 0.59 0.00 98.73

Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.00 0.00 0.00 0 0
sdb 3.37 0.00 0.02 0 0
sdc 476.04 6.93 0.17 35 0
dm-0 467.52 6.92 0.13 34 0
dm-1 9.31 0.01 0.03 0 0
Click to expand...

So not terribly high IO. When running a backup however this pretty much maxes this sucker out. The units are 3 1/2 years old and the drives are getting a little long in the tooth. I can feel the drive failures creeping up on me lol.

So my ? is what would you do.
I would really like to retire san2 and replace with a good zfs san attached thru either iscsi or NFS. I will then deligate san1 to backup status. Leave Openfiler or convert it to zfs?
If I were to convert it to zfs what replication process would you recommend between the two zfs's?
10gbe?
$5,000 USD budget.
 
impmonkey said:
So my ? is what would you do.
I would really like to retire san2 and replace with a good zfs san attached thru either iscsi or NFS. I will then deligate san1 to backup status. Leave Openfiler or convert it to zfs?
If I were to convert it to zfs what replication process would you recommend between the two zfs's?
10gbe?
$5,000 USD budget.
Click to expand...

ZFS (especially Solaris based but also FreeBSD) is expected to have a better stability and performance
especially with iSCSI and ZFS is unreached in data security and you get snapshots and clones as an add-on.
But your current SAN hardware is not really good for ZFS regarding Network Adapter, SAS controller and RAM.

I would replace the mainboards with a modern 1155 server mainboard (preferred Supermicro X9..-F) with at least 16 GB RAM,
and a LSI HBA (One internal 16 channel or 2 x 8 channel, cheapest is IBM 1015) and a Xeon
(Even a Dualcore 20W Low Voltage may be enough, the slowest Quadcore is really ok in any case)

Expect about 1000 USD minimum for each SAN upgrade
If you like, you can add 10 GBe Dual-Port Adapters (Intel, 500 USD each) to connect them directly (and the ESXi server)
or via a 10 GBe switch ex HP 2910 with 4 x 10 GBe option 2000 USD + CX4 or FC modul

To replicate ZFS you can use one of the zfs send scripts and applications (there are free and nonfree ones).
Replication is near realtime based on snaps and can be executed every few minutes to keep ZFS in sync.

another item:
in a three machine setup (10 GBe connect each to each directly), I would not only look for a backup/secondary SAN
but also for a backup VM server. Your Backup SAN could be an All-In-One (ESXi with virtualized SAN via pass-through).
In such a config, you have a Backup-SAN and a Backup VM server in one machine - all with 10 GBe -
within your 5k USD limit.
 
Last edited:
Cisco SG500X is only 1281$ and has also 4x10gbit SFP. As for NICs, Supermicro's integrated solutions (motherboard, lsi 2208 HW RAID, dual intel 10gbit) give you more bang for buck as standalone 10gbit NIC + motherboard, unless you need low power system.
 
Just going a few posts back, something left me confused. Are you guys saying that zfs raidz (1/2/3) only gives the performance of one drive? Could you guys explain that while keeping in mind that I'm used to the performance of traditional raid 5 and 6 being way higher than a single drive due to it reading from multiple drives simultaneously. Cause I also want to build a zfs system but I'm not sure if the performance hit moving from raid 6 to raidz2 will be worth it if you only get a single drive performance (and mirroring is not an option, just too much space wasted).
 
ian_stagib said:
Just going a few posts back, something left me confused. Are you guys saying that zfs raidz (1/2/3) only gives the performance of one drive? Could you guys explain that while keeping in mind that I'm used to the performance of traditional raid 5 and 6 being way higher than a single drive due to it reading from multiple drives simultaneously. Cause I also want to build a zfs system but I'm not sure if the performance hit moving from raid 6 to raidz2 will be worth it if you only get a single drive performance (and mirroring is not an option, just too much space wasted).
Click to expand...

If you look at sequential performance, where you read data track by track,
the performance increase on a raid-6 or ZFS Z2 compared to a single disk.

If you are looking at I/O with random read/writes, the sequential values are not helpful.
In such a case all heads of all disks must be positioned for every single read or write.
This is the reason why I/O of any striped Raid (Raid5/6, ZFS 1/2/3) is similar to one disk.
 
It's not overall performance they are talking about, but small random IOPS performance.
On large sequential transfers (ie throughput or bandwidth, measured in MB/s) you get the performance of multiple drives (minus overhead of course).


What you use your ZFS array for, is really what should decide your zpool layout!

You have similar (even if not identical) issues with more traditional raid levels/filesystems too!
 
Yeah, that makes sense. I knew raid 5/6 weren't great for i/o (no one seems to have a really good way of predicting / measuring raid i/o performance) but I think I might have missed that the discussion was aimed solely at i/o performance and not general read / write performance for large copies. Thanks for clearing that up :)
 
You must log in or register to reply here.
Back
Top