ZFS expansion

[mmmmmdonuts]

My all in one SAN/NAS is starting to run out of room and I am going to need to expand very soon. I am at 72% currently on a RAID-Z2, with 6, 2TB Samsung HDD's. Currently I have one IBM-1015 where the drives are connected. My backup solution currently is JBOD with 4, 2TB drives. The current purpose is mostly media files and a few datastores over NFS (not more than 100GB currently).

I am trying to figure out what the cheapest option in the long run will be to expand storage. My original intent when I originally went with RAID-Z2 was to upgrade to 4TB drives and use the original 2TB drives as backup. (Raid + current backup) With the price of 4TB drives still being ridiculous, do you think it’s worth it to buy 6, 3TB drives or would a better solution be to redo the pool and go with an 8 drive RAID-Z2, with 2 more additional 2TB drives? I know this is cheaper now but not necessarily in the future when 8 drives need to be purchased to expand storage.

I also am thinking about adding 2 SSDs in a RAID-1 configuration and using them to serve up my local datastores because performance is not the greatest. I know I can disable NFS syncs, but with 4 VMs reading on the drive all at once it helps a little bit but performance still isn’t the greatest, and I plan on adding more VM’s in the future. I can also fit another IBM-1015 if needed in my MB.

I am trying to limit a budget to $1000 max but am flexible and am trying mostly to think about the future and planning. Any suggestions?

 

[danswartz]

Lots of stuff here. A couple of things: if you have a read-heavy job mix, raidz* is not the best solution - some raid10 works best, since the reads can be balanced both drives of each mirrored vdev. NFS sync mode affects write slowness, not reads. If you add two drives, you need to rebuild your pool anyway, so you might want to think about two 4-drive raidz vdevs, which will double your random read rate. I tend to prefer raid10 for these setups, but if you are already running somewhat low with 4 data drives in raidz2, a raid10 will lose you a drive. Even going to 6 3tb drives in raid10 only gains you 1tb of space. I like the SE3016 jbod chassis - 16 3.5 inch drives, hot pluggable. Can score on ebay for under $200. I would get 6 more 2tb drives and the jbod/expander, and build a raid10 pool with 6 striped mirrors. You get 12TB usable space, and your random read rate should be excellent.

 

[mmmmmdonuts]

So you don't think SSD's are worth it at all? The SE3016 works with ESXI 5, iirc? In the future as far as the pool goes can I add more mirrors easily or do I have to rebuild the pool again? If I change the say one raid10 at a time to say to 3TB and the rest are still 2TB, my pool size increases automatically to 13TB right?

 

[danswartz]

For what you are describing it doesn't sound like SSDs are related to fixing any issues. The SE3016 is just an expander - it is the HBA that will dictate whether things work or not. You never said if you are using an all-in-one, but you are using ZFS, so whether it is virtualized or a different box, esxi is just the client, no? If it's virtualized ZFS, you would already be passing thru the m1015 so the expander is not relevant. Nice thing about raid10 is you can add pairs of mirrored disks at any time without rebuilding the pool.

 

[mmmmmdonuts]

Okay. Thank you for clearing that up. It is an all-in-one system and yes esxi is just the client.

 

[danswartz]

For an all in one I would just disable nfs sync entirely. It is all one box, so if it crashes or loses power, eveything will go down (e.g. I don't think worrying about corrupted data ia a real concern...)