• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Building ZFS + iSCSI Server

D

dxm765

n00b
Joined
Nov 9, 2014
Messages
6
Hello everyone! i did some poking around the various forums and i think i found soemwhere that will give me answers and help :) , ive been looking into expanding my storage for a while now. I currently have 8TB...but its not enough for what i need, here are the parts i have thus far and i will go into my questions after that :)

Motherboard: SUPERMICRO MBD-X8DTE-F-O
CPU's: Intel Xeon X5560 Quad core @ 2.80Ghz x2
CPU coolers: Noctua's (1366 Xeon)
RAM 24GB 1333Mhz DDR3 ECC
OS harddrives: RAID1 150GB raptors
HBA Cards: M1015 Crossflashed P16 IT Mode x3

Also possible use of SAS expander eventually.

My plan is to purchase x12 4TB hdds and create x2 6 hdd RAIDZ2's and then another x12 2TB hdd and create x2 6 hdd RAIDZ2's.
I.E: Group 1 x6 4TB hdds , Group 2 x6 4TB hdds , Group 3 x6 2TB hdds , Group 4 x6 2TB hdds giving me 24 hdds total.

I plan to use x2 Rosewell LV-4500 cases as they have 15 3.5" internal bays, one case will house the motherboard, HBA's and 12 hdd's, this case will also have a PCI bracket that will take my internal SAS to external SAS. I will be using 2 PCI brackets for Internal SAS to external SAS going from case 1 to case 2 which will then connect to another 12 hdds. This was the easiest method so that in the event i need to replace a harddrive in one of the cases i can simple power it down, remove the external SAS cables, and then unrack the server to do the replacement. This would also allow me to in theory use a SAS expander or 2 ( 3? :D ) and purchase one or 2 more internal to external SAS brackets and connect to another case ( Case 3 ) and have another 12 harddrives.

Some background:

I have all the parts listed above already and have been doing testing on a small scale, only x6 500GB hdds for right now configured in RAIDZ2. I am currently in the process of finding SSD's that i like/will function the best with ZFS caching. The reason i have selected Ubuntu for the OS is i have 2 dual port 10GB NICS i will be using and they are not supported in FreeNAS which was my original choice...with that being said the cards are supported with Ubuntu, i will be create the ZFS arrays in Ubuntu and then serving them to my main server ( Windows 2012 R2) via a direct 10GB connection between the 2 servers, the ubuntu server will be serving the storage via iSCSI and the windows server 2012 R2 will be running iSCSI initiator.

The plan is to offload all the storage on my windows server that currently contains my movies, tv shows, ISO's , and VM's + VM data.

At the moment i have my iSCSI target serving my server a zvol that is located on the RAIDZ2 of x6 500GB hdds
My objective is to have ZFS as the underlying file system and NTFS ontop of it....Now here is where the fun begins!

1) Is using a zvol an appropriate way/solution to serve the volume i need with iscsi?
2) I plan to use RAID 1 or RAID10 for 2-4 SSD's for caching, i would like recommendations on what SSD's (Brands / make / model) and best practices
3) Is there any issues with how i plan to setup my ZFS server? I.E. Hardware choices, possible bottle necks , etc
4) I would like some all around general input on this build, positive , negative, neutral and constructive criticism is welcome :)

I know its alot to read so thanks for taking the time to read it all and i look forward to reading some replies :)
 
May work as expected, maybe you should
- Be careful with SAS expander + Sata, test your expander + disks under load

- Add more RAM instead of using SSDs as a cache (RAM cache (Arc) is 100x faster than SSD (L2Arc)

- Do you really need iSCSI? Maybe you can use ZFS storage directly what gives you more flexibility with snaps (file based, accessable as Windows previous version)

-Zvol is ok

-you do not need to mirror or raid-10 SSD caches, just add them as single SSD cache for performance.
If one fails - no problem.

- about iSCSI stack.
Look also at the iSCSI Comstar on Oracle Solaris or the free Solaris fork OmniOS .
My first preference for ZFS and iSCSI - where ZFS comes from.
https://docs.oracle.com/cd/E23824_01/html/821-1459/fmvcd.html
https://b3n.org/freenas-vs-omnios-napp-it/

- Good SSDs for L2Arc are: Intel 3500,. Samsung Pro or Sandisk extreme
 
Last edited:
Feels like a waste, and too many points of failure.

Just strip it down, and use it as a samba or nfs server for your movies and tv shows.

Throw the ssd's into your windows box for your vm's.

It will cause a lot less issues for you in the long run.
 
_Gea: I will pick up one expander to start and go from there testing and stressing as you suggested which was the plan anyways :) , more RAM will certainly be added i'm using 24GB to start and will bump it up to 48 or 96 a short time soon once i figure out exactly what i plan to do, whether thats moving forward or redesigning. As per iSCSI i never planned to use snaps as i have everything backed up offsite thats constantly updating and saving revisions of files, as per SSD's i will pick up one to start and do some testing. As per the OS i will have to test them to see if my 10GB cards are functional in those OS's.

patrickdk: Could you please elaborate on these points of failures? As per the stripped down option and share it via samba or NFS would that allow me to mount reach RAIDz(X) as a different drive letter (primary reason for iSCI) , you would avoid SSD's for caching and simply use them for the VM's? Mind you some of the VM's have 500GB of data that is associated with them
 
The card has drivers for Solaris x86, so i wonder if it will work with x64, the 10GB cards i have are QLogic QLE3142
 
I cannot say if it will work but I would avoid to use any controller that is not supported out of the box
as driver support from a manufacturer is mostly bad and can be broken on OS updates outside Windows and partly Linux.

Illumos cares about mainstream server hardware and that is mostly Intel.

Try it.
 
Last edited:
Anything in the chain can break iscsi, why I wouldn't recommend it for normal home usage.

A network switch failure, a nic fialure, ups failure, server failure.

You have two servers, and network gear between them, anything having an issue there, will cause an iscsi issue. So I wouldn't recommend it for something as important as running your vm's on a remote server over iscsi at home. Unless you built your home for datacenter reliability.
 
You must log in or register to reply here.
Back
Top