[Advice] New File/Media server with ZFS

I am in the process of building a new server that will do:
- File Hosting,
- Media Server (plex?),
- bit torrent
- run some VMs,
- and maybe a little bit more.

The main priority is data integrity, so I want to use ZFS with SAS drives and ECC RAM. I do not have experience using ZFS.
I want to split the file groups into:
- Video (possibly split into TV and Movies) (raid6 equivalent)
- Music (radi6 equivalent)
- backups (raid6 equivalent)
- personal (raid5 equivalent)
-misc (raid5 equivalent)

What are your recommendations for the OS, HBA/RAID card, and hard drives?
This is the hardware I currently have:
- 24 Bay X8DT3-F Supermicro server,
- 2x X5650 cpu,
- 48GB ECC RAM
- LSI MegaRAID 9260

Server 2012R2
Your argument for ZFS is not valid at all.

HW RAID card will need BBU and you will need cover the power draw of the system, it will be hungry.

Backups DON'T need RAID-6, RAID-0 would be fine, they are backups, not primary data.

Backups on RAID 0?

Remember, if you have ANY versioning implemented in your backup system, your backup system holds primary data. If your backup data is gone, those older versions are gone. Do not use RAID 0 for your backups if you need or want older versions of your data.

I'm currently running a similar setup to what you're envisioning, but on a smaller scale (5 disks in a RaidZ1 configuration (Raid 5 equiv - rest of the specs in my sig). I have Plex and Virtualbox running as well and performance seems to be adequate.

With regards to disks, I'm of the opinion that you do NOT need SAS grade disks for a good platform. ZFS takes care of the data integrity element of it so long as the memory of the system is stable (thus, the use of ECC memory). I'm using consumer grade Seagate 3TB drives in mine and have been running since early last year without issue. I did have a drive go tango uniform a couple weeks ago and replacing it was very easy. The main goal with selecting drives is keeping in mind that replacement disks will always have to be the same capacity or larger, so if you have a 3.01 TB drive, you can't replace it with a 3.00TB drive without destroying the pool and rebuilding it (some will underprovision the drives slightly in order to avoid this potential complication as the "rounding" on 3TB by each drive maker is usually off by a few bytes).

For your setup, I think we'd need to know how many disks you plan on buying (all 24?) for your rig to better advise on how to set it up. It might turn out that you will want to use one large "pool" with RaidZ2 (Raid 6 equiv) and then create volumes/partitions for each of the subdivisions that you want. Once you get over a certain number of drives, two pools can be a better option depending on your desired level of redundancy - there are others in here that can better speak to those logistics (as there's an optimal number of drives for each pool that I'm not well versed in).

Lost-Benji said:

Server 2012R2
Your argument for ZFS is not valid at all.

HW RAID card will need BBU and you will need cover the power draw of the system, it will be hungry.

Backups DON'T need RAID-6, RAID-0 would be fine, they are backups, not primary data.

Click to expand...

Can you explain your rationale for Server 2012R2 above and beyond the argument for ZFS being invalid (also, why is ZFS invalid)?

I agree. Backups on raid0 is insane (just like backing up on tapes that you might not be able to read from when you need them.) Also, the ZFS comment is not helpful without context. The point of ZFS and ECC is to minimize the chances of having backups not being usable when you need them. On the other hand, if you are willing to back up to raid0, ZFS is pointless

Lost-Benji said:

Server 2012R2
Your argument for ZFS is not valid at all.
...
Backups DON'T need RAID-6, RAID-0 would be fine, they are backups, not primary data.

Click to expand...

Why RAID-0? If I lose 1 drive, I lose all the data. Also, the "back-up" portion is for backing-up files that are on other computers that are on the network, not the server.

David_Schroth said:

For your setup, I think we'd need to know how many disks you plan on buying (all 24?) for your rig to better advise on how to set it up. It might turn out that you will want to use one large "pool" with RaidZ2 (Raid 6 equiv) and then create volumes/partitions for each of the subdivisions that you want. Once you get over a certain number of drives, two pools can be a better option depending on your desired level of redundancy - there are others in here that can better speak to those logistics (as there's an optimal number of drives for each pool that I'm not well versed in).

Click to expand...

I plan on buying the drives in sets, so buy all the drive for torrents or all the drives for video, then on to the next set a few weeks later. For example, I am consider 6x 4TB drives or 5x 6TB for video; 4x 4TB or 3x 6TB or 5x 2TB for music.

Lost-Benji said:

Server 2012R2
Your argument for ZFS is not valid at all.

HW RAID card will need BBU and you will need cover the power draw of the system, it will be hungry.

Backups DON'T need RAID-6, RAID-0 would be fine, they are backups, not primary data.

Click to expand...

Putting your backups on RAID-0 is very bad advice, and here's why:

When (not if) one of the drives fails, you lose all of the data on the array. You have to notice the drive has failed, order another drive, wait for it to arrive, find time to install it, then transfer all backup data over to the new RAID-0 array before anything is backed up again. Doing a full backup from nothing takes a long, long time.

That's an awful long time to be without backup protection. Hard drive failures are inevitable, so you will inevitably be without any backups at all whenever a drive fails. And you have to start from scratch each time a hard drive fails, which is a pain.

Adding just one more drive for RAID-Z1 or similar protects you from all of these scenarios at the cost of just one drive.

Hmm nice system.

I would ditch the MegaRaid 9260 in favor of a 9207-4i4e though (that external connection will allow you to throw on a JBOD without getting extra parts and running a new SAS line within the chassis). In terms of the OS I think Linux is going to be the way to go for your ZFS needs. I like Solaris but without support you don't have anywhere near the virtualization options that exist in Linux, or the amount of programs.

I would do 2 x 5-disk vdev Raidz for MASTER_POOL and 2 x 6-disk vdev Raid2z for BACKUP_POOL which leaves you room for two spares (one for each pool). A single SSD for the internal non hotswap drive is fine but if you get creative you should be able to get two on that internal cage wired to the motherboard and configured for Raid 1 (use mdadm for that) would be gravy.

Install KVM and your QEMU tools and you are ready to go.

Gallager2014 said:

I am in the process of building a new server that will do:
- File Hosting,
- Media Server (plex?),
- bit torrent
- run some VMs,
- and maybe a little bit more.

The main priority is data integrity, so I want to use ZFS with SAS drives and ECC RAM. I do not have experience using ZFS.
I want to split the file groups into:
- Video (possibly split into TV and Movies) (raid6 equivalent)
- Music (radi6 equivalent)
- backups (raid6 equivalent)
- personal (raid5 equivalent)
-misc (raid5 equivalent)

What are your recommendations for the OS, HBA/RAID card, and hard drives?
This is the hardware I currently have:
- 24 Bay X8DT3-F Supermicro server,
- 2x X5650 cpu,
- 48GB ECC RAM
- LSI MegaRAID 9260

Click to expand...

David_Schroth said:

Can you explain your rationale for Server 2012R2 above and beyond the argument for ZFS being invalid (also, why is ZFS invalid)?

Click to expand...

OP has indicated he/she is a novice with a file system that is renowned for being a pain in arse to deal with. Then mentions and large hardware list including a HW RAID card (which the OP hasn't mentioned flashing it to IT mode). He/she has also indicated the want for VM hosting. Server 2012R2 just ticked the boxes. Also adding in the logic of only RAID-5 for personal data (the stuff that usually is NOT replacable) yet the stuff that is gets double parity....

AgentQ said:

Putting your backups on RAID-0 is very bad advice, and here's why:

Click to expand...

So the precious ZFS everyone loves so much needs to keep a full backup on an equally large array with parity (multiple actually) which then all of you are inplying that is kept on the same machine using the same hardware and same OS, oh come on. Stop compensating you lot.

Put your backups on a second machine or NAS (best use for a bloody NAS, backup target only), then worry about how many disks you want to loose to parity.

But the advantage to ZFS over RAID is that ZFS is suppose to prevent bit rot, which my current NAS is starting to suffer from.

I am willing to learn how to use ZFS. The problem I am running into is that Oracle has made it a PITA to find when they bought Sun.