Question about moving to ZFS

S

SirMaster

2[H]4U
Joined
Nov 8, 2010
Messages
2,168
Hey guys, so I’ve decided to re-work my home media server and want to move to ZFS.

I’ve been reading up on it and I’m trying to figure out the best way to set up my server to do what I want. I’m just a bit overwhelmed with all the ways in which to run ZFS and I’m not sure which I should go with. I was just hoping to get some feedback from everyone with ZFS experience as to which server config they think is the best to go with.

So what I have is an ivy bridge CPU on a z77 motherboard and it is the non-K edition so it does have VT-d.

What I want from my server is to run ZFS obviously, but I also require some way in which to run a decently performing Windows Server 2008 in a VM.
I’m most comfortable with my Windows Server 2008 as that is what my server currently runs. I also have some Linux experience (Debian based stuff) so I am somewhat comfortable in that environment.

Also, I have never used Solaris, but I’m willing to learn to manage it.


Some of the options I have come up with are as follows:
Run ESXi on my server and then run Server 2008 and Solaris as VMs through that.
Run Solaris natively and then run Server 2008 through VirtualBox on the Solaris host.
Run Debian Linux natively (ZFZ on Linux) and then run Server 2008 through VirtualBox on the Linux host.
I think I would like to stay away from FreeBSD since I also need to run CrashPlan, preferably on the box with native access to the storage pool and they do not support FreeBSD. They do however support Linux and Solaris.


I believe lots of people run ZFS on Solaris through ESXi, but I read a lot of information about how you shouldn’t virtualize VFS, even with VT-d. So what do you guys think about ZFS on ESXi first off. The ESXi route seems the most appealing to me as it really gives my server the most flexibility to run what I want, but I will forgo it if it's indeed not recommended for ZFS. Remember, I'm just a home media server user for the most part. But I am still interested in data integrety.

Also keep in mind this is just a home media server so performance is not crucial, as long as I can get about 70MB/s R/W via my gigabit network I’m going to be happy. I plan on starting with a 6x2TB RAID2Z vdev and 6x3TB RAID2Z vdev in one zpool.

What is somewhat important to me is future flexibility. I think I want to use zpool version 28 or 5000 or whatever (I’m a bit confused by the versions). I think 5000 is the open source version that a lot of software uses? Point is, I think I would like the ability to move my zpool between operating systems in the future of my server needs change. Perhaps I will move from Solaris to Linux, or Linux to Solaris, or to something like FreeNAS on FreeBSD. So I’m looking at using something that is going to give me options if at all possible. I think this just means I don’t want to use the latest and greatest Oracle Solaris version zpool 34 or whatever. I don’t think I will need those features. But feel free to advise me on this idea.

I would like a GUI for managing the ZFS if possible. I believe many people use napp-it. So I would need to use Solaris for that. Not sure which actual distribution I should use then in that case.

If I can’t have a GUI it’s not a big deal.


I think I may actually prefer ZFS on Linux since I’m certainly more comfortable in Linux than Solaris, and from what I understand ZFS on Linux is more than stable and fast enough for a home server user with my needs. But if there is a real good reason to forgo my comfort with Linux and use ZFS on Solaris I’ll certainly consider the advice.

So far I have been playing with ZFS on Linux on a VM to familiarize myself with the commands and general operation prior to migrating my server and everything.
So far I have created a zpool and added a second vdev to it. I have exported and imported the zpool and unmounted and remounted it. I’ve also familiarized myself with the scrubbing process. I have simulated a disk failure and a disk replacement. I have also practiced expanding the pool by replacing all disks in a vdev with larger disks.

Are there any other processes/commands you guys recommend I should play with in ZFS before starting my server setup? I’m not in any super rush. It will probably be a few weeks until I’m ready to start migrating.

Thanks for reading my long post and I look forward to hearing your opinions, advice, and suggestions on what I should choose.
 
Last edited:
I stopped reading at Z77 motherboard since that means you don't have ECC RAM.
ZFS without ECC isn't really a good idea, if things go bad they can go really bad.
 
I run ZFSonLinux on an Ubuntu server, that is a KVM host for several operating systems (Ubuntu, Windows 7, FreeBSD) at the same time. I'm completely satisfied with this solution, but I use no GUI at all, neither for my VMs, nor for ZFS.

ZFSonLinux was my first contact with ZFS, I did setup a open-frame backup system on my desk to get familiar with ZFS and the toolset first. After playing around for about 1 week I decided that I know enough to not screw up completely and transfered by whole storage to the backup server and migrated the server to ZFS... but of course not without a second backup. That is basically the best suggestion I can give, do not do anything to your data without a good backup.

Regarding ECC RAM, I do not think that it is an absolute requirement. The main benefit is in my opinion that you will get notified in case of one your DIMMs becomes unreliable. So you can react before it silently screws up your data, but that is true for any systems with a high data throughput, not restricted to ZFS. I personally would however not setup any new system that deals with a lot of data without ECC RAM, because the price difference is really low.
 
Last edited:
Yeah I know I don't have ECC, but building a new server at this point is not really an option.

I didn't get the 12TB of data I currently have now overnight. I have acquired it over years and have yet to be noticeably affected by not having ECC.

Remember this is a media server. A corrupted file once in a blue moon is not a problem.

Also, I do have a complete backup with full versioning (where versions are kept forever) and checksumming. So if files did silently change on my server they would get versions backed up. Then I could always restore the uncorrupted versions or just reacquire the file altogether.

When it's time to rebuild the guts of my file server in a year or 2, then I will certainly go with ECC ram at that time.

I really appreciate the concern but I'm looking for a "what's the best I can do with what I have solution", not a "rebuild everything from scratch immediately" solution.

I have been using FlexRAID on Windows Server 2008 for awhile now. So are you really saying that since I don't have ECC ram then it's not even worth migrating to ZFS? It wouldn't be any safer than what I use now? Because that's what I'm looking for. What's the best and safest solution for my server now, with the full intention of switching out the guts for ECC in a year or 2.

Also since it's a media server, my data is write once, read many, and I never really modify files.
 
Last edited:
SirMaster said:
..
So what I have is an ivy bridge CPU on a z77 motherboard and it is the non-K edition so it does have VT-d.

What I want from my server is to run ZFS obviously, but I also require some way in which to run a decently performing Windows Server 2008 in a VM.
..I
I believe lots of people run ZFS on Solaris through ESXi, but I read a lot of information about how you shouldn’t virtualize VFS, even with VT-d. So what do you guys think about ZFS on ESXi first off. The ESXi route seems eth most appealing to me as it really gives my server the most flexibility to run what I want, but I will forgo it if it's indeed not recommended for ZFS. Remember, I'm just a home media server user for the most part. But I am still interested in data integrety.
Click to expand...

only some hints
- If you need more than one OS on a server, ESXi is the most common way
- Whenever possible, use ESXi appliances (ready to use ESXi VMs)
- If you want to virtualize a storage OS, you should never rely on virtualized storage (VMFS).
You must use pass-through (vt-d, direct hardware access to a storage controller and to disks from a VM)
to get similar performance and reliability like a barebone NAS/SAN setup.

You can read my miniHowTo about my solution http://www.napp-it.org/doc/downloads/all-in-one.pdf
and use it together wit my ready to use storage appliance (preconfigured for ESXi 5.5/5.1 with napp-it)
from http://www.napp-it.org/downloads

about ECC
its like a airbag in your car. You mostly do not need but it can save your life some day.
You do not need ECC, not for any server with any filesystem. But the premium is quite small so you should never avoid when you have the choice.
Most persons strongly suggest ECC together with ZFS but the reason is more:
You have the securest filesystem on earth but any bitflip can corrupt data like with any ancient filesystem. Try to avoid this weakness.
 
So what I have is an ivy bridge CPU on a z77 motherboard and it is the non-K edition so it does have VT-d.
Click to expand...

You may not have vt-d in your BIOS. The only certified chipsets to run vt-d is Intel Q series and workstation C2x6 chipset. Some AsRock motherboard supports vt-d, but it's like playing roulette.
 
There were also some MSI mainboards with Hxx chipsets that had an option to enable VT-d, but also came across some boards with broken or at least fragile implementations over the last years.
 
My bios shows VT-d capability:
600x450px-LL-314e4ac9_asrock-extreme4-m-vtd.jpeg


I also found this:
http://www.tinkertry.com/asrock-z77-extreme4-m/

You think it would be wise to not trust it due to stability?
 
vegaman said:
I stopped reading at Z77 motherboard since that means you don't have ECC RAM.
ZFS without ECC isn't really a good idea, if things go bad they can go really bad.
Click to expand...

Despite what I said earlier about a new server not being "in the cards" right now. And just doing the best I can with what I have at the moment. I am still strongly considering switching out server parts to gain ECC.

I spent $400 on the CPU/mobo/ram back when I got it. Perhaps If i can get $250 for the parts now I can just eat the $150 loss. Not too huge of a loss for a mistake in choosing the wrong parts a year ago or so. It happens.

I just put together this possible replacement parts list.

http://www.newegg.com/Product/Product.aspx?Item=N82E16819116910
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182831
http://www.newegg.com/Product/Product.aspx?Item=N82E16820139978 X2

In the upgrade in addition to ECC I would go up from Ivy to Haswell, gain Hyperthreading (which is nice since I offload video encoding and other CPU intensive stuff to my server) and gain an extra 8GB of ram. Sounds good to me haha.
 
Last edited:
Another option, if you have a microcenter nearby, is getting an FX-6300 combo with free asus board (which may/probably support ECC). Then grab some ecc ram & you are set.
 
Well, If I go with ESXi I wont need video output, so that X10SL7-F board would be fine.

But if I go with ZFS On Linux I would like the digital video out utilizing the CPU's iGPU.

Also, for that X10SL7-F board can the 8 SAS connectors be set to VT-d passthrough? I need 12 sata ports for the ZFS and I was going to also get an IBM M1015.
 
Robstar said:
Another option, if you have a microcenter nearby, is getting an FX-6300 combo with free asus board (which may/probably support ECC). Then grab some ecc ram & you are set.
Click to expand...

I will certainly price out an AMD system too. I have a microcenter close enough to travel to. Bought my desktop's 4770K there.
 
SirMaster said:
Well, If I go with ESXi I wont need video output, so that X10SL7-F board would be fine.

But if I go with ZFS On Linux I would like the digital video out utilizing the CPU's iGPU.

Also, for that X10SL7-F board can the 8 SAS connectors be set to VT-d passthrough? I need 12 sata ports for the ZFS and I was going to also get an IBM M1015.
Click to expand...

No problem.
The onboard SAS controller behaves like an LSI 9217 with IR firmware.
You can flash it to IT mode that is better suited for ZFS
 
_Gea said:
No problem.
The onboard SAS controller behaves like an LSI 9217 with IR firmware.
You can flash it to IT mode that is better suited for ZFS
Click to expand...

Thanks. I'm liking this more and more. Looking like ESXi is still the way I want to go.

So does uATX mean Micro-atx? Will the board you linked fit in my ATX tower?

And what other SAS card would you recommend to get me from the 8 onboard ports to the 12 ports I need. Is an IBM M1015 still the cheapest bet? I'm still trying to keep things as cheap as possible wihle still having reliability and stability.

It's getting costly lol, but I do want it done right.

http://www.newegg.com/Product/Product.aspx?Item=N82E16819116906
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182821
http://www.newegg.com/Product/Product.aspx?Item=N82E16820139978 X2

$680 total so far haha.


But just to reiterate, you don't think I have anything to worry about or that I am not sacrificing any data integrety or reliability by virtuaizing ZFS so long as I use LSI + VT-d?

I think I am worrying too much about a home media server for ~2 users haha.
 
Last edited:
No problem, uATX fits perfectly in any case where you can use ATX.

And you do not virtualize ZFS. Your storage access is physical.
You only virtualize the storage OS and this is as stable as ESXi,
not perfect like any software but the nr 1 in professional environments.

I do this for years - since I brought up the All-IN-ONE idea
 
Seriously I think it's waaay too overkill for a simple media server.

I don't understand why you would like to have such a file system only for media. As I can understand from ZFS, it's a storage file system for critical data (and mainly designed for storage server!)
 
Basically, all data you store is critical in one way or another. Otherwise, why store it? Personally, I don't understand how people can not care about corrupted files in the long run. Watching a movie and encountering a corruption that can spread over several frames visibly doesn't bother you one bit?
 
I tend to agree. In some ways it's probably overkill. But building and tinkering with computers is my hobby so I don't mind the time spent. Especially when it is for a good purpose of a reliable data storage system.
 
TCM said:
Basically, all data you store is critical in one way or another. Otherwise, why store it?
Click to expand...

It is not hard to understand. A file ripped from mass media (movies, music, etc.) is simple to replace in the very unlikely event that there are enough random bit flips to be noticeable. Alternatively, if you are using SnapRAID or FlexRAID, which are much better choices for a media fileserver than RAIDZx anyway, you can fix or restore the file in question from the redundant storage (parity).
 
I'm having a hard time imagining how data can corrupt by itself ?

I mean ... The data is written on the drive ... it's written. How it could corrupt by itself... ? EXT4 and other file systems makes a checksum once the file is written to insure it is written correctly (files are not copied without insurance it is well written on disk I imagine)

So how a file could corrupt?

I do care about the integrity of my files. But seriously doubt that a file can corrupt with time.

In 15 years in the domain of computing, I still carry very old data that has never been corrupted ... I just want to know how a file could corrupt without, by example, an application crash when saving the file (the only corruption I got in the last decade comes from app crash while saving and yes, I know, ZFS prevents that problem).
 
Last edited:
Ghost26 said:
I'm having a hard time imagining how data can corrupt by itself ?

I mean ... The data is written on the drive ... it's written. How it could corrupt by itself... ? EXT4 and other file systems makes a checksum once the file is written to insure it is written correctly (files are not copied without insurance it is well written on disk!)
Click to expand...

That is not true. ext4 does not process data checksums (only metadata). The only commonly used filesystems that process checksums for data are ZFS and btrfs. However, if you are not using ZFS or btrfs, you can always do your own checksums on your data, although that does add a failure point between the time the data is written to disk and when it is read back so you can compute your own checksums.

As for how data on HDD could become corrupt, it is sometimes called "silent errors" or "bit rot". There is a very small chance that the data read from an HDD sector is different from the data written. The bit error rate is typically around 10^-14 or 10^-15. This is after (including) the built-in ECC that HDDs use -- the raw HDD BER is much higher than 10^-14 but it is mitigated, if not completely eliminated, by built-in ECC.
 
That is not true. ext4 does not process data checksums (only metadata). The only commonly used filesystems that process checksums for data are ZFS and btrfs.
Click to expand...

I'm not a file system expert, but there must be a way for the system to insure the data is well written ???? Not necessary a checksum, but a verification or something like that ? no ?

It makes a couple of weeks I read this forum because I'm looking at a storage solution. Everybody seems to literally venerate ZFS ( ;) ), but I still can't think of a moment I lost a file because it was corrupted mysteriously.
 
Ghost26 said:
I'm not a file system expert, but there must be a way for the system to insure the data is well written ????
Click to expand...

Sure, it is called a data checksum (or hash). As I said, the only commonly used filesystems that do data checksums are btrfs and ZFS.
 
So you're telling me that NTFS, EXT4, FAT32, whatever widely used file system does NOT ensure the data that the HDD writes on it's platter ?! By any way ???
 
Ghost26 said:
So you're telling me that NTFS, EXT4, FAT32, whatever widely used file system does NOT ensure the data that the HDD writes on it's platter ?! By any way ???
Click to expand...

Is your reading comprehension impaired by drugs or alcohol? :rolleyes:
 
Lol !!!! :p

No, seriously I'm totally awake, but I only have difficulty to conceive this ! How a file system could not verify what is written on the drive ! Lol ! I'm totally confused about this reality.

Now I better understand why everybody on this forum only swear by ZFS ... !
 
Ghost, think about this a bit. Unless you have some kind of end to end checksum system (like zfs), the only way to verify things would be to read back every block you write (synchronously), which would absolutely vaporize performance. Even then as joecomp said, the disk error detection isn't perfect...
 
Ghost26 said:
Yeah you're right. Absolutely logic when thinking of it.
Click to expand...

Of course, you should keep in mind that the magnitude of the risk is exceedingly small. A BER of 10^-14 means that if you write 12.5TB of data to HDDs and then read that 12.5TB of data back, on average there will be 1 bit flip in all of that 12.5TB of data.

And to put that in perspective, a blu-ray movie rip probably averages 25GB (less if transcoded). So that is 500 blu-ray movies, nearly 1000 hours of movie watching. During that nearly 1000 hours of movie watching, where each 1080p frame, 24 times per second, has about 2 million pixels, and each pixel has 24-bits of data. On average, then, 1/24 of one pixel will be off for 41.7 milliseconds out of nearly 1000 hours of displaying frames with about 2 millions pixels per frame at 24 frames per second. (Okay, that is a bit of an underestimate, since the stored blu-ray data is highly compressed, a one-bit error could result in multiple pixel errors for several frames, but even so, we are talking a small block of pixels for well under 1 second over nearly 1000 hours of movies).

Now, there are other possible links in the data chain that could introduce additional bit errors, but unless your data is generated on the same machine with a data-checksummed filesystem, no amount of filesystem data checksumming can keep your data free of bit errors.

That is why a checksum filesystem is an excellent choice for many industrial applications where data is generated on the same machines that run the checksummed-filesystem -- this is what people mean by end-to-end protection.

But for a typical media fileserver, the files are frequently generated outside of the machine running the filesystem, and over the lifetime of the data, the files may be transferred to other filesystems without checksum protection. In those cases, the data is better protected by checksums maintained independently of the filesystem -- for example, by SnapRAID or FlexRAID, or by a manual or semi-automated program which checksums your data and maintains the checksums in file(s) that can be copied to other filesystems.

Actually, the OP has stated that he has a system something like the last one I mentioned above. Since his data already is protected by checksums, he would actually be gaining double protection by moving to a checksummed filesystem. Which is certainly overkill for a media fileserver, but some people just do that sort of thing for fun, I suppose.
 
Last edited:
Yeah. I do keep a somewhat manual checksum of my files because they don't change too often and I run the compare when I remember to heh. It works out well with a versioning backup system like crashplan.

I'm looking at moving to a much more automated and more robust system. Plus I like to play with new stuff. I don't think I'll keep doing extra checksums after I move to zfs.
 
My 2 cents: I run ZFS on Linux on 2 servers that act as KVM hosts for a total of 25 production VMs and I am very happy with the results, echoing what omniscience said.

Versus the ESXi All-in-Ones this setup has a number of advantages:
- No VT-d requirements nor quirks
- No iSCSI or NFS involved thus better performance and one source of issues less
- Overall reduced complexity and no proprietary software required

IMHO what the ESXi AIO setup has going for it is:
- Windows guests tend to run better on VMware than KVM
- Setup is easier, with a good GUI, especially the networking part (a bit tricky with KVM)

I'd also advise against ZVOLs. With (sparse) RAW files for VM disks on a separate ZFS dataset each you avoid the ZVOL overhead (especially bad with RAIDZ2) and the handling is generally easier (it's just a file!).

What might be an issue for you is that current napp-it is not compatible with ZoL.
 
Nice to see someone else running ZoL and liking it. I still feel like that is a big option for me.

I spent last night installing both ESXi and the OmniOS/napp-it VM and playing around with them.

I did get it to work except I couldn't yet figure out how to share folders on the zpool via SMB.


I think going with native Linux might be the best for me. Technically I can do everything I want in Linux alone. ZFS, SMB, and then a whole slew of servers and services like SABnzbd+, sickbeard, subsonic, teamspeak 3, uTorrent, crashplan, and plex and more. Every service I use currently has a Linux version.

Although I'm far more comfortable with my Windows Server 2008 I think I may try to convert my physical 2008 Server into a VM and run it from VMWare Workstation on Linux rather than use KVM.

Then perhaps I can slowly wean myself off Windows as I move one service at a time from Windows to Linux. Becasue I really don't want to wrestle with everything all at once. I already practiced setting up ZFS and SMB on Linux (Debian 7) and I am happy with the results I got in a VM.

I see 3 reasons for the ESXi route for me.

1. Allows me to run the "real" solaris based zfs.

I'm not sure if this should even be a point. Should I even worry about the stability and more importantly the reliability of the current version of ZoL? I can tolerate kernel panics and such as this is far from a production environment. As long as my data is safe at the end of the day that's all I care about.

2. Allows me to use napp-it to manage the ZFS.

napp-it at first use seems nice. But I don't see myself using many ZFS features like snapshots or anything. I just want my vdevs and zpool and the ability to checkup on it with scrub and such and then replacing disks as needed.

3. Allows me to run Windows Server on the same machine without running VMWare Workstation (or VirtualBox, or KVM).

Could I even expect much better performance for a Windows Server guest running on ESXi compared to running on a Linux host through Workstation? If not then I don't mind just running VMWare Workstation as I am very used to the product.

What do you think? Are those 3 points compelling enough for me to add the extra complexity and more unfamiliar environment of ESXi + OmniOS?
 
zrav said:
I'd also advise against ZVOLs. With (sparse) RAW files for VM disks on a separate ZFS dataset each you avoid the ZVOL overhead (especially bad with RAIDZ2) and the handling is generally easier (it's just a file!).
Click to expand...

Can you detail your stance on zvols a bit more? I use zvols as backends for my VMs. I set the block size to 32 KiB (a complete stripe on an 11 disk RAIDZ3 with ashift=12) and set the logical block size for KVM to 32 KiB. Linux VMs will automatically set the ext4 RAID stride to account for the block size and I install Windows VMs on a 32 KiB cluster size NTFS. I assume that should alleviate any overhead and yield best performance.

I think zvols are in better because they support direct io, while files on ZFS do not. I'm unsure how ZFS handles direct io, for Linux filesystems it is supposed to bypass the page cache. The best would be if direct io on ZFS would not pollute the ARC with data that will get cached in the VMs cache anyway.
 
SirMaster said:
Nice to see someone else running ZoL and liking it. I still feel like that is a big option for me.

I spent last night installing both ESXi and the OmniOS/napp-it VM and playing around with them.

I did get it to work except I couldn't yet figure out how to share folders on the zpool via SMB.
Click to expand...

Quite easy
Goto menu zfs folders and click on "off" under SMB and the desired filesystem.
The SMB service will start automatically and the filesystem is shared with a
default ACL of everyone@=modify and root=full access
 
SirMaster said:
1. Allows me to run the "real" solaris based zfs.
Click to expand...
That's something to consider. Subjectively I'd call ZoL beta quality. Still, the 4 servers I run ZoL on have been rock solid except one a non-critical file backup server: Producing heavy I/O load from about 20 concurrent rsync processes causes occasional soft lockups.
2. Allows me to use napp-it to manage the ZFS.
Click to expand...
Managing ZFS is pretty easy IMHO, I never missed it. Still, I hope that with increasing popularity Gea will support ZoL in napp-it.
3. Allows me to run Windows Server on the same machine without running VMWare Workstation (or VirtualBox, or KVM).

Could I even expect much better performance for a Windows Server guest running on ESXi compared to running on a Linux host through Workstation? If not then I don't mind just running VMWare Workstation as I am very used to the product.
Click to expand...
I suspect the VM performance is about the same, as it's essentially the same technology, though ESXi has less host OS overhead.

omniscence said:
Can you detail your stance on zvols a bit more?
Click to expand...
https://github.com/zfsonlinux/zfs/issues/548 this affects all ZFS implementations but like you said, increasing block size reduces the overhead.
https://github.com/zfsonlinux/zfs/issues/824 this is ZoL specific. And there's more.
And finally I find managing file containers instead of volumes easier.
 
Thanks. The space issue is nothing exceptional, if you use 8 KiB blocks on a pool with 16 KiB stripe size like in that issue, you can't complain about waste of space. The optimal size would be 16 KiB block size or even larger if compression is used.
 
You must log in or register to reply here.
Back
Top