ZFS or UFS on SnapRAID components?

[TCM]

Hi,

I'm really intrigued by SnapRAID's features compared to ZFS for large static files: only one drive spinning, minimized data loss beyond the failure of all parity drives, can add one drive at a time, etc.

I'm using FreeBSD in a VM with 16G RAM and a ZFS RAID1 pool for some other VMs so far and I plan to use SnapRAID in the foreseeable future. ./configure && make gave me a single binary without any warning or errors, excellent first impression.

I'm just not sure about what filesystem to use for SnapRAID disks, ZFS or UFS. I know SnapRAID has its own checksums. However, those get checked only on a manual basis whereas ZFS' checksums are real-time. I know that checksums are just an indicator and ZFS can't "heal" in this scenario. Still, real-time checksums would alert me as soon as possible compared to SnapRAID's manual process. Also, individual ZFS disks could still use compression and deduplication which UFS lacks. However, since this will be a media library, those are not really strong points.

What I'm worried about is ZFS' memory requirements, specifically a separate ARC for each pool, since I'd be using one pool per disk.

What to do?

 

[JoeComp]

It really does not matter what you use for your data drives. I would not worry about it. Just use whatever is most convenient.

For your parity drive(s), however, you need to be a little more careful. XFS, ext4, and NTFS are known to work well for parity drives. ext3 and JFS are known not to work well for parity. As far as I know, no one has tested ZFS or UFS as a parity drive filesystem. The issue is whether fallocate() works efficiently or not with the filesystem when a very large (sparse) file is allocated.

 

[TCM]

As far as I know, no one has tested ZFS or UFS as a parity drive filesystem. The issue is whether fallocate() works efficiently or not with the filesystem when a very large (sparse) file is allocated.

fallocate() is Linux-specific.

I had no trouble or warnings when compiling, so I assume it uses something else on FreeBSD, thus making filesystem support for a specific syscall a moot point?

 

[JoeComp]

I had no trouble or warnings when compiling, so I assume it uses something else on FreeBSD, thus making filesystem support for a specific syscall a moot point?

It is not a compile time issue.

 

[TCM]

It is not a compile time issue.

More like a configure-time issue. But I already figured out that there is posix_fallocate() which it uses. No idea though how filesystems interact with it.

 

[JoeComp]

More like a configure-time issue. But I already figured out that there is posix_fallocate() which it uses. No idea though how filesystems interact with it.

The issue shows up at runtime. You need to test it. If it takes a long time (hours) just to create the parity file (before it even begins syncing), then the filesystem is not good for a parity drive.

 

[Rectal Prolapse]

This is probably relevant. Maybe.

http://www.mentby.com/sao-kiselkov/io-errors-with-wal-on-zfs.html

 

[TCM]

Something to keep in mind, thanks.

 

[UhClem]

For your parity drive(s), however, you need to be a little more careful. XFS, ext4, and NTFS are known to work well for parity drives. ext3 and JFS are known not to work well for parity.

ext3 works perfectly well.

 

[TCM]

ext3 works perfectly well. [JoeComp is welcome to argue, if he feels like embarassing himself again. ]

Irrelevant either way because Linux is not the topic

 

[Aesma]

For a media library I wouldn't worry about memory.

 

[TCM]

The media SnapRAID "pool" is only one in the system. There's another pool hosting VMs. Can I somehow take away ARC from pools selectively, give it all to the VM pool and leave only ~1G for _all_ the SnapRAID pools?

 

[danswartz]

I don't believe so.

 

[Aesma]

You said "I'm using FreeBSD in a VM with 16G RAM and a ZFS RAID1 pool for some other VMs".

Unless you're doing VMs on top of a VM then you already have more memory for the other VMs. If you need you can give less to the FreeBSD part.

 

[TCM]

The FreeBSD VM is my NAS. It hosts one pool back to ESXi for all other VMs and is supposed to host the media SnapRAID, all in one system.

 

[danswartz]

Here's a thought: ARC or L2ARC for media is silly, unless you have many gigabytes of RAM. Just disable caching for the media pool, then it will all go to the VM pool. set primarycache=none and secondarycache=none for the media pool.

 

[JoeComp]

ext3 works perfectly well.

Certainly not "perfectly".

If you read the FAQ, you will see that the author of SnapRAID recommends against ext3 unless the parity file is the only file on the filesystem:

http://snapraid.sourceforge.net/faq.html

ext3: No. It doesn't support the fallocate() command needed to allocated the parity files. You can anyway use it if you limit the use of the parity disk to contain only the parity file.

 

[UhClem]

More like a configure-time issue. But I already figured out that there is posix_fallocate() which it uses. No idea though how filesystems interact with it.

(It's ironic, but ...) The above is absolutely irrelevant.
SnapRAID does not use posix_fallocate(). You think it does, because you saw it in the configure file(s), but that is obsolete (as of v1.8--about 1.5 years ago) and should have been removed.
[ in the sources, posix_fallocate() only remains in a couple of (also obsolete) comments. ]

Do take note of what JoeComp warned you of, with respect to the possibility of a pause/delay/hang following the "syncing ..." message. It is a run-time thing, but was eliminated for all then-tested filesystems. Since you could be using an untested-by-SnapRAID filesystem, ...

 

[UhClem]

Certainly not "perfectly".

ext3 parity file works perfectly for me.

2 things to note:
1) I am well aware of the (original) problem with ext2/ext3 parity file behavior. After all, I was the one who discovered, reported, and provided the fix for it. [link] (I presume that you [JoeComp] remember that )
2) Since that glitch had existed through v1.7, ext2/ext3 did not get much/any use/testing--which surprised me, and is the reason I initially thought there must have been some unusual conditions in my environment (hence, the "only in rare situatiion" comment in my thread title)

I have continued to make (heavy) use of ext3 for all my SnapRAID arrays, with no problems.

If you read the FAQ, you will see that the author of SnapRAID recommends against ext3 unless the parity file is the only file on the filesystem.

Interesting! (I have not read the FAQ, again, after the bug fix ~1.5 years ago. Else I would have inquired ... and I shall do so tomorrow.) I suspect that the caveat is based on theory, or just "playing it safe", but I am open to the possibility of some specific catastrophic scenario.

 

[brutalizer]

"I know that checksums are just an indicator and ZFS can't "heal" in this scenario"
What do you mean with this? ZFS can detect all data corruption as usual, running on single disk. ZFS can also correct and heal all data corruption on single disk - if you toggle "copies=2". This means every bit will be stored twice on the disk, effectively halving the storage capacity on the disk. You can also toggle "copies=3" if you really care about safety.

ZFS is running on Linux with great success. 4GB RAM for ZFS would be sufficient.

 

[TCM]

Yes but that would defeat the whole purpose. The redundancy and healing part is supposed to be taken over by SnapRAID, otherwise I could ditch SnapRAID entirely and just make a RAIDZ2.

I _want_ the features I listed in the first post, when I said "only one drive spinning, minimized data loss beyond the failure of all parity drives, can add one drive at a time, etc."