ZFS, Largeblocks, Recordsize, and VDEV Drive Counts

[Zarathustra[H]]

Hey all,

So, I've been in a middle of a project of one by one replacing and resilvering my old 4TB drives in my pool with 10TB drives in order to grow it in capacity.

My plan is to use my old drives, 12x 4TB WD Reds in a a spare backup server and use them for backups.

In the old days, prior to largeblocks, sub.mesa wrote in some detail on these forums (in a seven year old post I can no longer seem to find) about the optimal number of drives in a vdev depending on type. It went something like this:

The theory is as follows (in laymans terms):

In order to be compatible with older OS:es, 4k sector drives (needed for the modern large drive sizes) fool operating systems into thinking they are 512byte sector size drives.

Whenever an OS makes a request that is not aligned with 4k, the drive is forced into 512byte emulation mode, which significantly slows it down.

Since ZFS uses 128kb chunks, he came up with the following formula:

128KiB / (number of drives - parity drives).

If the resulting number is divisible by 4, then you shouldn't have a problem. If it is NOT divisible by 4, then you might.

If we follow these calculations, optimal configurations would be as follows:

RaidZ: 3, 5 or 9 drives. (17 drives also winds up being divisible by 4, but this is above 12 recommended as max by ZFS documentation)

RaidZ2: 4, 6 or 10 drives. (and 18, which is above 12, as above and not recommended)

Raidz3: 7 and 11 drives (and 19 which is above 12, as above and not recommended)

All this time I have been running with two 6 drive RAIDz2 vdevs for this purpose, as 6 drives is an optimal number of drives according tot he theory above.

In order to get the most out of my old drives, which are smaller than my new ones, I was considering using a single RAIDz3 vdev instead of two RAIDz2 vdevs. I can do this because I don't wind up caring at all about pool performance, since all it has to, being a remote backup, is keep up with writes at WAN speed, and pretty much anything can do that. I chose RAIDz3 as these are now older drives, and I want to have a little bit more failure tolerance.

According to the calculations above, 11 disks is an ideal size for RAIDz3, but then I'd have a spare drive. If I could - instead - set them all up as a 12 disk RAIDz3, I'd gain an extra 4TB I didn't otherwise have.

The question is, would this non-ideal vdev size be a problem?

I had read elsewhere that with the advent of the "largeblocks" feature in ZFS, these VDEV disk size guidelines are no longer relevant. Is that really the case?

I'm also confused as to what largeblocks really does. I've read it increases blocksize up to 1mb, but if I look at my current pool it both has largeblocks=on AND has a recordsize of 128k?

root@proxmox:~# zpool get all |grep -i blocks
rpool    feature@large_blocks                          enabled                                       local
zfshome  feature@large_blocks                          enabled                                       local

root@proxmox:~# zfs get all |grep -i recordsize
rpool                                              recordsize            128K                        default
zfshome                                            recordsize            128K                        default

Do you both have to set largeblocks=on and THEN increase recordsize?

Appreciate any info!

 

[_Gea]

Just set recordsize to a value > 128k like 512k or 1M.
This activates large_block support

If you create a vdev with a disknumber that does not meet a golden number (power of 2 for datadisks) then you will waste some space (can be up to 10%). You can reduce that when you increase blocksize. For a backupsystem this is perfectly ok. If this would be a database or VM server this may affect performance negatively. Such systems would more prefer a recorsize of say 32k


see
http://illumos.org/man/5/zpool-features

large_blocks


The large_block feature allows the record size on a dataset to be set
larger than 128KB.

This feature becomes active once a recordsize property has been set
larger than 128KB, and will return to being enabled once all
filesystems that have ever had their recordsize larger than 128KB are
destroyed.