![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
-
Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.
3 or 4 HDDs in RaidZ1?
- Thread starter hotzen
- Start date
128 divided by (total_nr_disks - parity_disks); so do not count the parity disks.
That means that for 4K disks the following combinations are optimal:
RAID-Z: 2, 3, 5, 9 or 17 disks
RAID-Z2: 3, 4, 6, 10 or 18 disks (anything under 6 disks produces low results for another reason, though)
As you can see the RAID-Z2 is always one disk more; since we don't count parity disks we get the same number of data disks: 1, 2, 4, 8, 16. Those values mean you get a clean fraction when doing 128 / n where n is the number of data disks.
For sub-optimal configurations, you would want to use the sectorsize override which often can boost write performance decently. For optimal configurations this may not be required.
Beware that if you create a pool of 512-byte sector disks (i.e. without sectorsize override) then that pool CANNOT be expanded with real "native" 4K sector disks coming in the future. If you do the sectorsize override now, you can add native 4K sector disks later. Just one more thing to consider. ;-)
That means that for 4K disks the following combinations are optimal:
RAID-Z: 2, 3, 5, 9 or 17 disks
RAID-Z2: 3, 4, 6, 10 or 18 disks (anything under 6 disks produces low results for another reason, though)
As you can see the RAID-Z2 is always one disk more; since we don't count parity disks we get the same number of data disks: 1, 2, 4, 8, 16. Those values mean you get a clean fraction when doing 128 / n where n is the number of data disks.
For sub-optimal configurations, you would want to use the sectorsize override which often can boost write performance decently. For optimal configurations this may not be required.
Beware that if you create a pool of 512-byte sector disks (i.e. without sectorsize override) then that pool CANNOT be expanded with real "native" 4K sector disks coming in the future. If you do the sectorsize override now, you can add native 4K sector disks later. Just one more thing to consider. ;-)
A
Huh, I was not aware of this - so when 4k disks which report their 4k sector size properly become available (i.e. not the 512e stuff) you still won't be able to add a vdev of them to a storage pool which has vdev's of standard 512 disks? Ashift is a per vdev setting, not a per pool setting? Or is something else causing the problem?
I thought 512b drives had a similar "optimal" configuration not dissimilar of the 128 / N suggested here.
The ZFS Tuning wiki doesn't seem to make the distinction on of which drive (sector wise) fit into the "optimal" config that I can see.
Yeah sub, clarify this.
Do you mean you can't add the 4k native disks to an existing vdev (without a performance hit, I thought its still "possible")?
I mean that's a bummer too since eventually the existing vdevs we have are all going to have to undergo disk replacements. I don't think most home users, or businesses for that matter, can afford to build another pool with the capacity (unless 6TB drives become cheap... lol) and move it over. I guess if they start allowing vdev removal it could help.
I'm guessing Oracle will have to compensate somehow, like I said, we are all eventually going to have this scenario. Whether or not they release the binaries so FreeBSD, ON, and the likes can take advantage... who knows.
Lol, have they released the binaries for the current version of zfs in S11E?
I think the ashift-value, which is enforced by the 4k-override is indeed on a per-pool basis.
And I think you can say that oracle has indeed released the zfs *binaries* by releasing SE11. The source-release seems to be planned for the Solaris 11 (non-express) release. That's what OpenIndiana/Illumos is waiting for...
And I think you can say that oracle has indeed released the zfs *binaries* by releasing SE11. The source-release seems to be planned for the Solaris 11 (non-express) release. That's what OpenIndiana/Illumos is waiting for...
The ashift value is stored in each vdev, but the top-level vdev ashift value will persist for all the pool. So if you start with 512-byte sector disks, you cannot add 4K sector disks later. Also see:
http://opensolaris.org/jive/thread.jspa?threadID=135641
I would like to stress that currently, there are no disks sold which report as having 4K sectors. All 4K disks currently sold use sector emulation and report 512 bytes to the operating system, to remain compatible with older OS like Windows XP and many RAID controllers as well. This limitation only applies to future disks which report as having 4K sectors instead of the current generation which still reports 512-byte sectors even though they physically have 4K sectors.
On FreeBSD/ZFSguru, you can use the sectorsize override to force your pool to use ashift=12 (optimal for 4K disks). This would also allow adding of vdevs with native 4K disks in the future.
On FreeBSD platform, there could be developed a GEOM class that emulates sector size, then you can add a native 4K disk to an existing normal 512-byte sector pool (ashift=9).
And yes, Oracle will release new ZFS versions as CDDL sourcecode, but always with a delay as i understood. Fortunately IllumOS and FreeBSD can work together to improve ZFS regardless of the incompatible version list; i.e. make improvements that remain compatible with the on-disk format.
http://opensolaris.org/jive/thread.jspa?threadID=135641
I would like to stress that currently, there are no disks sold which report as having 4K sectors. All 4K disks currently sold use sector emulation and report 512 bytes to the operating system, to remain compatible with older OS like Windows XP and many RAID controllers as well. This limitation only applies to future disks which report as having 4K sectors instead of the current generation which still reports 512-byte sectors even though they physically have 4K sectors.
On FreeBSD/ZFSguru, you can use the sectorsize override to force your pool to use ashift=12 (optimal for 4K disks). This would also allow adding of vdevs with native 4K disks in the future.
On FreeBSD platform, there could be developed a GEOM class that emulates sector size, then you can add a native 4K disk to an existing normal 512-byte sector pool (ashift=9).
And yes, Oracle will release new ZFS versions as CDDL sourcecode, but always with a delay as i understood. Fortunately IllumOS and FreeBSD can work together to improve ZFS regardless of the incompatible version list; i.e. make improvements that remain compatible with the on-disk format.
gremlin190
n00b
- Joined
- Feb 25, 2011
- Messages
- 56
Is this true or not? i planned on making two pools of 4 hd with 512k sector drives. Is this not the best choice?
Hmm, thanks for the link submesa. Looks like on solaris based platforms, unless you want to try your luck with the hacked ashift binary, you are currently S.O.L. when disks which report 4k sectors become available (can't really expand the pool with vdev's of those).
Alternatively it doesn't look like there is a downside to having an ashift of 12 on the pool even if all your vdev's are ashift's of 9? (4k on the pool, 512 on the vdev's). Would this then be a suggested route for someone going with a FreeBSD based distro? If you start your pool with an ashift of 12 you should be able to add whatever vdev's you want, and the ashift on the vdev itself should handle appropriate alignment? (If I'm understanding this correctly)
Alternatively it doesn't look like there is a downside to having an ashift of 12 on the pool even if all your vdev's are ashift's of 9? (4k on the pool, 512 on the vdev's). Would this then be a suggested route for someone going with a FreeBSD based distro? If you start your pool with an ashift of 12 you should be able to add whatever vdev's you want, and the ashift on the vdev itself should handle appropriate alignment? (If I'm understanding this correctly)
You didn't say RAIDZ1 or 2?
From what the Wiki states, anything between 3 and 9 per vdev is acceptable.
That advice (not too many devices in one pool) is unrelated to anything with 'sectors' - it's just general advice that many devices in a single RAID-Z (1/2/3) would have a negative impact on latency. Since the slowest disk counts, it is quite likely a 20-disk RAID-Z (1/2/3) would have at least one disk taking more time than usual; a dud. That could impact your performance quite a bit.
Also your disks would be given lower block size I/O if you have many devices in a RAID-Z (1/2/3) - with a 2-disk RAID-Z you have 1 data disk and thus 128K per disk. with 2 max 64K, with 4 max 32K, etc.
Generally i would not pay too much attention to the wiki; some stuff is also outdated due to changes in ZFS and FreeBSD.
Also your disks would be given lower block size I/O if you have many devices in a RAID-Z (1/2/3) - with a 2-disk RAID-Z you have 1 data disk and thus 128K per disk. with 2 max 64K, with 4 max 32K, etc.
Generally i would not pay too much attention to the wiki; some stuff is also outdated due to changes in ZFS and FreeBSD.
Depends what level. I think RAID-Z should not be considered very secure regardless of how many disks. For real protection a RAID-Z2 would be superior. That could survive one complete failure + BER - which is decent protection for modern storage. But for more than 10 disks, i would consider the use of RAID-Z3 instead if you really want all in one vdev.
The preferred option is to have multiple vdevs, especially since random I/O performance scales with vdevs; each vdev has about the same random I/O as a single disk! This is something you can solve by using a SLOG + L2ARC configuration though, preferably using modern SSD with supercapacitor.
So i would not heed to much attention to 'do not put more than x disks in a vdev' - it's just general advice that smaller vdevs often are better.
The preferred option is to have multiple vdevs, especially since random I/O performance scales with vdevs; each vdev has about the same random I/O as a single disk! This is something you can solve by using a SLOG + L2ARC configuration though, preferably using modern SSD with supercapacitor.
So i would not heed to much attention to 'do not put more than x disks in a vdev' - it's just general advice that smaller vdevs often are better.
Ok you're contradicting yourself. Above you say "optimal" config for 4K disks is:
But then you're saying that vdev size is not factored by sector size. So which is it?
Then you also go on to say that generally smaller vdev groups are ideal but then suggest up to 18 disks (16 disks + 2 parity) in a vdev is perfectly acceptable for RAID-Z2.
Right now my pool at work is 14 disks. 12 disks in the pool (so 10 for data, 2 for parity) with 2 hot spares. According you that's acceptable for 4K sector drives but these are 512byte sector drives.
There is separate advice offered, each advice has its own technical issue.
The optimal configurations for 4K disks is due to the SECTOR SIZE. Not ANYTHING else. That's why a 17-disk RAID-Z would be recommended from a viewpoint of the sector size. But from a viewpoint of protection, a 17-disk RAID-Z is bad.
The advice 'do not put too many disks in a single vdev' has NOTHING TO DO with the sectorsize. It is general advice that the more disks in a vdev, the lower performance that vdev could have. A 17-disk RAID-Z would cause the slowest disk of all 17 to determine the performance level for that vdev. So the more disks you have, the higher chance that at least one disk has a much higher latency than the others. If so, that higher latency would apply to all HDDs, not just the single HDD. Such is the nature of RAID-Z where full stripe blocks are compressed onto the entire vdev, much like RAID3 does.
I hope you understand about the two SEPARATE advice offered and completely different technical background?
I also did not suggest 18 disks, i only stated the configurations that for the SECTOR SIZE were optimal. So 17 disks in a RAID-Z would be good from a point of 4K sectorsize and ZFS allocation. But if would be bad when considering the low redundancy level and the many disks in one vdev. But that was unrelated to the sector sizes; i wanted you guys to understand how this works. Seems i done the opposite and just created more confusion, sorry for that.
If you want a final list of optimal configurations, use this one:
RAID-Z: 5 disks
RAID-Z2: 6 disks or 10 disks
That has all advices combined, and as you can see the RAID-Z2 would lack configurations with 3 and 4 disks as well.
Also from the point of having too many disks in a single vdev, a 17-disk RAID-Z would cause the slowest disk in that array to dictate the performance; which could be low with many disks in a single vdev, due to the change
The optimal configurations for 4K disks is due to the SECTOR SIZE. Not ANYTHING else. That's why a 17-disk RAID-Z would be recommended from a viewpoint of the sector size. But from a viewpoint of protection, a 17-disk RAID-Z is bad.
The advice 'do not put too many disks in a single vdev' has NOTHING TO DO with the sectorsize. It is general advice that the more disks in a vdev, the lower performance that vdev could have. A 17-disk RAID-Z would cause the slowest disk of all 17 to determine the performance level for that vdev. So the more disks you have, the higher chance that at least one disk has a much higher latency than the others. If so, that higher latency would apply to all HDDs, not just the single HDD. Such is the nature of RAID-Z where full stripe blocks are compressed onto the entire vdev, much like RAID3 does.
I hope you understand about the two SEPARATE advice offered and completely different technical background?
I also did not suggest 18 disks, i only stated the configurations that for the SECTOR SIZE were optimal. So 17 disks in a RAID-Z would be good from a point of 4K sectorsize and ZFS allocation. But if would be bad when considering the low redundancy level and the many disks in one vdev. But that was unrelated to the sector sizes; i wanted you guys to understand how this works. Seems i done the opposite and just created more confusion, sorry for that.
If you want a final list of optimal configurations, use this one:
RAID-Z: 5 disks
RAID-Z2: 6 disks or 10 disks
That has all advices combined, and as you can see the RAID-Z2 would lack configurations with 3 and 4 disks as well.
Also from the point of having too many disks in a single vdev, a 17-disk RAID-Z would cause the slowest disk in that array to dictate the performance; which could be low with many disks in a single vdev, due to the change