ZFS and 512n vs 512e vs 4kN drives

slowfox

n00b
Joined
Jan 11, 2011
Messages
5
Guys, I will try to keep it very short as I tend to get into unnecessary details.

My ultimate goal is to choose between using 512n, 512e, or pure 4kN drives for a new ZFS storage appliance that will either consist of eight 4TB drives in four mirror VDEVs, or eventually Raidz2 in order to mitigate any unrecoverable read errors that might occur during re-silvering a mirror in case of disk failure.

I understand the difference between 512n, 512e and 4kN drives and that 512e drives are basically 4k sector drives with emulation for backward compatibility for software that supports only 512b sectors. I know that with 4k drives there is more efficiency as there is eight times less gaps between sectors and eight times less markers.

Still it is quite difficult for me to make a decision on which drives to choose from as with HGST (Ultrastar 7k6000 SAS) I can choose between 512n, 512e and 4kN and with Seagate (Enterprise Capacity 3.5 SAS) I can choose between 512e and 4kN).

One of the things that bothers me given the fact that ZFS will always use the physical sector size to create VDEVs is if there would be any difference between using 512e and 4kN drives from the same model when using these in ashift=12 vdevs? My understanding here is that ZFS treats 512e and 4kN drives the same way in such vdevs and my guess is that any operations involving pure 4k reads and writes from and to 512e drive would not cause any in-drive processor usage e.g. causing performance degradation which is the case when using the drive in 512e mode. Would I be right to assume that?

I am trying to figure out if there is any reasoning behind getting pure 4kN drives that would get another 15% on top of my bill as opposed to 512e drives.

On the other side of the coin, having the option from HGST to choose 512n 4TB drives I am trying to figure out if that would be of any benefit other than optimizing overhead for certain file sizes in RaidZ setups. And I do understand that 512n drives could be little bit slower due to the higher number of operations needed to acquire sectors.

Any opinions are highly appreciated!



~Cheers~
 
Last edited:

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,071
There is only one item to care.
If you use older 512Bytes/sector disks with or without real 4k physical then ZFS will create a vdev with ashift=9 (512Byte sectorsize).
If such a disk fails, you cannot replace with a newer 4k disk

In such a case you should force ashift=12 to allow such replacements with the options
- add a 4k disk to a vdev prior creation to force ashift=12
- modify sd.conf where you can override disk parameters in Illumos
ZFS and Advanced Format disks - illumos - illumos wiki
- force ashift during creation of a vdev (ZoL)

Every newer disk is 4k and tells ZFS about its real 4k physical sector size, does not matter an emulation mode.
ZFS will then automatically use ashift=12
 

slowfox

n00b
Joined
Jan 11, 2011
Messages
5
There is only one item to care.
If you use older 512Bytes/sector disks with or without real 4k physical then ZFS will create a vdev with ashift=9 (512Byte sectorsize).
If such a disk fails, you cannot replace with a newer 4k disk

In such a case you should force ashift=12 to allow such replacements with the options
- add a 4k disk to a vdev prior creation to force ashift=12
- modify sd.conf where you can override disk parameters in Illumos
ZFS and Advanced Format disks - illumos - illumos wiki
- force ashift during creation of a vdev (ZoL)

Every newer disk is 4k and tells ZFS about its real 4k physical sector size, does not matter an emulation mode.
ZFS will then automatically use ashift=12

I understand the differences between different raid setups with ZFS and I've been using ZFS for many years already. I am also aware of the replacement limitations in vdevs created with ashift=9.

The real questions here are:

1) Is there any difference between 512e and 4kn from ZFS point of view and from drive point of view when ZFS reads/writes from/to it, that would justify 15%-20% for 4kN on top of what I would pay for 512e? Since ZFS respects the physical sector size of the drives if I use such 512e drives would ZFS do 4K only reads/writes to them?

2) If I am guaranteed I can get 512n replacements over my 5 year warranty, besides space efficiency (which would be applicable only for raidz) is there other reason I would choose 512n instead of 512e/4kN?

~Cheers~
 

SirMaster

2[H]4U
Joined
Nov 8, 2010
Messages
2,161
There is only one item to care.
If you use older 512Bytes/sector disks with or without real 4k physical then ZFS will create a vdev with ashift=9 (512Byte sectorsize).
If such a disk fails, you cannot replace with a newer 4k disk

Hmm, this is not my experience. 4K native disks works fine with ashift=9. They are just lower performance.

If you plan to ever replace a disk with a 4K disk in the future it's best to just create the pool ashift=12 in the first place as 512b disks perform fine with ashift=12.
 

slowfox

n00b
Joined
Jan 11, 2011
Messages
5
Hmm, this is not my experience. 4K native disks works fine with ashift=9. They are just lower performance.

If you plan to ever replace a disk with a 4K disk in the future it's best to just create the pool ashift=12 in the first place as 512b disks perform fine with ashift=12.

Why would a 4k native drive be a lower performance? I am just wondering, there are 8 times less operations involved in finding a sector. And there is less overhead from gaps between sectors. That generally means that heads are much faster in acquiring these sectors. Given that how would you justify stating that 4k native drives have lower performance?
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,071
As ZFS can use any of the disks, you must look at the implications

512N native
512B locical and physical sectorsize

Advantages
- regarding available capacity with small files or Raid-Z and number of disks per vdev that is not a power of 2

Disadavantage
- all newer high capacity disks are 4k physical sectorsize, so you buy old stuff
- smaller disks ex 10/15k rpm models are obsolet as SSDs offer 10x the iops
- you cannot replace a disk with a newer 4k disk unless you do not force ashift=12

So no, I would not buy


first gen of 4k disks
512B locical and reported physical sectorsize but in reality 4k

Advantage
- work with every OS

Disadvantage
- reduced write performance
- you cannot replace a disk with a newer 4k disk unless you do not force ashift=12

So no, do not buy


512e, 2nd gen of 4k disks
512B locical but reports correct 4k physical sectorsize

Advantage
- can work with every OS (reduced write performance in 512B emulation)
- ZFS use 4k with ashift=12

So no problem


4k native
4k locical and physical sectorsize

Works only with newer OS

So no problem but with ZFS no advantage regarding sectorsize over 512E
 

slowfox

n00b
Joined
Jan 11, 2011
Messages
5
As ZFS can use any of the disks, you must look at the implications

512N native
512B locical and physical sectorsize

Advantages
- regarding available capacity with small files or Raid-Z and number of disks per vdev that is not a power of 2

Disadavantage
- all newer high capacity disks are 4k physical sectorsize, so you buy old stuff
- smaller disks ex 10/15k rpm models are obsolet as SSDs offer 10x the iops
- you cannot replace a disk with a newer 4k disk unless you do not force ashift=12

So no, I would not buy


first gen of 4k disks
512B locical and reported physical sectorsize but in reality 4k

Advantage
- work with every OS

Disadvantage
- reduced write performance
- you cannot replace a disk with a newer 4k disk unless you do not force ashift=12

So no, do not buy


512e, 2nd gen of 4k disks
512B locical but reports correct 4k physical sectorsize

Advantage
- can work with every OS (reduced write performance in 512B emulation)
- ZFS use 4k with ashift=12

So no problem


4k native
4k locical and physical sectorsize

Works only with newer OS

So no problem but with ZFS no advantage regarding sectorsize over 512E



Well that's a pretty good answer to my question :) Very much appreciated Gea for laying it down that way!

I was seeing things the same, I guess I just wanted another confirmation that ZFS would utilize 4k in 512e drives ;)

Thank you!
 

SirMaster

2[H]4U
Joined
Nov 8, 2010
Messages
2,161
Why would a 4k native drive be a lower performance? I am just wondering, there are 8 times less operations involved in finding a sector. And there is less overhead from gaps between sectors. That generally means that heads are much faster in acquiring these sectors. Given that how would you justify stating that 4k native drives have lower performance?

I said 4K drives are slower when using ashift=9. That means ZFS is writing 512b sectors to them which is where the performance problem comes up since the disk has to use it's 512b emulation layer.

If you use ashift=12 on the 4K drive then performance will be great as it should be.
 

slowfox

n00b
Joined
Jan 11, 2011
Messages
5
I said 4K drives are slower when using ashift=9. That means ZFS is writing 512b sectors to them which is where the performance problem comes up since the disk has to use it's 512b emulation layer.
If you use ashift=12 on the 4K drive then performance will be great as it should be.

This is not exactly what you said as we can see here:

Hmm, this is not my experience. 4K native disks works fine with ashift=9. They are just lower performance.
If you plan to ever replace a disk with a 4K disk in the future it's best to just create the pool ashift=12 in the first place as 512b disks perform fine with ashift=12.

And this is just not the case as 4Kn (advanced 4Kn format) can only work with ashift=12 as they don't have emulation for 512bytes as opposed to the 512e (advanced format drives) which have 4K physical sectors, report 512b logical sectors and can do the emulation. Therefore your statement for ashift values is only true for 512e drives, and yes with 512e drives we know that there is write performance issues due to the read-modify-write involved during the emulation.

If someone is reading your post in the way you have written it, they can get the wrong impressions that first 4Kn drives can work with ashift=9 and second that they suffer performance issues.

~Cheers~
 

Stereodude

2[H]4U
Joined
Oct 20, 2000
Messages
3,285
I don't know how ZFS works, but 4kN drives are problematic if programs can write directly to the hardware and try to use 512byte sector access. This can happen in Windows. I don't know if Linux with ZFS allows the same thing or not. On the Windows side there was basically no upside to using 4kN drives over 512e drives, only a large potential pitfall.
 

drescherjm

[H]F Junkie
Joined
Nov 19, 2008
Messages
14,938
if programs can write directly to the hardware and try to use 512byte sector access. This can happen in Windows.

Are you talking of externals here? I have only seen this issue with switching between internal and external caused by external units that fake 4K sectors on 512e drives so that older operating systems like windows XP can support > 2TB partitions without the use of GPT.


I have had no issue with 4kN drives on linux.
 
Last edited:

Stereodude

2[H]4U
Joined
Oct 20, 2000
Messages
3,285
Are you talking of externals here? I have only seen this issue with switching between internal and external caused by external units that fake 4K sectors on 512e drives so that older operating systems like windows XP can support > 2TB partitions without the use of GPT.

I have had no issue with 4kN drives on linux.
No, I'm talking about internal 4kN drives. People have reported I/O timeouts and/or very slow I/O for programs that try to access the drives directly bypassing Windows API's. Also, Windows 7 doesn't support 4kN. Windows 8.x and 10 do.
 
Top