Change from 512 byte sectors to 4k size?

C

c3k

2[H]4U
Joined
Sep 8, 2007
Messages
2,330
Folks,

I've found out that ALL my drives are formatted to have 512 byte sectors. 4k seems to be the way to go.

For my Data drives, I can just add a new drive, format to 4k, then manually copy from old Data (512) to new Data (4k). That should work...

How do I move my old OS (512) to a new OS (4k) drive?

- Disk Cloning won't work since the sectors are different.

- Reinstalling the OS is a major PITA. I really don't want to do it. (Plus, this issue spans 3 different rigs. If I really don't want to do it with one machine, I really don't want to do it for three...)

Any thoughts?

Thanks,
Ken
 
Formatting today means initialising the filesystem; what exactly do you mean with 'format to 4k' or 'format to 512 byte' - that is not clear to me.

Almost all harddrives and SSDs use 512 byte sectorsize, even though the harddrive uses 4KiB sectors and the SSD uses 8KiB pages. But this remains transparent (hidden) to the host. The host only sees 512 byte sectors, for compatibility reasons.
 
This only affects drives that are 4k native. These drives, even today, are rare.

The 4k drives you will be finding are 4k with 512 emulation, some call them 512e.

There is no change needed to make these drives work, just copy everything over. To make them work optimally, you should 4k align when you move. Most backup programs will do this for you. Or you could use an alignment program after you move to the new drive.
 
OS is W7/64.

Drives are Samsung SSD 840 Pro, 840 Evo, 850 Evo; or WD Black, 1 Tb spinners (WD1002FAEX).

By "formatting", I mean using W7/64 to reformat the drives. One of the options is sector size. The options include "default" (or "optimized"?), or the various sizes chosen manually. I =thought= I'd always picked "default". Shrug.

Obviously, based on my question, you folks know a lot more about this than I do. Is a 4k byte sector size better than a 512? Does it matter?

(FWIW, system info and fsutil show physical and logical sectors as 512. My understanding is that 512e would show one of them (physical) as 4k.)

Thanks,
Ken
 
c3k said:
OS is W7/64.

Drives are Samsung SSD 840 Pro, 840 Evo, 850 Evo; or WD Black, 1 Tb spinners (WD1002FAEX).

By "formatting", I mean using W7/64 to reformat the drives. One of the options is sector size. The options include "default" (or "optimized"?), or the various sizes chosen manually. I =thought= I'd always picked "default". Shrug.

Obviously, based on my question, you folks know a lot more about this than I do. Is a 4k byte sector size better than a 512? Does it matter?

(FWIW, system info and fsutil show physical and logical sectors as 512. My understanding is that 512e would show one of them (physical) as 4k.)

Thanks,
Ken
Click to expand...

Best performance for small files is 512 because it matches the HD/SSD sector size.

Best performance for movies is 64K. Why? Because there is less overhead when the OS asks for a file.


It's like the difference between a rubbermaid container vs a shipping container. If you want to store DVDs, its more convient to use a rubbermaid contailer because it is easily accessed if you want to pick a single DVD. If you want to store larger bulk things like a car, you could in theory store them in rubbermaid containers, but you would have to cut it up and reassemble it each time you ask for it. But if you stored your car in a shipping container you could pull the shipping container and have the car ready to go with minimal effort. But likewise it does not make sense if you want to store a single DVD in a shipping container.
 
c3k said:
By "formatting", I mean using W7/64 to reformat the drives. One of the options is sector size.
Click to expand...
This is wrong. I think you mean cluster size, not sector size. That is a major difference. The sectorsize is determined by the storage device, not by the filesystem. Cluster size is a filesystem setting and the SSD/HDD will not care what you select here.

Leave it at its default setting.
 
Lol...it's obvious I'm in over my head.

Here's an image from System Info:



Where it shows 512 Bytes/Sector...I thought 4k per sector would be the "advanced" mode and result in lower EEC overhead.

At this point, knowing that I don't know what I'd like to know about this, it'd probably best to follow some of the advice posted upstream and just leave this alone. ;)

It will gnaw at me for years. Sigh.

Ken
 
c3k said:
Lol...it's obvious I'm in over my head.

Here's an image from System Info:



Where it shows 512 Bytes/Sector...I thought 4k per sector would be the "advanced" mode and result in lower EEC overhead.

At this point, knowing that I don't know what I'd like to know about this, it'd probably best to follow some of the advice posted upstream and just leave this alone. ;)

It will gnaw at me for years. Sigh.

Ken
Click to expand...

The thing is this setting is really applicable to HDDs. In an ideal world all the file system clusters will be sequential, as in they will be in order physically on the disk so they can be read in order when asked for. In this case there is not much difference between 512 and 4k or other settings at normal speeds. The efficiency difference comes once the drive begins to become fragmented, if you are trying to read a movie file and the drive has to be running all over looking for small 512b "rubbermaid" containers its really inefficient.

The only times you really see wow night and day differences in cluster size, are at 10Gbps speed (1000MB/sec) and USB. You really see lower CPU utilization at really high speeds but thats not home usage and you are probably going to be using some other features to try and counter this. That and you really gain efficiency (80MB/sec+ on USB2) with USB by going to a 512K or 1MB cluster, though you waste a lot of space for normal files.
 
staticlag said:
The thing is this setting is really applicable to HDDs. In an ideal world all the file system clusters will be sequential, as in they will be in order physically on the disk so they can be read in order when asked for. In this case there is not much difference between 512 and 4k or other settings at normal speeds. The efficiency difference comes once the drive begins to become fragmented, if you are trying to read a movie file and the drive has to be running all over looking for small 512b "rubbermaid" containers its really inefficient.

The only times you really see wow night and day differences in cluster size, are at 10Gbps speed (1000MB/sec) and USB. You really see lower CPU utilization at really high speeds but thats not home usage and you are probably going to be using some other features to try and counter this. That and you really gain efficiency (80MB/sec+ on USB2) with USB by going to a 512K or 1MB cluster, though you waste a lot of space for normal files.
Click to expand...

So does that mean if you have a NAS that is on 10Gb E you want the NAS to be on 4K?

Doesn't 4k waste a lot of space on small files?
 
SomeGuy133 said:
So does that mean if you have a NAS that is on 10Gb E you want the NAS to be on 4K?

Doesn't 4k waste a lot of space on small files?
Click to expand...

Yes, it should be much greater. Rarely do you have small files on a non-OS drive.

Most NAS'es use some type of RAID that will use a stripe much larger than 4K, ZFS uses 128K. (freeNAS, omnios, solaris)

The problem once you go over like 800MB/sec CPU usage skyrockets because the overhead the filesystem adds cripples the actual NAS transfer, you can mitigate this with large clusters/stripes, thats why these large sizes are the default for most enterprise file systems.

99% of the time for home use, using a larger cluster is good for NAS. Only exception is if you are hosting a SQL database or popular website.
 
staticlag said:
Yes, it should be much greater. Rarely do you have small files on a non-OS drive.

Most NAS'es use some type of RAID that will use a stripe much larger than 4K, ZFS uses 128K.

The problem once you go over like 800MB/sec CPU usage skyrockets because the overhead the filesystem adds cripples the actual NAS transfer

99% of the time for home use, using a larger cluster is good for NAS. Only exception is if you are hosting a SQL database or popular website.
Click to expand...

So using SnapRAID on 4K would be better if it is a Plex/backup/DAS. OS I can leave as 512B right? Server will also be my Photoshop PC so OS i would like to leave 512B

EDIT: what if their are Encrypted Containers? I plan on Encrypting large amounts of data with Veracrypt

Sorry I guess it is hijack day for me
 
Your OS drive should never be part of your file raid, it should be its own separate drive.

Let me make it easy, if the total size of the drive is over 1TB , it should be at least 4k for home use. I would recommend 64k
 
staticlag said:
Your OS drive should never be part of your file raid, it should be its own separate drive.

Let me make it easy, if the total size of the drive is over 1TB , it should be at least 4k for home use. I would recommend 64k
Click to expand...

i know they are separate but I don't really want a 500GB SSD as 4k.

There isn;t an issue with 4k/64K with encrypted containers?
 
SomeGuy133 said:
i know they are separate but I don't really want a 500GB SSD as 4k.

There isn;t an issue with 4k/64K with encrypted containers?
Click to expand...

I'm not sure how the encryption works to be honest, to really give a good rec you should check their forums. Because I'm not sure how the encryption deposits data inside the container file. I can tell you that there will be essentially no "wasted space" overhead though, since it has already reserved the entire cluster for the container. But I can generalize and say you won't be getting max performance by any means, if you are storing media files 4k+ (64k) is best case usage and you should probably treat it similarly.
 
staticlag said:
Most NAS'es use some type of RAID that will use a stripe much larger than 4K, ZFS uses 128K. (freeNAS, omnios, solaris)
Click to expand...
ZFS uses a dynamic stripesize - and thus can vary from <sectorsize> (usually 512B) to 128KiB, or with large_blocks feature up to 1024K. This does not mean every file will take 128KiB.

Stripesize also should not be confused with cluster size. You can have a 1024K stripesize but a 512B cluster size so that small files will take only 512 bytes, not 1024K.

Also, if you have a 512 byte or 4KiB clustersize, and you do large writes, this does not mean the host will do many small I/O requests. Writing a large file on an NTFS filesystem with 4K clustersize, will result in >= 128KiB request size, not 4KiB.

Likewise, having a 1024KiB stripesize and you want to write only 64KiB, that does not mean the host has to write 1024KiB. Generally the stripesize remains completely hidden from the filesystem. Only ZFS and other third generation filesystems are different because they have the filesystem and RAID part melted together. ZFS uses 'recordsize' instead of stripesize.
 
_CiPHER_ said:
ZFS uses a dynamic stripesize - and thus can vary from <sectorsize> (usually 512B) to 128KiB, or with large_blocks feature up to 1024K. This does not mean every file will take 128KiB.

Stripesize also should not be confused with cluster size. You can have a 1024K stripesize but a 512B cluster size so that small files will take only 512 bytes, not 1024K.

Also, if you have a 512 byte or 4KiB clustersize, and you do large writes, this does not mean the host will do many small I/O requests. Writing a large file on an NTFS filesystem with 4K clustersize, will result in >= 128KiB request size, not 4KiB.

Likewise, having a 1024KiB stripesize and you want to write only 64KiB, that does not mean the host has to write 1024KiB. Generally the stripesize remains completely hidden from the filesystem. Only ZFS and other third generation filesystems are different because they have the filesystem and RAID part melted together. ZFS uses 'recordsize' instead of stripesize.
Click to expand...

interesting so I bet SnapRAID probably does the same thing. Thats pretty cool stuff
 
Since this is not about drives, but about ntfs cluster size.

I recommend always use default (4k) setting for ntfs os drives, there are a lot of ntfs features that require the default setting, and that get disabled if it is changed.

If your formatting your disk for non-os usage, then it might be useful to set it to 16k, or 64k, depending on the usage. If you are storing pictures and movies, on a dedicated disk, go 64k. If you are using it for sql or other high performance usage, 16k. But remember those other things will be disabled when not on 4k, like native compression, and a few other things I can't remember at the moment.
 
You must log in or register to reply here.
Back
Top