Hard drive defrag / questionable data transfer speeds?

[FenFox]

1.) I'm wondering if any of you guys manually defragment drives since Windows 10 does this automatically on a weekly basis?

Looking at the Optimize Drives application I can see that all my drives are 0% fragmented.

I notice that when I transfer large single files (50-90 GB) from an internal drive to an external drive, my transfer speeds will fluctuate. The transfer speed can dip to 20 MB/s and go up to 100+ MB/s. There are no other tasks being performed on these drives during the transfer.

So I decided to check and see if this was a fragmentation issue. The internal drive is about 3 years old while the external is brand new. I've checked the SMART status of these drives and everything is fine. I expect that transfer speeds would fluctuate if there are many files of differing sizes being transferred at the same time, but with singular large file transfers, why is the transfer speed not more consistent?

On another note, why does Windows defrag SSDs? Everything I've read indicates that you should never defrag an SSD.

 

[auntjemima]

I'll preface this with I didn't know Windows 10 defragged anything. Second preface, I didn't know defrag was still a thing.

Are you copying to a thumb drive or an actual external drive?

 

[TrailRunner]

Everything that you've read that says that you should never defrag an SSD is 100% correct - moving the data around (as defragging does) would cause memory cells to get marked 'occupied' and leave the SSD's garbage collection more work. The Windows 10 (and 8) defrag operation, when run on an SSD, doesn't actually do the traditional spinning-disk move-the-data type of defrag; it actually just runs a TRIM operation on the SSD, which will revert unused memory cells back to 'clean' state, and that restores the performance of the SSD. So doing a 'defrag' with Windows 10 on an SSD is safe - because you're not actually doing the defrag, you're just doing a TRIM.

Especially never ever run defrag on a USB thumb drive - those devices don't have controllers that are advanced enough to have TRIM, and if you are 'successful' in performing the defrag you'll be performing the HDD style of defrag. This will especially kill the performance and reliability because the controller in those devices isn't smart enough to handle a 100% dirty drive, and because there's no TRIM, no way to get the performance back up to like-new condition. Related to this- don't use NTFS or any other journaling filesystem on a USB flash drive - those filesystems are quite 'busy' and result in far more writes to the device than a simpler file system such as FAT or the newer exFAT.

 

[FenFox]

Everything that you've read that says that you should never defrag an SSD is 100% correct - moving the data around (as defragging does) would cause memory cells to get marked 'occupied' and leave the SSD's garbage collection more work. The Windows 10 (and 8) defrag operation, when run on an SSD, doesn't actually do the traditional spinning-disk move-the-data type of defrag; it actually just runs a TRIM operation on the SSD, which will revert unused memory cells back to 'clean' state, and that restores the performance of the SSD. So doing a 'defrag' with Windows 10 on an SSD is safe - because you're not actually doing the defrag, you're just doing a TRIM.

Especially never ever run defrag on a USB thumb drive - those devices don't have controllers that are advanced enough to have TRIM, and if you are 'successful' in performing the defrag you'll be performing the HDD style of defrag. This will especially kill the performance and reliability because the controller in those devices isn't smart enough to handle a 100% dirty drive, and because there's no TRIM, no way to get the performance back up to like-new condition. Related to this- don't use NTFS or any other journaling filesystem on a USB flash drive - those filesystems are quite 'busy' and result in far more writes to the device than a simpler file system such as FAT or the newer exFAT.

Heh, I've always used NTFS on my thumb drives, but I've only ever used 3x. Never had a problem with NTFS on a thumb drive. I may just use exFAT from now on though. The reason I never used FAT is due to the 4 GB restriction. "Far more writes to the device" ?

Still, I'm wondering why my transfer speeds fluctuate as outlined in my OP? (Internal HDD to external HDD when transferring a single 50-90 GB file).

 

[FenFox]

I'll preface this with I didn't know Windows 10 defragged anything. Second preface, I didn't know defrag was still a thing.

Are you copying to a thumb drive or an actual external drive?

Yeah, I don't even know how I knew that Windows does a weekly scheduled defrag. I either stumbled across this information in Windows when looking for something else or read about it in an article from years ago. I haven't actually manually defragmented an internal drive since XP so I mostly forgot about it. Just type "defrag" into the Windows search bar and you'll be able to see that Windows by default schedules weekly HDD defrags. I think Windows schedules a monthly defrag for SSDs, but not entirely sure.

I'm copying from an internal HDD (desktop) to an external HDD when I notice the fluctuations. But I speculate this would occur even from internal to internal. Haven't tested this yet as this was never something I really cared to look too deeply into. But from what I read lately, fluctuations mostly occur when transferring an assortment of file sizes at the same time, not typical to see when transferring a single large file at the same time. Though, I could have misinterpreted what I read.

 

[pendragon1]

it could be windows defender or what ever av youre using. try disabling it and then do a transfer. it could also be a cache filling up, maybe?!
ssds dont get defragged, they get optimized(trim garbage collection etc) and windows 10 will do it automatically.

 

[Denpepe]

it could be windows defender or what ever av youre using. try disabling it and then do a transfer. it could also be a cache filling up, maybe?!
ssds dont get defragged, they get optimized(trim garbage collection etc) and windows 10 will do it automatically.

Yeah I would assume the disk that writes the data it's cache get's filled and it slows down to catch up.

 

[FenFox]

Yeah I would assume the disk that writes the data it's cache get's filled and it slows down to catch up.

Is there anything you can do about the "cache filling up" ?

I just tested 2x file transfers today. 1x 65 GB and 1x 52 GB.

On one transfer I was achieving 104+ MB/s and on the other I hit 154+ MB/s (These transfer rates were stable throughout - only increasing, never decreasing).

Just strange how there can be a 54+ MB/s disparity and in other transfers (not these ones) the transfer can dip to 20-40 MB/s for a bit before increasing.

 

[likeman]

Caching is disabled by default on external disks/usb (when data has finished its finished, but on NTFS even with write caching not enabled small risk of data loss if not ejected) to get full speed enabling write cache get get better speeds (assuming USB 3) just remember to eject it

From Windows vista/7 onwards it has always defragment automatically (Windows 7 supports automatic disable defrag when it's a ssd, run Windows performance index rescan) Windows 8 and higher detects it immediately (defrag on Windows 8 and higher does a trim command when you do defrag on ssd)

 

[Denpepe]

Is there anything you can do about the "cache filling up" ?

I just tested 2x file transfers today. 1x 65 GB and 1x 52 GB.

On one transfer I was achieving 104+ MB/s and on the other I hit 154+ MB/s (These transfer rates were stable throughout - only increasing, never decreasing).

Just strange how there can be a 54+ MB/s disparity and in other transfers (not these ones) the transfer can dip to 20-40 MB/s for a bit before increasing.

Are these SSD's or spinning rustbuckets? If spinners it might be the location of the data on the media which makes it slower to read.

 

[pendragon1]

Are these SSD's or spinning rustbuckets? If spinners it might be the location of the data on the media which makes it slower to read.

pretty sure they are spinners and that is a very good point too.

 

[Deleted member 245375]

Just had a thread about such a topic and such information recently:

https://hardforum.com/threads/need-faster-data-transfer-speeds.1961591/

It all depends on the storage media (the type, aka hard drives, SSD, NVMe, etc) and the sizes of the files being transferred. Even if you had two NVMe SSDs with sustained reads and writes of 2GB/second when you start transferring a lot of smaller files that speed drops through the floor because of how things work.

Specs are always going to promote the best possible speeds for storage media in the best possible situation - in real world day to day usage, don't ever expect those numbers to reflect accurate information 'cause they never really do. The only time those "press specs" or best possible speed numbers matter is in the benchmarking.

 

[FenFox]

I should've been more specific I suppose.

Transferring from an internal 5400 RPM to an external 5400 RPM.
Not SSDs.

 

[Deleted member 245375]

Doesn't matter what the storage medium is, as I was hinting at with that info and that link to the other thread where more details are presented. It's how modern storage devices work, and just because technology improves that doesn't mean everything is going to be super-ultra-hella-crazy-awesome-megafast regadless.