Follow along with the video below to see how to install our site as a web app on your home screen.
Note: This feature may not be available in some browsers.
He is asking about a storage drive not an OS drive. I will presume you are using Win7 or 8 and there is no defragmenting that needs to be done. You will have no problems with filling the drive and there will be no speed reduction or any other problems.
If you fill a disk more than 90% it will be very very slow (less than 10MB/sec or so) - this is because all filesystems have great problems with more than 90% full. This slowness dissappears as soon you delete some files. So you can fill the disk until everything gets really really slow, and delete some files and it will be quick again. But the disk will not be damaged, no. Everything will be very slow, that is the only problem.
I have not found this to be true at all.
I regularly rsync data from my large ZFS array onto 2TB, 3TB, and 4TB NTFS backup disks and they write at around 100MB/s even during the last 1GB of the drive's free space, right up until rsync dies due to the disk literally filling up and maybe a couple KB of free space. Then I just delete the last file rsync tried to copy (since it's a partial file) and move onto the next disk.
I have never encountered any issue filling drives completely full, at least not with NTFS.
It is my understanding that different file-systems handle being filled up differently.
But I think the problem would normally come from free-space fragmentation as you delete files and add new files. If you have 10GB free but it's in 10,000 1MB chunks, then yes writing a single 10GB file will probably be slower.
However, I am sequentially filling my drives from start to end in one copy process and I never delete a file from the disk so it has no problem writing until the very end of the disk at full speed the whole time.
I haven't seen any problems reading at full speed from a completely full drive either which is all my backup disks need to do when I run a checksum on all it's contents every so often.
I will repeat, there is no speed reduction at all (none) if you are using Win 7 or 8 on a storage only drive or even a partitioned storage area. Many years ago it was recommended for the OS portion of a drive to have a 50% free space. Why? Mainly because Windows kept all of the updates added to the OS as time went by and you needed the added space for that.
Actually ZFS is known to have serious performance issues when the zpool starts getting full and its recommended to only ever be 80% (I think it was) full.
Also to the guy commenting about SSD... that only matters if the SSD and your drivers/OS supports trim.
This is very strange, I have dabbled with computers for 25 years and have never seen no speed penalty when disks are full. There are ALWAYS a speed penalty when disks are full. This applies to ALL filessystems, not just ZFS. My friend had this speed penalty problem on NTFS, I told him to delete some files and the storage disk was fast again. Myself has speed penalty on ZFS on a storage disk, until I deleted some files.I will repeat, there is no speed reduction at all (none) if you are using Win 7 or 8 on a storage only drive or even a partitioned storage area.
Again, ALL filesystems get a speed penalty when they are full. If you never had a speed penalty on full disks - that is strange. It is like you are saying that "yes, my PC started page RAM because I loaded a 10GB workload onto my 4GB RAM pc - and it was not slower". If your PC starts to swap - performance WILL decrease drastically. That is a fact. Implying otherwise is strange and against common wisdom. Same with full disks - they WILL be slow.I have not found this to be true at all.
...
I have never encountered any issue filling drives completely full, at least not with NTFS.
...
It is my understanding that different file-systems handle being filled up differently.
There are several threads where people complain of bad ZFS performance, and after some discussion it turned out their raidz was >95% full. After deleting some files, performance was good again. And this applies to ALL filesystems, including ZFS, NTFS, ext4, etc. If your PC starts to swap RAM out to disk it WILL be slow - that applies to all OSes, including Windows, Solaris, Linux, etc. Same with full disks.Actually ZFS is known to have serious performance issues when the zpool starts getting full and its recommended to only ever be 80% (I think it was) full.
Again, I believe it to be related to free-space fragmentation which if you are just using the disk or array to store large files that you don't delete is never a problem.
This is very strange, I have dabbled with computers for 25 years and have never seen no speed penalty when disks are full. There are ALWAYS a speed penalty when disks are full. This applies to ALL filessystems, not just ZFS. My friend had this speed penalty problem on NTFS, I told him to delete some files and the storage disk was fast again. Myself has speed penalty on ZFS on a storage disk, until I deleted some files.
What you are saying, is against common wisdom. You are wrong.
As far as full filesystems slowing down, you are, of course, correct. There is minimal slowdown due to a filesystem being full if fragmentation is minimal (except, perhaps, for certain write or delete operations in COW filesystems, where there could be some issues on a full filesystem).
But all HDDs have lower throughput on the inner cylinders. Most 2+ TB HDDs will have sequential throughput of about 50% at the innermost cylinders (as compared to the outermost cylinders).
So, you can expect a slowdown of as much as 50% if you fill up a typical HDD and then read from the files on the innermost cylinders. Of course, if you read from the files on the outermost cylinders (even on a full HDD) then you will see full speed (unless there is significant fragmentation).
Since the post you were replying to was talking about massive slowdowns, on the order of 10-20% of full-speed, you were correct in pointing out that such slowdowns do not actually occur unless the drives are significantly fragmented (and even then it would have to be a specific type of fragmentation to cause such a severe slowdown).
I guarantee you that rsync writes at 80-100MB/s write speed (according to rsync) until the drive is completely full, even during the last 1GB of free space it's still writing at 80-100MB/s throughput.
It all has to do with access patterns.
If you start with an empty drive, and write it until it is full, you will have good write speeds all the way down to the last sector.
The performance penalty for having a nearly full drive that most people are familiar with comes from fragmentation of the filesystem and has nothing to do with the hardware itself. Even then general comments are difficult at best because how the performance deteriorates if at all is completely dependent on how files were written and deleted in the past.
Now that is bizarre. Either something is throttling your speed on the outermost HDD cylinders (but not the innermost cylinders), or somehow the writes are being randomly distributed among the cylinders rather than filling the outermost cylinders first and the innermost cylinders last.
There is a plethora of empirical evidence (as well as a solid conceptual explanation) that the throughput of HDDs decreases from the outermost to the innermost cylinders. A typical 5K rpm 2TB HDD would have sequential throughput of say 120 MB/sec (100-150 depending on platter density) on the outermost cylinders, but decreases to around 60 MB/sec (50-75, depending) on the innermost cylinders.
A higher density platter (such as in a 4TB HDD), or a 7K rpm HDD, could be around 160 MB/sec on the outermost cylinders and around 80 MB/sec on the innermost cylinders.
It is really strange that you do not experience slow downs with a full disk. I have experienced it several times, as soon as I delete files, performance goes up again when the disk is full. But if you say that you dont experience slow downs - you dont. I dont believe you are lying.
This means there is something we dont understand here. Sometimes performacne will go down, other times, not.
http://tuxera.com/forum/viewtopic.php?f=2&t=7780
Maybe we should start an investigation, when will full disks slow down, and in what circumstances will they not slow down? Is it dependant on fragmentation?
http://forums.theregister.co.uk/forum/1/2012/01/17/windows_8_server_refs/#c_1286646
"... We have an HP D2D (StorOnce) VTL which runs (effectively) a rebadged RedHat with ext3 - we get access to 80 or 90% of the filesystem, when I was talking to the designers about it they specifically said that if you fill ext3 up to over 90% it is totally crippled in terms of performance, due to fragmentation and not defragmentable if there wasn't enough free space. NTFS isn't defragmentable, either if you don't have enough free space..."
I have shown links that when a disk is full, performance will suffer. And as soon you delete files, performance will be normal again. This is also my experience.To be clear, the performance penalty is related to fragmentation not used capacity.
I have shown links that when a disk is full, performance will suffer. And as soon you delete files, performance will be normal again. This is also my experience.
If this slow down was related to fragmentation - the disk should be slow even after deleting files. Because disk gets full -> slow down -> fragmented (hypothesis) -> delete files (disk is still fragmented) so performance will not go go up. Right?
Ergo, something is strange here. Clearly we dont really know what causes slow down. It should not be fragmented, because deleting files does not defragment a disk. It would be interesting to understand this.
We need some credible research papers, or articles by experienced filesystem developers. It is obvious our experiences differ.
My and my friends experiences are that we fill a storage disk (not the system disk) and performance slows to a crawl, and then we delete files and performance goes back to normal again. This was on ZFS and NTFS disks. If fragmentation was the cause, as you suspect, then deleting files counts as defragmentation? Or what? Maybe only the last files got fragmented, and when we deleted them, fragmentation was gone again? But that is unlikely as our storage disks are very likely fragmented, copying lot of files back and forth.
And if you read the net, there are lot of testomonies and recommendations to not fill up a disk, otherwise it will be slow.