New Win7 installation - harddisk write cache behaviour

[Iserlohn]

I reinstalled my PC a couple of weeks ago. Win7 x64, fully updated. I also upgraded my PC from 16GB to 32GB (still good ol' Sandy Bridge system, i7 2600K, Asus P8P67Pro)

Now I used to run Teracopy, but I'm having trouble with it now, it won't take over from the Windows copy handler. I'll fix that soon.

However, the default Windows copy handler suddenly got magic wings. It copies everything from local hard disks and SSDs instantaneously. You don't even see the copy handler window. I'm assuming this is caused by the increase in RAM memory and Windows somehow deciding to use a lot of cache memory.

This makes me rather nervous, as I don't have a UPS on this PC and I can't really monitor when the cache has been emptied and write is complete. Is there any way to tell Windows to use less cache? Or is it a choice between the current situation and no write cache at all?

 

[Deleted member 245375]

Go into Device Manager - open the properties page for the given drive(s), click Policies and choose what you feel will work best for you. In my own personal experience of using Teracopy (when it worked, surprisingly it failed more often than not and just crashed out, bad coding but that's just my experience) vs just using Windows itself offered no benefits in terms of the copy process. It never beat out the basic Windows file copy APIs (which it does actually make use of, don't believe the hype) and sometimes took significantly longer.

The only reason I was using it was the checksum verification feature but that literally doubles (almost, but not quite) the time required to do a file copy. The faster method for me was generate MD5 checksums using Hashcheck (a context menu handler add-on extension), do the file copy, then verify the checksums afterward. Again, that's just my experience with Teracopy - I always thought it was better but I decided to do actual testing one time and it just wasn't any faster in the long run and the crashes that happened during copy operations of a lot of files was irritating because I'd have to basically start all over again.

With respect to Windows, Superfetch will basically cache any files of any kind once they're read into RAM or copied someplace. You can use the RAMMap tool from Mark Russinovich founder of SysTernals which was bought up by Microsoft long ago) to check out what's in RAM at any given moment in time, and I mean absolutely everything that's in RAM, every single byte of it. Fantastic app, small and fast, gives you the ability to see what's in the Superfetch cache at any given time as well, and offers options to totally clear out RAM on various levels. Definitely a great tool to use sometimes if you're interested in exactly what's going, especially with 32GB of RAM. You can find RAMMap at:

https://technet.microsoft.com/en-us/Library/Ff700229.aspx

With 32GB of RAM you'll note (if you check out what's sitting there in RAM with that tool) that most every file you touch gets cached. I've used machines with 32, 48, 64, and even 128GB of RAM in the past (rendering workstations) and after some massive rendering job I'd check on the RAM utilization with RAMMap and find gigantic files (like 30-60GB in size) just cached in RAM weeks later after the task was done. It's not hurting anything, and it's not actually using the RAM (meaning with such files cached in RAM the actual RAM available is 30-60GB less), it's just cached there (because the workstations remained on 24/7 of course) if needed.

With respect to the caching and potential data loss, my suggestion is this and I've been giving it out for decades: if the computer is important to you, and with the investment to put 32GB in it I'd say it is, get a UPS because it's not only for a power outage and keeping the hardware running long enough to do a proper shutdown so no data is (hopefully) lost, it's going to provide much cleaner power output to the computer itself which is always a good thing.

Spending money on a nice computer and not spending just a wee bit more to get a UPS - doesn't have to be some $500 thing, could be as simple as a 500 watt $50 APC UPS or something similar - is just somewhat illogical to me, but that's my opinion so take it as such.

Good luck.

 

[Iserlohn]

Thanks for the very elaborate response!

I agree Teracopy isn't perfect. It doesn't crash with me once I get it to work on a machine, but I do need to disable UAC or it won't kick in at all. And I too only use it because of the verification. I very often have to copy stuff, and it needs to be reliable, speed is secondary.

Your method would work too if I could automate it, manually it would be too time consuming for my purposes.

Unfortunately, what you say confirmed my suspicions, and I either have to switch write cache on or off (I had it default off on Teracopy). Really it is a pain Microsoft still does not implement at least an optional verification switch on their copy. Even their command line Robocopy had it, why can't they put it into the default Windows copy?

As for the UPS, it is connected to a UPS from another machine (just the power cleaning and power surge protection output, no battery backup - that's reserved for the server) but adding one especially for this would be a bit expensive nonetheless. After all, even if a power outage happens, in order to not lose important work, it would have to provide power for up to 12 hours (I can't interrupt the jobs and have them restart once power is restored; they have to be redone completely). What I'd really need is a powergenerator, considering power reliability is dropping every year, especially in winter. But those things are expensive

Anyway, I'll disable write caching per your advice, that should solve it for now. Thanks again.

 

[Deleted member 245375]

I'll add just one thing from my experience (several decades with Windows, even longer with computers in general, since the early 1970s):

NTFS has error checking built in, it always has had a level of error correction and CRC checksums with respect to copying/moving files from one location to another as long as the file system is NTFS. In my (again) decades of using Windows on thousands of different machines, thousands of different configurations, and thousands of installations dealing with copying/moving millions upon millions of files overall I have never ever had one single instance of a copy or move operation on NTFS fail because of some file corruption and by that I mean the resulting file after a copy/move operation is not the same file I was copying/moving in the first place.

The reason the basic file copy/move operation in Windows - even when done from the command line with Robocopy - don't typically have a "verify" option is because, as just mentioned, it's part of the basic operations to begin with. If there's a CRC checksum done on the operation and if it's not a match it'll tell you.

I have had NTFS warn me in several instances that a file had a corruption noted during the copy/move operation and so I needed to replace the original which was corrupt in the first place, not by NTFS or some other problem itself - a few times it was an issue with the hard drive and it developing a bad sector where the file(s) were located originally, other times it was that the original file was corrupt when I acquired it the first time, etc.

But with all this time using NTFS, and for all the flack it catches and the grief Microsoft has dropped on it for NTFS being "old and slow and ancient and it needs to be removed and replaced" and all that other crap, I can honestly say NTFS has never failed for me in my experience.

Having said that, don't even get me started on other file systems with respect to file corruption like ext2/3/4/ReiserFS/BRTFS and HFS+ (which I despise in no small degrees but it's the best Apple can do so that's that). Had problems over the years with all of them and in alarming frequency with some of them *cough*HFS+*cough*