Efficient copying of large files

A

ameoba

Supreme [H]ardness
Joined
Jan 9, 2001
Messages
6,412
Ugh. Just copied a bunch (2GB) of files from one partition to another on the same drive.

That sucked.

I've got DMA turned on and have reasonably fast drives, but this is just sick how long it takes. Considering that I've got 1GB of RAM, there should be a better way of handling this than trying to stream the files from one drive to the other; it'd obviously be faster if it read some large chunk into memory then dumped it out to the other drive in one operation, preventing the HDD from thrashing.

any way to get windows to do this by default?
 
I don't know that more ram would help this. The bottleneck is always going to be the hard drive. Look at it this way, you first read 1gb of data to ram and then write it form the ram, it is just sitting idly in the ram until you finish the write. More ram just leaves you more room for things to sit in while things are still waiting to be written.

The only suggestions I could give would be to look into a couple of those nice 10k drives. I don't have one, but have heard they have great performance.
 
If you've got 2K or XP and don't mind a little CLI, then Robocopy (built into XP) might work a bit faster for you........
 
I don't know that more ram would help this.
Click to expand...

Well, not in normally copying, but, if you think about what's happening, if you try to stream files from one partition to another on the same drive, only transfering one block of data (which I think is something like 4KB) at a time, you end up reading 4KB, moving across the drive, writing 4KB, moving across the drive, reading 4KB... and end up spending a lot of time moving the drive heads across the drive. If you could read a large block, even just 1MB at a time, you'd only be spending 1/256th as much time moving between where you're reading and where you're writing... At some point, the law of diminishing returns kicks in, so there's a sweet spot, probably between 1MB and 32MB per chunk...
 
I would like to add that the speed also depends on the file system status on the drive. If you want to copy a file from drive A to drive B and drive B doesn't have enough contiguous free space, the copy will suck, regardless of ram. The cluster size could make a different too.
 
the easiest way to improve speed is probably to just move one of the drives to the secondary ide channel, so they can both be accessed in parallel
 
Originally posted by cloaked
the easiest way to improve speed is probably to just move one of the drives to the secondary ide channel, so they can both be accessed in parallel
Click to expand...

He is speaking of separate drives in there, but note this:
"Ugh. Just copied a bunch (2GB) of files from one partition to another on the same drive."

Copying inside one drive is slow, but I agree with ameoba.
As for how, well, smartdrv left the building with win'98 (or ME), soo... you can hope XP optimizes this intelligently. I doubt it, but you can hope.
 
I'm afraid HD is the slowest part of a comp (not counting CD/floppy).

It has lagged behind all the other parts of the comuter dreastically. We have faster CPU's faster/larger RAM and ok bigger gisks but the access/seak time hasn't gone up that much.

We are still relying on a spinning disk of metal and thus there is a limit on how fast it can be spun - the highest IDE I have seen is 10,000rpm and SCSI 20,000rpm

faster RPM and bigger HD cache = bigger data trasfer rate.

The one thing you must note abt HD is that they are constant angular freq devices thus there is faster data access near the hub of the disc.
 
You must log in or register to reply here.
Back
Top