How do i copy alot of files 130k+ 13TB

Freak1

Limp Gawd
Joined
Sep 9, 2009
Messages
191
My server of 13TB is full (Vista) and now i want move the data to the new server (2008R2)

I tryed windows explore copy / Paste but it starts copying and says 85.xxx items and 1,7TB remaning so i stopped that again. I tryed total commander but the tranfer is quiet slow (30-40mb/sec) where windows explore is about 100-105 mbit/sec so it will take 80 hours

I tryed using windows server backup but that won't tage backup of a network drive.

So any recomendations?
 
I like Teracopy But I've never tried that much data. The most I've ever moved was 300GB.

* Copy files faster. TeraCopy uses dynamically adjusted buffers to reduce seek times. Asynchronous copy speeds up file transfer between two physical hard drives.

* Pause and resume file transfers. Pause copy process at any time to free up system resources and continue with a single click.

* Error recovery. In case of copy error, TeraCopy will try several times and in the worse case just skips the file, not terminating the entire transfer.

* Interactive file list. TeraCopy shows failed file transfers and lets you fix the problem and recopy only problem files.


* Shell integration. TeraCopy can completely replace Explorer copy and move functions, allowing you work with files as usual.

* Full Unicode support.

* Windows 7 x64 support.
 
I like Teracopy But I've never tried that much data. The most I've ever moved was 300GB.

I was recently pointed to TeraCopy, moved all of my data from my 2 640GB drives to my 2TB drive, it was twice as fast as windows and It tested every file after.
 
Use SuperCopier, they finally updated the beta to work with windows 7 too.
I've used this to transfer 10+TB workloads no problem, millions of files, etc.
All other copier programs I've tried choked on those loads, and of course explorer.exe copy failed miserably.
 
ive used KReplicator from karenware (google it to get exact name) to copy ~2tb from one server to another for a customer.
Just set up and run
 
As far as the copy speed is concerned in my case, I plan on sticking my new controller with the new array in the old server and try to get better speeds that way...
But even than I wouldn't move more than couple of hundred gigs at a time...
 
Unstoppable Copier and Teracopy are worth trying.
I don't know if a raw sector-by-sector copy would be faster.
 
You can try all these other guy's suggestions, but only SuperCopier in my experience works with huge workloads (millions of files). I've had "out of memory" errors from robocopy, synctoy, windows explorer, etc. If you need faster copy speed from SuperCopier, increase the buffer size in advanced options and minimize the thread window (as updating the thread window slows down the copy process). SuperCopier can also gracefully automate error handling if you tell it what you want it to do in the options, and is pretty flexible in queuing.

If you have local access to both partitions, it might be faster to do a sector-by-sector copy using e.g. acronis true image clone drive.
 
I use Teracopy and have done a few 1.5tb transfer. It is a lot faster than Windows, and has the option for CRC checks. I believe it has a pause and resume function too, but I've never used it.
 
You can try all these other guy's suggestions, but only SuperCopier in my experience works with huge workloads (millions of files). I've had "out of memory" errors from robocopy, synctoy, windows explorer, etc. If you need faster copy speed from SuperCopier, increase the buffer size in advanced options and minimize the thread window (as updating the thread window slows down the copy process). SuperCopier can also gracefully automate error handling if you tell it what you want it to do in the options, and is pretty flexible in queuing.

If you have local access to both partitions, it might be faster to do a sector-by-sector copy using e.g. acronis true image clone drive.

Just as another point of data I've used robocopy in production use for millions of files and haven't ran out of memory yet but I may just be lucky. What version of robocopy were you using at the time? Robocopy behavior varies quite a lot depending on version in my experience.

SuperCopier does look like a handy utility, thanks for the recommendation.
 
Thanks to everyone making suggestions. I ended up trying SuperCopier it is in the process of copying right now, i will report in later.
 
In my experience, Teracopy crashed so often that it caused more problems than it helped.
 
I was, at the time, using robocopy when it was a separate download from Microsoft, so maybe it was because it was an older version. I see now that it's included standard in windows7 so maybe those issues are fixed now. Robocopy with the /mirror switch is pretty handy to incrementally update my offline backup, but if I'm copying a lot of files into an empty partition, I tend to use supercopier instead.

I should also remember to put that date switch mentioned up there in my batch file as I did notice that the directory dates ended up with the date of creation, rather than the original directory date. Not that big of a deal but it is kind of annoying.
 
Robocopy or richcopy, which is a gui for robocopy.

I've used robocopy to migrate quite a bit of data and using the logging capability helps make sure everything transferred properly.

I don't believe robocopy supports multithreading, richcopy does.
 
I don't believe robocopy supports multithreading, richcopy does.

Code:
-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows

-------------------------------------------------------------------------------

  Started : Tue Dec 08 14:21:04 2009

              Usage :: ROBOCOPY source destination [file [file]...] [options]

             source :: Source Directory (drive:\path or \\server\share\path).
        destination :: Destination Dir  (drive:\path or \\server\share\path).
               file :: File(s) to copy  (names/wildcards: default is "*.*").

::
:: Copy options :
::

[...]

           /MT[:n] :: Do multi-threaded copies with n threads (default 8).
                      n must be at least 1 and not greater than 128.
 
In a case like that, I'd use robocopy to keep folder dates intact via something like
Code:
robocopy source destination /MIR /DCOPY:T

That was definitely a deal breaker for Robocopy until they included the copying of the date of creation :)
 
Thanks again guys.

SuperCopier ended up making some errors and halted 3 times. So i tryed robocopy yestoday and its been running for about 20 hours now and has copied 7,5TB and is running at about 100 mb/sec so im very happy.

When the copy is done how do i make sure the files are 100% the same. Make a SFV file will proberly take as long as the copy right?
 
When the copy is done how do i make sure the files are 100% the same. Make a SFV file will proberly take as long as the copy right?

You can use rsync in checksum mode configured to do a dry run and just report files that would be copied. Something like

rsync -a -c -n -v c:\source d:\dest

You can get an rsync build for windows here http://www.itefix.no/i2/node/10650
 
Robocopy is the fastest Windows copy tool I have found. I work in the I/T industry and we have to move massive amounts of data more than we would like to. Nothing has been faster then Robocopy for this task.
 
Yeah, the problem with that many files is that without a per-file CRC check you don't have a good way of verifying if your data was copied properly. With millions of files, how are you going to notice if 100 didn't copy over?
 
If you're copying files rather than bits: I usually find that ftp is faster than robocopy, for large files. If you like robocopy, check out hobocopy.
 
So I tried ViceVersa but when it compares it only checks if the files are of different dates and size it doesn’t check if a file has been corrupted during the transfer
 
Do any of these tools copy NTFS permissions along with doing a CRC check like TeraCopy does? I'm looking through TeraCopy options but it doesn't seem to state anything of the sort.
 
Back
Top