best way to transfer 4TB of data

S

savager1

n00b
Joined
Jan 5, 2007
Messages
36
Hello,

I have roughly 2.7TB (after taking RAID into account) of data that needs to be transferred.

The transfer is only on my local LAN between a NAS server and RAID box. Here's an explanation and a list of problems I've encountered.

Source device (which has all the files) is a OpenSolaris box running opensolaris 10 which doesn't have NTFS support (not that I could see) I was told to download some files, but they didn't appear to exist anymore. Currently the device is running in RAIDZ1 [which is like RAID5 If I have that correct]. Now the...

Destination device (where I want to copy the files to) is a Hotway RAID box USB3.0 connection. That's connected to my Windows 7 box (tried to plug into NAS server, it recognised it etc, but couldn't mount due to NTFS). I can format the destination device, just as long as the files can get onto there.

I downloaded DeltaCopy and tried to use it and the rsync files but it didn't quite work. Maybe I wasn't using it properly.

So, the big problems are how long it takes obviously as I have large and small files ranging in size. The issue also is that Unix reports different file sizes to Windows for the same files. That's not a problem but it's hard to tell whether WinSCP has copied the file in full or is it just the differing sizes. Either way it would take forever to compare all the files etc. Also, to add to this, I don't really want to leave the NAS on all the time copying files over...so when I start / stop transfers I cannot tell if it's just the differing sizes of the the OS's or did the file not copy over correctly.

So, I was after anyone who can suggest the quickest way to transfer 2.7TB from one OS Unix in this case to the RAID box connected via USB 3.0 to my Windows 7 box?

Thanks in advance.
 
Someone may come along with a better solution but my suggestion is share the drive on the windows box. Mount the share to the opensolaris box and then just use rsync to transfer them.
 
Robocopy is built into Windows 7.

IMO, share the folder from your opensolaris box and then mount it on your Windows 7 machine. Then you have your USB drive mounted as well, and just tell robocopy to mirror the data from the share to the other share.

eg:
O: = opensolaris mount
U: = USB mount
robocopy O:\ U:\ /MIR

That command would run it all. Then it will give a summary at the end letting you know about any errors etc. and it doesn't matter if you have to stop it part way.. you can just run the command again and it will go through all the files and only copy what hasn't been done yet.

More info here
 
Is this just a one time thing? Just mount an SMB share in windows and drag the files to your USB 3.0 drive. 3TB at 1GbE speeds should take just under 9 hours. But it it's a one time thing just fire it up and let it go.

If you are worried that you are somehow going to introduce subtle errors than the robocopy option mentioned above is fine - but really there's no need to make this complicated at all.
 
SMB/CIFS + GUI
NFS + GUI
rsync
Robocopy

Or, if you don't want to use the network, set up a solaris VM on your windows box, pull the drives out of the solaris box, put them in the windows box, pass through the raw drives, do a zpool import, smb share it to the host windows box, and copy things over that way. Might be a bit faster thanks to a virtual network instead of a real one, but by the time you're done setting everything up, you could have just rsync'd or robocopied.
 
I use FastCopy ( http://ipmsg.org/tools/fastcopy.html.en ) when moving a large amount of data from point a to point b in a windows environment. Use the move function so when a file is done copying it moved from point a to point b and no longer resides on a point a

I've used this to migrate 10TB luns before and it works well and is straight forward to use.
 
You must log in or register to reply here.
Back
Top