Moving Data From One Server To Another?

C

Carlosinfl

Loves the juice
Joined
Sep 25, 2002
Messages
6,633
So I need to migrate large bulks of data (~300 GB's) from server 1 (10.1.1.100) to server 2 (10.1.1.101) & was wondering how I should transfer the data between servers? Do I use rsync? tar over ssh? scp?

Anyone have any recommendations on what is the easiest / fastest / secure method?
 
If you're not looking for security, tar over netcat is one of the fastest ways available. It will be cleartext though, so make sure you trust the network.

Edit: You might have a case of "Easy, fast, secure: Pick two" on your hands" :D
 
If I need to move that much, II take the drive out of one computer and put it in the other and then copy the data.

That's probably not the answer you were looking for.
 
Boomslang said:
If you're not looking for security, tar over netcat is one of the fastest ways available. It will be cleartext though, so make sure you trust the network.

Edit: You might have a case of "Easy, fast, secure: Pick two" on your hands" :D
Click to expand...

So how would that command look like if I would like to tar /builds directory and all the sub folders / files from server 1 to the /builds directory of server 2?

Jeff Rage said:
If I need to move that much, II take the drive out of one computer and put it in the other and then copy the data.

That's probably not the answer you were looking for.
Click to expand...

That wont work. Server 1 is SCSI (hot swapable) and server two is SAS.
 
Carloswill said:
So how would that command look like if I would like to tar /builds directory and all the sub folders / files from server 1 to the /builds directory of server 2?
Click to expand...

Destination:
$ cd / ; nc -l -p 8888 -q0 | tar x

Source:
$ cd / ; tar c builds | nc 10.1.1.101 8888 -q0

I do that all the time for large transfers on my home LAN, but, as a matter of good practice, never over any network I don't control (including, of course, the Internet). Use encryption for that; something like:

Source:
$ cd / ; tar c builds | ssh 10.1.1.101 'cd / && tar x'


If you need to do this repeatedly, use rsync.

-Corey
 
A tar pipe via netcat is definitely the fastest way to go.
bugfood said:
Destination:
$ cd / ; nc -l -p 8888 -q0 | tar x
Click to expand...
You might consider using the -p option for tar, depending on what the umask is for the user on the receiving end, and what sort of copy you need.
 
Does "netcat" retain file permissions like rsync? I have to admit netcat looks strange and confusing. How much more of benefit is using netcat over rsync?
 
Carloswill said:
Does "netcat" retain file permissions like rsync? I have to admit netcat looks strange and confusing. How much more of benefit is using netcat over rsync?
Click to expand...

netcat wont, but the -a (archive) switch to tar will.

the advantage of netcat over rsync is speed. it has alot less overhead. While this is minor when moving say.. 10meg, 300Gig that extra overhead in the data being sent starts increasing the overall transmission.
 
eeyrjmr said:
netcat wont, but the -a (archive) switch to tar will.
Click to expand...

Hmm. My GNU tar 1.19 doesn't have a -a option, but that kind of behavior is default. As far as I know the only files it won't copy are sockets, which is ok because socket-listening programs should recreate them before listening anyway.

the advantage of netcat over rsync is speed. it has alot less overhead. While this is minor when moving say.. 10meg, 300Gig that extra overhead in the data being sent starts increasing the overall transmission.
Click to expand...

Rsync calculates, sends, and compares MD5 checksum information of the files (or parts of files) that get transferred. This should catch any data corruption when it isn't detected by the TCP checksum. If you can live with that small but real possibility, and you don't want the small transmission overhead and substantial processing overhead (for fast links), use netcat.

Rsync's real claim to fame is dramatically speeding up copies when some of the data has already been transferred (e.g., an old copy or an interrupted copy). Only new data is actually transferred.

-Corey
 
Thanks for all the info guys. I will mess with netcat however I simply used the following rsync command which is working fine:

Code: 
[root@server_2 ~]# rsync root@server_1:/builds /builds/ --progress -e ssh -vax
 
rsync -a is what you want.

Edit: On second thought: Mount the disk from the old machine in the new one and copy it as a locally mounted filesystem.
 
I tend to use ssh for all of my server to server communications. It strikes a nice balance between security, speed, and ease of use for me. sshfs is especially handy.
 
doh said:
rsync -a is what you want.

Edit: On second thought: Mount the disk from the old machine in the new one and copy it as a locally mounted filesystem.
Click to expand...

Why would I want that -a?

Also someone else already suggested the mounting of the drive and copying it locally. Both servers have different types of drive mediums. One is SCSI and the other is SAS.
 
You could do this, if you want some security:
Code: 
tar -czpf - FILES_TO_TRANSFER | ssh USER@REMOTE_ADDRESS tar -xzpf - -C REMOTE_DIRECTORY
This works nicely for me, keep in mind that the 'z' flag means gzip. You might want to change this either use bzip2 or no compression at all. You can add -blowfish to the ssh command and you might crank out a little extra speed.
 
How about setting up a NFS server on the old server and mount it as any directory you want and start copying. Of course I wouldn't do that unless I had a gigabit connection, but that would probably be the easiest. secure, probably not.
 
doh said:
rsync -a is what you want.

Edit: On second thought: Mount the disk from the old machine in the new one and copy it as a locally mounted filesystem.
Click to expand...

We move dozens of gigabytes up to several terabytes of data around on our linux storage farm all the time, using exactly this method, rsync -avz from the source to the target with the target locally mounted. -z compresses the data before transferring it.
 
You must log in or register to reply here.
Back
Top