Within the Linux/Unix environment there are two highly versatile command line tools you can use to copy data from one server to another. These are SCP and RSYNC.
SCP
SCP (or Secure Copy) is protocol that can be used to transfer files across an ssh connection. It works in a similar manner to the cp command, and can accept the recursive -R switch to allow it to copy a directory, along with all of the contained subdirectories and files within.
The basic syntax to copy a file to an external host is:
scp filename user@host:destination
To copy a file from an external host we would use:
scp user@host:destination filename
If the remote host uses a port other than 22 for ssh connections, you can use the -P switch to specify the correct port. e.g.
scp -P2222 filename user@host:destination
So, if we wanted to copy the file hello.txt from fred's home directory, across to bert's home directory on the external host 10.0.0.1, which runs ssh on port 2222, we would use this command:
scp -P222 /home/fred/hello.txt bert@10.0.0.1:/home/bert
This will prompt us for berts password, unless we have exchanged ssh keys copied fred's public key to bert's list of authorised_keys. See the bottom of this post for information.
You can also rename the copied file, as shown below:
scp -P222 /home/fred/hello.txt bert@10.0.0.1:/home/bert/goodbye.txt
The file hello.txt is copied across to bert's home directory and saved as goodbye.txt.
RSYNC
Rsync is a fast and highly versatile file copying tool, used for synchronizing files and directories. It can copy data locally, or to & from an external machine using a remote shell. It's speed is largely due to the use of the delta-transfer algorithm, which only identifies parts of the source file that are identical to the destination file, and only transfers the difference, rather than the whole file. It was intended as a replacement for rcp and scp but, because of the effectiveness of the delta-transfer, it is most commonly used for backups and site/server migrations.
The basic command format is:
rsync
There are many options that can be added to this command, but to sync a local file to a remote machine we would normally use something like this:
rsync -ave ssh /sourcedir/* username@remotemachine.com:/destinationdir/
So, if we were trying to sync the home directory on our local server and the home directory on a remote machine on the 10.0.0.1 IP address, we would use the following:
rsync -ave ssh /home/* root@10.0.0.1:/home/
The options used are explained below:
-a, --archive :archive mode is recursive and preserves permissions.
-e, --rsh=COMMAND :specifies the remote shell - in this case 'ssh'. Always the last option, as it must be followed by the shell definition and then the remote connection string.
-v, --verbose :increase verbosity - generates onscreen messages, detailing what rsync is doing.
-z, --compress :compress file data during the transfer
Two useful options not shown in the example are:
-n, --dry-run :perform a trial run with no changes made
This allows you to test your rsync command before running it. It will generate a list of which files will be sync'ed, so you can check that everything is correct.
The second option allows you to define a different ssh port. By default, rsync will try to make an ssh connection to port 22. If ssh on your server runs on a different port, lets say port 2222, you will need to add "ssh -p2222" to the directive to specify the port (NB: the speech marks need to be included).
If we add both of these options to the previous example, we get:
rsync --dry-run -ave "ssh -p2222" /home/* root@10.0.0.1:/home/
It's common to use the full --dry-run switch, rather than just -n, as it can be easy to miss when nestled amongst the other single letter switches, which can lead to confusion if you don't realise you're running a dry run and not a live sync.
Note: The ssh connections will prompt for a password unless you exchange server keys. Generate a key by running: ssh-keygen -t rsa You will be prompted to enter a passphrase, after which it will generate a public key - called id_rsa.pub - inside your user's .ssh directory. If you're running this as root, it will be inside /root/.ssh/. The contents of this file need to be added to the .ssh/authorized_keys file on the target server.
