1.2 Transferring data - UNR-HPC/pronghorn GitHub Wiki

Pronghorn supports the following transfer modes: SFTP, SCP, and Rsync.

Transferring files via SCP

The scp command allows files to be copied to, from, or between different hosts. It uses ssh for data transfer and provides the same authentication and same level of security as ssh.

Examples

  • Copy the file "example.txt" from a remote host to the local host
scp [email protected]:example.txt /some/local/directory
  • Copy the file "example.txt" from the local host to a remote host
scp example.txt [email protected]:/some/remote/directory
  • Copy the directory "example_a" from the local host to a remote host's directory "example_b"
scp -r example_a [email protected]:/some/remote/directory/example_b
  • Copy the file "example.txt" from remote host "remotehost_1.edu" to remote host "remotehost_2.edu"
scp your_username@remotehost_1.edu.edu:/some/remote/directory/example.txt your_username@remotehost_2.edu:/some/remote/directory/
  • Copy the files "example_1.txt" and "example_2.txt" from the local host to your home directory on the remote host
scp example_1.txt example_2.txt [email protected]:~
  • Copy an entire directory from one host to another use the -r switch and specify the directory
scp -r /some/local/directory [email protected]:/some/remote/directory

Syncing files via Rsync

SCP will always overwrite existing files. So it's a handy utility to transfer new files. Rsync will compare files locally and remotely and only transfer the bits that are different, thereby saving time. Rsync is also able to preserve file permissions and compress files in realtime as the transfer is taking place.

Rsync provides many options for passing arguments to augment it's functionality. In the examples below we will use:

  • -a flag: Stands for "archive" and syncs recursively and preserves symbolic links, special and device files, modification times, group, owner, and permissions.
  • -z flag: Compresses files during transfer.
  • -P flag: Combines the flags --progress and --partial. The first of these gives you a progress bar for the transfers and the second allows you to resume interrupted transfers:

The following command is a "push" operation because it pushes (syncs) a local directory to a remote system.

rsync -azP ~/some/local/source/directory username@remotehost:/some/remote/destination/directory

For Pronghorn the command might look more like this:

rsync -azP ~/some/local/source/directory [email protected]:/data/gpfs/home/username/destination_directory

Conversely, a "pull" operation is used to sync a remote directory to the local system.

rsync -azP username@remote_host:/some/remote/destination/directory ~/some/local/directory

For Pronghorn the command might look more like this:

rsync -azP [email protected]:/data/gpfs/home/username/source_directory ~/some/local/directory

Other Applications

There's a multitude of software applications that can be used for file transfers.

Popular options include:

Windows

macOS

Linux

  • Use the terminal!