Transferring data with SFTP

Network Attached Storage (NAS), such as sosiknas1, are sometimes not configured with rsync; a good alternative then is SFTP. With sftp, the “Local” machine is the device that initiates the sftp connection and the “Remote” machine is the device being connected to. The main commands used here are mget to get files from Remote to the Local machine, and mput to put files from Local to the Remote device.

Commands below are assumed to be issued from the HPC.

Note: If the dataset is large (>10GB or >10k files), connect to a dedicated node so as not to bog-down the default login node.

• srun -p interactive -N 1 -n 1 --mem=1gb --pty bash

From Sosiknas to HPC

1. Change Directory to whatever directory you want to transfer data into
   • cd path/to/local/wherever
2. Connect to the NAS via sftp, example:
   • sftp [email protected]
   • Enter password
   • Your command prompt should now be sftp>, indicating that you are in the sftp interface
3. Change Directory to the parent directory of the folder you want to send to hpc
   • cd path/to/remote/wherever
   • See SFTP Navigation Tips below
4. Use the mget command (with -r for recursion) to transfer files to Local
   • mget -r dir_on_remote_to_transfer
5. Exit sftp. You are done.
   • bye

From HPC to Sosiknas

Steps are very similar to transferring to the HPC, so use the same commands as above where appropriate. The main difference is to use mput, instead of mget. ssh into the HPC

1. Change Directory to the parent directory of the folder you want to transfer
2. Connect to the NAS via sftp
3. Change Directory to the NAS directory you want the hpc folder to be transferred into
4. Use the mput command (with -r for recursion) to transfer files from Local
   • mput -r dir_on_local_to_transfer
5. Exit sftp. You are done.

Advanced: One-liner

It's possible to issue multiple commands as a one-liner using the schema below, where individual commands are deliminated by \n. A user still needs to enter a password after this one-liner is issued (unless the have ssh keys already set up).

echo -e "CMD1\nCMD2" | sftp user@host

From SosikNAS to HPC Example

• echo -e "lcd dir/on/hpc/to/transfer/into\ncd /dir/on/sosiknas/to/transfer/from\nmget -r dir_on_remote_to_transfer" | sftp user@sosiknas

From HPC to SosikNAS Example

• echo -e "lcd dir/on/hpc/to/transfer/from\ncd /dir/on/sosiknas/to/transfer/into\nmput -r dir_on_local_to_transfer" | sftp user@sosiknas

SFTP Navigation Tips

When connected to a remote device, it is still possible to navigate the filesystem of the local machine. Commands like “cd, ls, pwd, mkdir” that work on the remote device can also be used on the local machine by prepending the letter “l” (lowercase L) to the commands.

cd lcd: these are used to change directories on the remote/local machine

ls lls: these show the content of the remote/local directory you are in

pwd lpwd: these tell me the full path of the remote/local directory you are in

mkdir lmkdir: make a new folder in the remote/local machine’s current directory.

help: this lists commands available to you in the sftp interface