Data transfer - Green-Biome-Institute/Xsede GitHub Wiki

Now you have your data to be analyzed, first you need to put in on Xsede servers. You do not want to transfer your data to your HOME directory as that directory is only 25GB. This is a space for keeping your shell scripts and other small important files, as this space is backed up. Checking the storage spaces you can use can be done by command:

$ projects

which will tell you where you can store data. It likely be here: /ocean/projects/bio210016p/your_user_name/

There are many ways to transfer files, we will start with scp.

Secure copy protocol(scp)

This is an easy and quick way to transfer files between Xsede and your local computer. This is for transfering your files to Xsede from you local source:

$ scp -P 2222 directory_on_local_computer [email protected]:directory_on_xsede_to_transfer

Likewise, you can transfer files from Xsede to you local computer:

$ scp -P 2222 [email protected]:directory_on_xsede_with_Data your_local_directory

You will be prompted to your password and authentication through duo as if you are logging in. You can transfer whole folders or files with scp. For specific files you can do:

$ scp -P 2222 directory_on_local_computer/**file_name** [email protected]:directory_on_xsede_to_transfer

Likewise, you can transfer files from Xsede to you local computer:

$ scp -P 2222 [email protected]:directory_on_xsede_with_Data/**file_name** your_local_directory

You can also use wild cards "*" for transferring a specific type of data, such as files that start with say "GBI" like this:

$ scp -P 2222 [email protected]:directory_on_xsede_with_Data/GBI* your_local_directory

For more information on wild cards, refer to here.

scp is not great for larger files (>5GB). The reason for this is, scp cannot pick up where it left off if your connection is lost. So you might end up up/downloading same files over and over again. This is not very optimal especially if your connection is not stable. For larger files check out rsync:

rsync

rsync is great for transferring large files and basically syncing 2 folders. It works similarly to scp but if your connection is lost, the transfer continues from where you left off. You will need to enter the command again once you have an internet connection. To transfer from your local computer to Xsede, do:

rsync -rltpDvp -e 'ssh -l your_userName' source_directory data.bridges2.psc.edu:target_directory

To transfer file from Xsede, do:

rsync -rltpDvp -e 'ssh -l your_userName' data.bridges2.psc.edu:target_directory source_directory

When I was doing this, rsync did not work with my Xsede password. Instead I needed to create an account with PSC directly here. With this new password, you can enter it when prompted for rsync and voila!

With rsync you will not see the progress as you would with scp, but your files are transferring, trust me! But if you don't, you can always go to the directory where you are putting your files and check the status.