How to Move Files - ucsf-wynton/tutorials GitHub Wiki

FTP Client

If you prefer a GUI tool, you can use any number of FTP client apps to transfer files from your machine to Wynton:

• Fetch
• CyberDuck
• FileZilla

The FTP connection hostname would be a Wynton transfer node: dt1.wynton.ucsf.edu or dt2.wynton.ucsf.edu.

Note: Gladstone user have additional options for high-speed data transfers to/from Gladstone, local and Dropbox locations: See internal confluence docs.

RSYNC

By command line, you can transfer files from any machine to your Wynton home directory:

rsync filename.tsv [email protected]:~/

Dropbox

If you set the sharing permissions on a file (or folder) in Dropbox to allow anyone to view via a link, then copy the link and use wget from Wynton to get a local copy of the file:

wget "https://www.dropbox.com/s/sdfd89f0a8f9af8/my_file.tsv"

Note: You may want to remove ?dl=0 from the end of the link to preserve the output filename or use the -o parameter (see next examples).

If you are downloading a directory, note that Dropbox will automatically send it as a .zip archive. Here are some examples of downloading and unzipping a directory.wget:

wget "https://www.dropbox.com/sh/sdfd89f0a8f9af8/aasd0f98sda0f8d9f8sd?dl=1" -o my_folder.zip && unzip my_folder.zip -d my_folder

Using curl, you need to specify -L to follow redirects:

curl -L "https://www.dropbox.com/sh/sdfd89f0a8f9af8/aasd0f98sda0f8d9f8sd?dl=1" -o my_folder.zip && unzip my_folder.zip -d my_folder

Note: When downloading a directory, there may be a size limit ~1GB. To get around that, you can manually create a zip or tar.gz archive of the directory and download that as a single file. In some cases, it may help to change dl=0 to dl=1 in the share link before downloading.

Google Drive

For files of size less than 100K, wget works on drive links following the transformation...

From: https://drive.google.com/file/d/123ABC/view?usp=sharing
To: https://drive.google.com/uc?export=download&id=123ABC

SFTP

If you are copying files from an SFTP server, e.g., sequencing files from UCSF's CAT Core, log in to a Wynton data transfer node. Then, login to sftp as follows:

sftp [email protected]

Next, navigate to the directory where the files you want to copy are located. Finally, to copy them to your current directory on Wynton login node (the location from which you logged in to sftp), use the following command.

get -r *

Scratch Space

If you are copying files that will only be needed temporarily, for example as input to a job, then you have the option of copying them directly to a global scratch space at /wynton/scratch/. There is 492TiB of space available for this purpose. It is automatically cleared after 2 weeks, but you should go ahead and delete the files when you no longer need them.

Note: it is good practice to first create your own subdirectory here and copy to that

mkdir /wynton/scracth/my_own_space
scp filename.tsv [email protected]:/wynton/scratch/my_own_space

See documentation for more about local and global scratch spaces.