Using sharing server for eQTLGen analyses - molgenis/systemsgenetics GitHub Wiki

In this extended manual we give instructions how to access and use our cher-ami.hpc.rug.nl sharing server for:

  1. Downloading the tools and files needed for running the trans/PRS-eQTL analyses to your own analysis server (shared account: umcg-eqtlgen).

  2. Uploading your result files to your quest account in cher-ami servers (individual upload account: umcg-guest[0-9]).

Some of the usage instructions are already documented here: http://wiki.gcc.rug.nl/wiki/DataSharing

There may be differences in the availabilities/functionalities of different download/upload tools according to your local cluster setup. Here we demonstrate the usage of two commonly used UNIX tools: lftp and sftp.

If lftp is not installed into your enviroment, then opt for sftp. If you are experiencing problems in using sftp, please also check the functionality of tools such as rsync and scp.

In case additional help is needed please contact to Helpdesk dot GCC dot Groningen at gmail dot com with cc: urmo.vosa @ gmail.com.

Keep in mind that your private key has to be accessible for those tools in your current work environment to connect with cher-ami accounts. Therefore, if you private key is on your local desktop and you are using (trusted and secure) remote analysis server for analysis, you should use additional flag -A in your ssh command when connecting to your analysis server.

Downloading the files needed for analysis

Using lftp

        # go to the local folder where you want to run analysis:

        cd local/analysis/folder/

        # start lftp

        lftp

        # connect with the shared eQTLGen account:

        lftp :~> open -u umcg-eqtlgen,none -p 22 sftp://cher-ami.hpc.rug.nl

        # download needed folder:

        lftp :~> mirror folder_needed_for_analysis

        # download single file needed for analyses:

        lftp :~> get file_needed_for_analysis

        # exit from the remote server:

        lftp :~> exit

Using sftp

Please note that flag -r does not work for all cluster setups.

        # go to the local folder where you want to run analysis:

        cd local/analysis/folder/

        # connect to the shared eQTLGen account: 

        sftp -r [email protected]

        # download folder needed for analysis:

        sftp> get –r folder_needed_for_analysis

        # download individual file needed for analysis:

        sftp> get file_needed_for_analysis

        # exit from the remote server

        sftp> bye

Uploading the output files of finished analyses

The folder structure for delivering the results is:

|--[NameOfCohort1]
   |--trans_eQTL_results
      |--trans_eQTL_results_PC_corrected
      |--trans_eQTL_results_PC_uncorrected
   |--PRS_eQTL_results
      |--PRS_eQTL_results_PC_corrected
      |--PRS_eQTL_results_PC_uncorrected
|--[NameOfCohort2]
   |--trans_eQTL_results
      |--trans_eQTL_results_PC_corrected
      |--trans_eQTL_results_PC_uncorrected
   |--PRS_eQTL_results
      |--PRS_eQTL_results_PC_corrected
      |--PRS_eQTL_results_PC_uncorrected

Using lftp

        # go to the local folder where data is stored:

        cd local/folder/with/your/data/

        # start lftp

        lftp

        # connect with your guest account:

        lftp :~> open -u [your_guest_accountname],none -p 22 sftp://cher-ami.hpc.rug.nl

        # make remote upload directories and subdirectories for trans-eQTL and PRS-QTL results. e.g:

        lftp :~> mkdir LIFEa1
        lftp :~> mkdir LIFEa1/trans_eQTL_results
        lftp :~> mkdir LIFEa1/trans_eQTL_results/trans_eQTL_results_PC_corrected
        lftp :~> mkdir LIFEa1/trans_eQTL_results/trans_eQTL_results_PC_uncorrected
        lftp :~> mkdir LIFEa1/PRS_eQTL_results
        lftp :~> mkdir LIFEa1/PRS_eQTL_results/PRS_eQTL_results_PC_corrected
        lftp :~> mkdir LIFEa1/PRS_eQTL_results/PRS_eQTL_results_PC_uncorrected
        
        # go to the folder where you upload the results. e.g:

        lftp :~> cd LIFEa1/trans_eQTL_results/trans_eQTL_results_PC_corrected       
 
        # use put command for uploading the individual files and/or mirror command for uploading whole directories:

        lftp :~> put individual_result_file_in_local_server
        lftp :~> mirror -R result_folder_in_local_server
        
        # exit from the remote server when all necessary files are uploaded:

        lftp :~> exit

Using sftp

Please note that flag -r does not work for all cluster setups.

        # go to the local folder where data is stored:

        cd local/folder/with/your/data/

        # connect to the shared eQTLGen account: 

        sftp -r [your_guest_accountname]@cher-ami.hpc.rug.nl

         # make remote upload directories and subdirectories for trans-eQTL and PRS-QTL results. e.g:

        sftp> mkdir LIFEa1
        sftp> mkdir LIFEa1/trans_eQTL_results
        sftp> mkdir LIFEa1/trans_eQTL_results/trans_eQTL_results_PC_corrected
        sftp> mkdir LIFEa1/trans_eQTL_results/trans_eQTL_results_PC_uncorrected
        sftp> mkdir LIFEa1/PRS_eQTL_results
        sftp> mkdir LIFEa1/PRS_eQTL_results/PRS_eQTL_results_PC_corrected
        sftp> mkdir LIFEa1/PRS_eQTL_results/PRS_eQTL_results_PC_uncorrected

        # upload the folder with results:

        sftp> put –r result_folder_in_local_server

        # upload individual result file:

        sftp> put individual_result_file_in_local_server

        # exit from the remote server when all necessary files are uploaded:

        sftp> bye