Using Google SDK To Manage Storage - steviep42/immport GitHub Wiki

Working with Project Storage

Once you have a working SDK initialized you can now issue commands within your terminal window to manage the storage on the Immport project. On GCP this is known as "reference database" project. If you haven't used Amazon or Google before then the commands to work with storage are a little different from the UNIX command line although related. There are a couple of ways to "sync" (synchronize) folders between the Google resources and your local laptop. The definitive reference for the gsutil command and what you can do with it relative to storage management can be found here although I will present a couple of examples specific to the Immport project.

List the contents of the immport folder

$ gsutil ls gs://immport
  gs://immport/Immport Data Sets/
  gs://immport/Shuzhao-prototype-22studies/
  gs://immport/Shuzhao_example_SDY80/
  gs://immport/Shuzhao_example_VZV
  gs://immport/tmp_to_organize/

How much disk space is used by a folder on the Google project

 $ gsutil du -sh gs://immport/Shuzhao_example_SDY80
   113 MiB     gs://immport/Shuzhao_example_SDY80

Recursively list the contents of a folder

 $ gsutil ls -r gs://immport/Shuzhao_example_SDY80
   gs://immport/Shuzhao_example_SDY80/:
   gs://immport/Shuzhao_example_SDY80/LOG_fcs_20190102.txt
   gs://immport/Shuzhao_example_SDY80/README.txt
   gs://immport/Shuzhao_example_SDY80/SDY80_pH1N1_2009_Transcriptomics.txt
   gs://immport/Shuzhao_example_SDY80/biosample.txt
   gs://immport/Shuzhao_example_SDY80/neut_ab_titer__data_matrix.txt
   gs://immport/Shuzhao_example_SDY80/neut_ab_titer__observation_annotation.txt
   ..
   ..
   gs://immport/Shuzhao_example_SDY80/datasrc/fcs_analyzed_result_2018-04-02_07-02-30.tsv
   gs://immport/Shuzhao_example_SDY80/datasrc/neut_ab_titer_2018-04-02_07-00-43.tsv

Copy a project folder down to your local hard drive

   $ gsutil -m cp -r gs://immport/Shuzhao_example_SDY80 .
   Copying gs://immport/Shuzhao_example_SDY80/LOG_fcs_20190102.txt...
   Copying gs://immport/Shuzhao_example_SDY80/README.txt...                        
   Copying gs://immport/Shuzhao_example_SDY80/SDY80_pH1N1_2009_Transcriptomics.txt...
   Copying gs://immport/Shuzhao_example_SDY80/archived/subjLabel_BTM_SDY80_pH1N1_2009_Transcriptomics.txt...