Examples

download scripts

rerun all previous downloads recreating all download scripts from the ESGF API

This will recreate download scripts for all variable / experiment combinations noted in vars_all.txt and experiments_all.txt

cd /nird/home/jang/ESGF_download/bin
./get_dl_scripts.py --all

Because this will do a lot of work, you will see the following warning for 10 seconds before it starts:

(base) [jang@login-nird-3 bin]$ ./get_dl_scripts.py
WARNING: you have chosen to run ./get_dl_scripts.py without any command line option.
this will run with this script with the following defaults:
- variables are taken from file ./vars.txt
- experiments are taken from file ./experiments.txt
- directory for ESGF download scripts: /nird/home/u1/jang/ESGF_download/dl_scripts
- directory for data to download: /nird/projects/NS9252K/CMIP6
Silence this message using the -s switch.
This script will now sleep for 10 seconds.

At the end you will see some information lines that tell you how to run the download scripts like this:

2019-10-08 09:53:45,560:INFO:To run the just created scripts, you might want to run one of the following commands:
2019-10-08 09:53:45,560:INFO:/usr/bin/parallel -v -j 8 -a runfile_20191008095314_c11302b8-c441-4e6e-a7fc-e23ec9e2dd9d_login-nird-1.run
2019-10-08 09:53:45,562:INFO:screen -S CMIP8.dl /usr/bin/parallel -v -j 8 -a runfile_20191008095314_c11302b8-c441-4e6e-a7fc-e23ec9e2dd9d_login-nird-1.run
(base) [jang@login-nird-1 bin]$ screen -S CMIP8.dl /usr/bin/parallel -v -j 8 -a runfile_20191008095314_c11302b8-c441-4e6e-a7fc-e23ec9e2dd9d_login-nird-1.run

get download scripts for additional variables

./get_dl_scripts.py --variables areacella

This will get the download scripts fro the variable areacella for all experiments noted in the experiments_all.txt

general interface for get_dl_scripts.py

(base) [jang@login-nird-3 bin]$ ./get_dl_scripts.py -h
usage: get_dl_scripts.py [-h] [--all] [--allvariants]
                         [--downloaddir DOWNLOADDIR]
                         [--experimentfile EXPERIMENTFILE]
                         [--experiments EXPERIMENTS [EXPERIMENTS ...]]
                         [-o OUTFILE] [--scriptdir SCRIPTDIR]
                         [--logdir LOGDIR] [--logfile LOGFILE]
                         [--runfile RUNFILE] [-s] [-v] [--varfile VARFILE]
                         [--variables VARIABLES [VARIABLES ...]]

command line interface to get_dl_scripts.py

optional arguments:
  -h, --help            show this help message and exit
  --all                 read variables and experiments from files containing
                        all experiments and variables
  --allvariants         query all realisations, not just r1*
  --downloaddir DOWNLOADDIR
                        download directory of the to be downloaded files;
                        defaults to /nird/projects/NS9252K/CMIP6/
  --experimentfile EXPERIMENTFILE
                        read experiments from file
  --experiments EXPERIMENTS [EXPERIMENTS ...]
                        experiments to query
  -o OUTFILE, --outfile OUTFILE
                        name of the runfile
  --scriptdir SCRIPTDIR
                        output directory for the downloaded download scripts;
                        defaults to ../dl_scripts/
  --logdir LOGDIR       output directory for log files; defaults to
                        /nird/home/jang/logs
  --logfile LOGFILE     logfile; defaults to
                        /nird/home/jang/logs/get_dl_scripts_20191008102649.log
  --runfile RUNFILE     runfile; defaults to runfile_20191008102649_ec85df7c-b
                        c69-4d3d-9451-130b474fa004_login-nird-3.run
  -s, --silence         silence warning message if called without parameter
  -v, --verbose         switch on verbosity
  --varfile VARFILE     read variables from file
  --variables VARIABLES [VARIABLES ...]
                        variables to query

run with parallel