access_BomAccessDocumentation_bom conda_nwp pytools_nwp pytools27_installation - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki

Installing Env nwp_pytools27 on raijin

Installing packages from the environment file

Load the conda module and and go to the folder of the environment file

wml548@raijin5 ~]module use /projects/access/modules
wml548@raijin5 ~]module load miniconda3/4.4.9
wml548@raijin5 ~]$conda --version
conda 4.4.9
wml548@raijin5 ~]cd /projects/access/apps/conda

This is the environment files nwp_pytools27_1.0.0.yml. The env will be installed at /projects/access/apps/conda

conda env create --prefix /projects/access/apps/conda/nwp_pytools27 -f nwp_pytools27_1.0.0.yml

Installing mule and um-utils

mule and um-utils need to installed separately. Download the mule-2017.8.1 and um_utils-2017.8.1 from the following location

wml548@raijin1 tmp]$wget https://github.com/SciTools/mule/releases/download/2017.08.1/mule-2017.08.1.tar.gz
wml548@raijin1 tmp]$wget https://github.com/SciTools/um_utils/releases/download/2017.08.1/um_utils-2017.08.1.tar.gz

You need to untar/unzip the files to a specified location.

Install the two packages

cd $PATH_TO_mule-2017.8.1
wml548@raijin5 ~]module use /projects/access/modules
wml548@raijin5 ~]module load miniconda3/4.4.9
wml548@raijin1 conda]$conda activate /projects/access/apps/conda/nwp_pytools27
(/projects/access/apps/conda/nwp_pytools27) wml548@raijin1 mule]$python --version
Python 2.7.16 :: Anaconda, Inc.
(/projects/access/apps/conda/nwp_pytools27) wml548@raijin1 mule]$python setup.py install
...
Installed /projects/access/apps/conda/nwp_pytools27/lib/python2.7/site-packages/mule-2017.8.1-py2.7.egg
Processing dependencies for mule==2017.8.1
Finished processing dependencies for mule==2017.8.1

cd $PATH_TO_um-utils-2017.8.1
(/projects/access/apps/conda/nwp_pytools27) wml548@raijin1 um_utils]$python setup.py install
Installed /projects/access/apps/conda/nwp_pytools27/lib/python2.7/site-packages/um_utils-2017.8.1-py2.7.egg
Processing dependencies for um-utils==2017.8.1
Finished processing dependencies for um-utils==2017.8.1

Fixing cartopy pre-loaded path

cd /projects/access/apps/conda/nwp_pytools27/lib/python2.7/site-packages/cartopy

update __init__.py

config = {'pre_existing_data_dir': '',
          'data_dir': _data_dir,
          'repo_data_dir': os.path.join(os.path.dirname(__file__), 'data'),
          'downloaders': {},
          }

as

config = {'pre_existing_data_dir': '/projects/access/apps/conda/shared_data/cartopy/data',
          'data_dir': _data_dir,
          'repo_data_dir': os.path.join(os.path.dirname(__file__), 'data'),
          'downloaders': {},
          }

Fixing basemap POOJ_LIB path

Details see the https://github.com/matplotlib/basemap/issues/419.

cd  /projects/access/apps/conda/nwp_pytools27/lib64/python2.7/site-packages/mpl_toolkits/basemap/ 

# open the file __init__.py: ddd the following on line 155:
os.environ['PROJ_LIB'] = '/projects/access/apps/conda/nwp_pytools27/share/proj'

# just above the following line 
pyproj_datadir = os.environ['PROJ_LIB']

Check the list in the environment

wml548@raijin5 ~]module use /projects/access/modules
wml548@raijin5 ~]module load miniconda3/4.4.9
wml548@raijin1 conda]$conda activate /projects/access/apps/conda/nwp_pytools27
(/projects/access/apps/conda/nwp_pytools27) wml548@raijin1 um_utils]$conda list

Please check the [./nwp_pytools27_1.0.0_list list].

Set up the module

cd /projects/access/modules/conda
touch nwp_pytools27

nwp_pytools27 is as

#%Module
source /opt/Modules/extensions/extensions.tcl

set name "nwp_pytools27"
set version "1.1.0"
set install-contact "Wenming Lu <[email protected]>"
set install-date    "09-08-19"
set help "nwp_pytools27"
set url https://access-svn.nci.org.au/svn/

set prefix "/projects/access/apps/$name"

prepend-path PATH /projects/access/apps/conda/nwp_pytools27/bin

# Add line break after docommon to avoid warnings from module switch
source /projects/access/modules/common

Test

TO BE ADVISED BY Vinod

Possible issues

  1. conda may not be able to keep the consistency all the time; for example
conda env create --prefix /projects/access/apps/conda/nwp_pytools27 -f nwp_pytools27_1.0.0.yml

outputs error message the other day using the same conda and environment file

wml548@raijin1 conda]$/projects/access/apps/miniconda3/bin/conda env create --prefix /projects/access/apps/conda3/nwp_pytools27 -f nwp_pytools27_1.0.0.yml
Solving environment: failed

ResolvePackageNotFound: 
  - netcdf-fortran==4.4.4=h4363f12_9

At this stage, it is not sure the error message may vary.

  1. If having problems activating env, please add the following line to your.bashrc
. /projects/access/apps/miniconda3/etc/profile.d/conda.sh