NWPTOOLS_PY2

Installing Env nwptools_py2 on Gadi

Note this env used to be named as nwp_pytools27 on Raijin.

Installing packages from the environment file

Load the conda module

$>module use /projects/access/modules
$>module load miniconda/4.6
$>conda --version
conda 4.6.14

Go to the folder of the environment file and produce the env

cd /projects/access/apps/miniconda/yml
conda env create -f nwptools_py2.201911.yml

The env will be produced by default in the location of /projects/access/apps/miniconda/4.6/envs/ and given the name as nwptools_py2.201911 which is specified in the env file nwptools_py2.201911.yml.

Installing mule and um-utils

mule and um-utils need to installed separately. Download the mule-2017.8.1 and um_utils-2017.8.1 from the following location

$>wget https://github.com/SciTools/mule/releases/download/2017.08.1/mule-2017.08.1.tar.gz
$>wget https://github.com/SciTools/um_utils/releases/download/2017.08.1/um_utils-2017.08.1.tar.gz

You need to untar/unzip the files to a specified location.

Install the two packages

$>cd $PATH_TO_mule-2017.8.1
$>module use /projects/access/modules
$>module load miniconda/4.6
$>conda activate nwptools_py2.201911
$>python --version
Python 2.7.16 :: Anaconda, Inc.
$>python setup.py install
/projects/access/apps/miniconda/4.6/envs/nwptools_py2.201911/lib/python2.7/site-packages/setuptools/dist.py:472: UserWarning: Normalizing '2017.08.1' to '2017.8.1'
  normalized_version,
running install
...
Installed /scratch/access/apps/miniconda/4.6/envs/nwptools_py2.201911/lib/python2.7/site-packages/mule-2017.8.1-py2.7.egg
Processing dependencies for mule==2017.8.1
Finished processing dependencies for mule==2017.8.1

$>cd $PATH_TO_um-utils-2017.8.1
$>python setup.py install
/projects/access/apps/miniconda/4.6/envs/nwptools_py2.201911/lib/python2.7/site-packages/setuptools/dist.py:472: UserWarning: Normalizing '2017.08.1' to '2017.8.1'
  normalized_version,
running install
...
Installed /scratch/access/apps/miniconda/4.6/envs/nwptools_py2.201911/lib/python2.7/site-packages/um_utils-2017.8.1-py2.7.egg
Processing dependencies for um-utils==2017.8.1
Finished processing dependencies for um-utils==2017.8.1

Fixing cartopy pre-loaded path

$>cd /projects/access/apps/miniconda/4.6/envs/nwptools_py2.201911/lib/python2.7/site-packages/cartopy

update __init__.py

config = {'pre_existing_data_dir': '',
          'data_dir': _data_dir,
          'repo_data_dir': os.path.join(os.path.dirname(__file__), 'data'),
          'downloaders': {},
          }

as

config = {'pre_existing_data_dir': '/projects/access/bom/shared_data/cartopy/data',
          'data_dir': _data_dir,
          'repo_data_dir': os.path.join(os.path.dirname(__file__), 'data'),
          'downloaders': {},
          }

Fixing basemap PROJ_LIB path

Details see the https://github.com/matplotlib/basemap/issues/419.

$>cd  /projects/access/apps/miniconda/4.6/envs/nwptools_py2.201911/lib/python2.7/site-packages/mpl_toolkits/basemap/ 

# open the file __init__.py: add the following on line 155:
os.environ['PROJ_LIB'] = '/projects/access/apps/miniconda/4.6/envs/nwptools_py2.201911/share/proj'

# just above the following line 
pyproj_datadir = os.environ['PROJ_LIB']

Check the list in the environment

$>module use /projects/access/modules
$>module load miniconda/4.6
$>conda activate nwptools_py2.201911
(nwptools_py2.201911)$>conda list

Set up the module

$>cd /projects/access/modules
$>mkdir -p nwptools_py2
$>cd nwptools_py2
$>touch 27201911

The module file27201911 is as

#%Module
source /opt/Modules/extensions/extensions.tcl

set name "nwptools_py2"
set version "201911"
set install-contact "Wenming Lu <[email protected]>"
set install-date    "2019-11-27"
set help "nwptools_py2"
set url https://access-svn.nci.org.au/svn/

set conda_ver 4.6
set prefix "/projects/access/apps/miniconda/${conda_ver}/envs/"

prepend-path PYTHONPATH /projects/access/apps/miniconda/latest/envs/esmf_7.1/lib/python2.7/site-packages
prepend-path PATH "$prefix/${name}.${version}/bin"

# Add line break after docommon to avoid warnings from module switch
source /projects/access/modules/common

Note that an external ESMF is linked to make the module work within nwptools_py2.201911.

Test

$>module use /projects/access/modules
$>module load nwptools_py2/201911
 python
Python 2.7.16 |Anaconda, Inc.| (default, Mar 14 2019, 21:00:58) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import iris
import mule/projects/access/apps/miniconda/4.6/envs/nwptools_py2.201911/lib/python2.7/site-packages/iris/fileformats/grib/__init__.py:59: IrisDeprecation: The module iris.fileformats.grib is deprecated since v1.10. Please install the package 'iris_grib' package instead.
  "The module iris.fileformats.grib is deprecated since v1.10. "
>>> import mule
>>> import netcdftime
>>> import um_utils  
>>> import ESMF
>>> import numpy
>>> import scipy
>>> import cartopy
>>> import matplotlib
>>> import mpl_toolkits
>>> import mpl_toolkits.basemap
>>> 

More complicated tests will be done by Vinod.

Issues

import gdal fails