access_BomAccessDocumentation_bom conda_nwp pytools_nwp pytools27_development - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki

Development notes for nwp_pytools27

The following is an initial list of packages for conda/nwp_pytools27 environment, given in the format of yml file. There are obviously a few packages that are not in the channels. Claire Carouge from the CoE recommend we create our own channel. Otherwise, this has to be installed out of the source code directly.

There are a few things to be noted:

1. The mule package in the anaconda cloud is quite old (vn 1.2) and the um_utils does not exists. So these packages were installed manually from the source files downloaded off the GitHub.The source files I downloaded (vn 2017.08) can be found under:

```
Raijin:/g/data/dp9/vvk548/softwares/sources
```

To install them into a conda environment, activate the conda environment, extract the gzip file somewhere in your personal space and then install them using the setup.py fil. i.e., go to the directory where the sources are unzipped, and type:

```
module load conda/python27 (we use the conda env already installed by CoE)
conda activate <conda env name>
conda remove mule (if you have them installed already)
python setup.py install 
```

Do the same for um_utils. You can verify the install by "conda list".

2. The iris version that works with both GES and RES is found to be iris=1.13.0. Later versions of iris (tested 2.0.0) were found to have a compatibility issue with numpy by calling the dask module. Basically, later version of iris calls dask which inturn calls numpy.nancumprod function, available only on numpy 1.13.0 or later. However, upgrading to a later version (numpy1.16.4) resulted in an unstable installation with further dependency issues. Hence, we reverted back to the earlier numpy=1.11.3 version. However, the iris=1.13.0 is not compatible with eccodes and cdo packages available via the conda-forge channel. Hence we had to omit these two modules from our installation.

name: nwp_pytools27
channels:
  - jochym
  - scitools/label/archive
  - terradue
  - auto
  - conda-forge
  - defaults
dependencies:
  - _r-mutex=1.0.0=anacondar_1
  - affine=2.2.2=py27_0
  - agg-regrid=0.2.2=py27h637b7d7_1000
  - alembic=1.0.10=py_0
  - appdirs=1.4.3=py27h28b3542_0
  - arm_pyart=1.10.1=py27hd352d35_0
  - arrow=0.13.1=py27_0
  - asciitree=0.3.3=py_2
  - asn1crypto=0.24.0=py27_0
  - atomicwrites=1.3.0=py27_1
  - attrs=19.1.0=py27_1
  - backports=1.0=py_2
  - backports.functools_lru_cache=1.5=py_2
  - backports.lzma=0.0.13=py27hd36a4d9_0
  - backports.os=0.1.1=py27_0
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.ssl_match_hostname=3.4.0.2=py27_0
  - backports.tempfile=1.0=py_1
  - backports.weakref=1.0.post1=py_1
  - backports_abc=0.5=py_0
  - basemap=1.2.0=py27h673bf1a_2
  - basemap-data-hires=1.2.0=0
  - beautifulsoup4=4.7.1=py27_1
  - biggus=0.15.0=py_1
  - binutils_impl_linux-64=2.31.1=h6176602_1
  - binutils_linux-64=2.31.1=h6176602_7
  - blas=1.1=openblas
  - bleach=3.1.0=py27_0
  - blosc=1.16.3=hd408876_0
  - bokeh=1.2.0=py27_0
  - boost-cpp=1.67.0=h14c3975_4
  - boto3=1.9.162=py_0
  - botocore=1.12.163=py_0
  - bottleneck=1.2.1=py27h035aef0_1
  - brewer2mpl=1.4.1=py_3
  - bsddb=1.0=py27_1
  - bwidget=1.9.11=1
  - bzip2=1.0.6=h14c3975_5
  - ca-certificates=2019.5.15=0
  - cairo=1.12.18=6
  - cartopy=0.17.0=py27hbb7e04d_1
  - cdat-lite=6.0.1=py27hdea14a1_5
 - cdat_info=8.1.1=py_2
  - cdtime=3.1.2=py27h6091dcd_1
  - certifi=2019.6.16=py27_0
  - cf-units=2.1.1=py27h3010b51_1000
  - cf_units=1.2.0=py27_0
  - cfchecker=3.1.1=py27h24bf2e0_0
  - cffi=1.12.3=py27h2e261b9_0
  - cftime=1.0.3.4=py27hdd07704_1
  - cfunits=3.0.0=py_0
  - chardet=3.0.4=py27_1
  - click=7.0=py27_0
  - click-plugins=1.1.1=py_0
  - cligj=0.5.0=py27_0
  - cloudpickle=1.1.1=py_0
  - cmarkgfm=0.4.2=py27h7b6447c_0
  - cmocean=2.0=py_0
  - colorcet=2.0.1=py_0
  - compliance-checker=3.0.2=py27_0
  - configobj=5.0.6=py27_1
  - configparser=3.7.4=py27_0
  - contextlib2=0.5.5=py27_0
  - cryptography=2.7=py27h72c5cf5_0
  - cryptography-vectors=2.7=py_0
  - curl=7.64.1=hbc83047_0
  - cycler=0.10.0=py27_0
  - cyordereddict=1.0.0=py27h14c3975_1002
  - cython=0.29.10=py27he6710b0_0
  - cytoolz=0.9.0.1=py27h14c3975_1
  - dask=1.2.2=py_0
  - dask-core=1.2.2=py_0
  - datashader=0.7.0=py_0
  - datashape=0.5.4=py27_1
  - db=5.3.28=1
  - dbus=1.13.6=he372182_0
  - decorator=4.4.0=py27_1
  - defusedxml=0.6.0=py_0
  - descartes=1.1.0=py27_0
  - distarray=2.12.2=py_1
  - distributed=1.28.1=py27_0
  - docopt=0.6.2=py27_0
  - docutils=0.14=py27_0
  - ecmwf-api-client=1.4=py27_0
  - ecmwf_grib=1.14.7=2
  - ecmwfapi=1.3=py27_0
  - entrypoints=0.3=py27_0
  - enum34=1.1.6=py27_1
  - esgf-pyclient=0.1.8=py27_1001
  - esmf=7.1.0r=ha678704_2
  - esmpy=7.1.0=py27h24bf2e0_3
  - expat=2.2.6=he6710b0_0
  - f90nml=1.0.2=py27_1000
  - fastcache=1.1.0=py27h7b6447c_0
  - fasteners=0.14.1=py_3
  - ffmpeg=3.4.1=0
  - fftw=3.3.8=h7b6447c_3
  - fiona=1.7.0=np111py27_0
  - fontconfig=2.11.1=6
  - freeglut=3.0.0=hf484d3e_5
  - freetype=2.5.5=2
  - freexl=1.0.5=h14c3975_0
  - fs=2.4.5=py27_0
  - funcsigs=1.0.2=py27_0
  - functools32=3.2.3.2=py27_1
  - future=0.17.1=py27_0
  - futures=3.2.0=py27_0
  - g2clib=1.6.0=h838ce51_4
  - gcc_impl_linux-64=7.3.0=habb00fd_1
  - gcc_linux-64=7.3.0=h553295d_7
  - gdal=2.0.0=py27_1
  - geopandas=0.4.1=py_0
  - geos=3.7.1=he6710b0_0
  - geotiff=1.4.2=h9c7010b_0
  - gettext=0.19.8.1=hd7bead4_3
  - gfortran_impl_linux-64=7.3.0=hdf63c60_1
  - gfortran_linux-64=7.3.0=h553295d_7
  - giflib=5.1.4=h14c3975_1
  - glib=2.58.3=hf63aee3_1001
  - gmp=6.1.2=h6c8ec71_1
  - gmpy2=2.0.8=py27h10f8cd9_2
  - gnutls=3.6.5=hd3a4fd2_1002
  - gprof2dot=2017.9.19=py_0
  - graphite2=1.3.13=h23475e2_0
  - graphviz=2.38.0=0
  - gsl=2.4=blas_openblash5c7cb9b_1004
  - gst-plugins-base=1.14.4=hdf3bae2_1001
  - gstreamer=1.14.4=h66beb1c_1001
  - gsw=3.0.6=py27_0
  - gunicorn=19.9.0=py27_0
  - gxx_impl_linux-64=7.3.0=hdf63c60_1
  - gxx_linux-64=7.3.0=h553295d_7
  - h5netcdf=0.7.4=py_0
  - h5py=2.8.0=py27h989c5e5_3
  - harfbuzz=0.9.39=1
  - hdf4=4.2.13=h3ca952b_2
  - hdf5=1.10.2=hba1933b_1
  - hdfeos2=2.20=h64bfcee_0
  - hdfeos5=5.1.16=h647bee3_2
  - heapdict=1.0.0=py27_2
  - html5lib=1.0.1=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.8=py27_0
  - image-meta-tag=0.7.8=py27_1000
  - imagehash=4.0=py_0
  - imageio=2.5.0=py27_0
  - importlib_metadata=0.17=py27_1
  - ipaddress=1.0.22=py27_0
  - ipykernel=4.10.0=py27_0
  - ipython=5.8.0=py27_0
  - ipython_genutils=0.2.0=py27_0
  - ipywidgets=7.4.2=py27_0
  - iris=1.13.0=py27h24bf2e0_2
  - iris-grib=0.12.0=py27_0
  - iris_grib=0.9.1=py27_1
  - isodate=0.6.0=py27_0
  - jasper=1.900.1=hd497a04_4
  - jbig=2.1=hdba287a_0
  - jinja2=2.10.1=py27_0
  - jmespath=0.9.4=py_0
  - joblib=0.13.2=py27_0
  - jpeg=9c=h14c3975_1001
  - json-c=0.13.1=h1bed415_0
  - jsoncpp=1.8.4=hfd86e86_0
  - jsonschema=3.0.1=py27_0
  - jupyter=1.0.0=py27_7
  - jupyter_client=5.2.4=py27_0
  - jupyter_console=5.2.0=py27_1
  - jupyter_core=4.4.0=py27_0
  - kealib=1.4.7=h77bc034_6
  - kiwisolver=1.1.0=py27he6710b0_0
  - krb5=1.16.3=h05b26f9_1001
  - lame=3.100=h14c3975_1001
  - lcms=1.19=0
  - libaec=1.0.4=hf484d3e_0
  - libblas=3.8.0=5_h6e990d7_netlib
  - libboost=1.67.0=h46d08c1_4
  - libcblas=3.8.0=5_h6e990d7_netlib
  - libcdms=3.0.1=hcacfb3f_4
  - libcf=1.0.2=py27h6e3784b_1005
  - libcurl=7.64.1=h20c2e04_0
  - libdap4=3.19.0=h8c95237_1
  - libdrs=3.1.0=h6e3784b_1000
  - libdrs_f=3.1.2=h0ebed7e_2
  - libedit=3.1.20181209=hc058e9b_0
  - libffi=3.2.1=hd88cf55_4
  - libgcc=7.2.0=h69d50b8_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgdal=2.0.0=1
  - libgfortran=3.0.0=1
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libiconv=1.15=h63c8f33_5
  - libkml=1.3.0=h590aaf7_4
  - liblapack=3.8.0=5_h6e990d7_netlib
  - libmagic=5.36=h14c3975_0
  - libmo_unpack=3.1.2=hf484d3e_1001
  - libnetcdf=4.6.1=h10edf3e_2
  - libogg=1.3.2=h7b6447c_0
  - libopenblas=0.2.20=h9ac9557_7
  - libopus=1.3=h7b6447c_0
  - libpng=1.6.37=hbc83047_0
  - libpq=11.2=h20c2e04_0
  - libsodium=1.0.16=h1bed415_0
  - libspatialindex=1.8.5=h20b78c2_2
  - libspatialite=4.3.0a=hb08deb6_19
  - libssh2=1.8.2=h1ba5d50_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtheora=1.1.1=h5ab3b9f_1
  - libtiff=4.0.10=h2733197_2
  - libtool=2.4.6=h7b6447c_5
  - libuuid=2.32.1=h14c3975_1000
  - libvorbis=1.3.6=h7b6447c_0
  - libvpx=1.7.0=h439df22_0
  - libxcb=1.13=h1bed415_1
  - libxml2=2.9.9=he19cac6_0
  - libxslt=1.1.33=h7d1a2b0_0
  - linecache2=1.0.0=py27_0
  - llvmlite=0.29.0=py27hd408876_0
  - locket=0.2.0=py27_1
  - lxml=4.3.3=py27hefd8a0e_0
  - lz4-c=1.8.1.2=h14c3975_0
  - lzo=2.10=h49e0be7_2
  - make=4.2.1=h1bed415_1
  - mako=1.0.10=py_0
  - mapclassify=1.0.1=py27_1
  - markupsafe=1.1.1=py27h7b6447c_0
  - matplotlib=1.5.1=np111py27_0
  - mechanicalsoup=0.11.0=py_0
  - memory_profiler=0.55.0=py27_0
  - metpy=0.8.0=py27_0
  - mistune=0.8.4=py27h7b6447c_0
  - mo_pack=0.2.0=py27h3010b51_1003
  - mock=3.0.5=py27_0
  - monotonic=1.5=py_0
  - monty=0.9.6=py27_0
  - more-itertools=5.0.0=py27_0
  - mpc=1.1.0=h10f8cd9_1
  - mpfr=4.0.1=hdf1c602_3
  - mpi=1.0=mpich
  - mpich=3.2.1=hf39692e_8
  - mpmath=1.1.0=py27_0
  - msgpack-python=0.6.1=py27hfd86e86_1
  - multipledispatch=0.6.0=py27_0
  - multiprocessing=2.6.2.1=py27_0
  - munch=2.3.2=py27_0
  - mysql-connector-c=6.1.11=h597af5e_0
  - mysql-python=1.2.5=py27h7b6447c_0
  - natgrid=0.2=py27_2
  - nb_conda=2.2.1=py27_0
  - nb_conda_kernels=2.2.2=py27_0
  - nbconvert=5.5.0=py_0
  - nbformat=4.4.0=py27_0
  - nc-time-axis=1.2.0=py_0
  - ncurses=6.1=he6710b0_1
  - netcdf-fortran=4.4.4=h4363f12_9
  - netcdf4=1.3.1=py27h62672b6_4
  - netcdftime=1.0.0a2=py27_0
  - nettle=3.4.1=h1bed415_1002
  - networkx=2.2=py27_1
  - nomkl=3.0=0
  - nose=1.3.7=py27_2
  - notebook=5.7.8=py27_0
  - numba=0.44.1=py27h962f231_0
  - numcodecs=0.5.4=py27he6710b0_0
  - numexpr=2.6.9=py27h637b7d7_1000
  - numpy=1.11.3=py27_blas_openblash1522bff_1205
  - numpy-base=1.14.3=py27h2b20989_0
  - olefile=0.46=py27_0
  - openblas=0.3.3=h9ac9557_1001
  - openblas-devel=0.2.20=7
  - openh264=1.8.0=hdbcaa40_1000
  - openjpeg=2.3.0=h05c96fa_1
  - openssl=1.1.1c=h7b6447c_1
  - ossuuid=1.6.2=hf484d3e_1000
  - owslib=0.17.1=py_0
  - packaging=19.0=py27_0
  - pandas=0.23.4=py27h04863e7_0
  - pandoc=2.2.3.2=0
  - pandocfilters=1.4.2=py27_1
  - pango=1.39.0=0
  - param=1.9.1=py_0
  - partd=0.3.10=py27_1
  - pathlib=1.0.1=py27_1
  - pathlib2=2.3.3=py27_0
  - patsy=0.5.1=py27_0
  - pbr=5.1.3=py_0
  - pcre=8.43=he6710b0_0
  - pendulum=2.0.4=py27_1000
  - pep8=1.7.1=py27_0
  - pexpect=4.7.0=py27_0
  - pickleshare=0.7.5=py27_0
  - pillow=4.2.1=py27_0
  - pint=0.9=py27_2
  - pip=19.1.1=py27_0
  - pixman=0.32.6=0
  - pkg-config=0.29.2=h1bed415_8
  - pkginfo=1.5.0.1=py27_0
  - plotly=3.10.0=py_0
  - pluggy=0.12.0=py_0
  - pooch=0.5.1=py27_0
  - poppler-data=0.4.9=0
  - postgresql=11.2=h20c2e04_0
  - proj4=5.2.0=he6710b0_1
  - prometheus_client=0.6.0=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - psutil=5.6.2=py27h7b6447c_0
  - psycopg2=2.7.6.1=py27h1ba5d50_0
  - pthread-stubs=0.3=h0ce48e5_1
  - ptyprocess=0.6.0=py27_0
  - py=1.8.0=py27_0
  - pybufr-ecmwf=0.82=py27_1
  - pycairo=1.10.0=py27_0
  - pycodestyle=2.5.0=py27_0
  - pycparser=2.19=py27_0
  - pyct=0.4.6=py27_0
  - pyct-core=0.4.6=py_0
  - pydap=3.2.2=py27_1000
  - pyepsg=0.4.0=py27_0
  - pygeoif=0.7=py_1
  - pygments=2.4.2=py_0
  - pygrib=2.0.2=py27he706d3e_4
  - pyhdf=0.10.1=py27h6053bd1_0
  - pykdtree=1.3.1=py27hdd07704_2
  - pyke=1.1.1=py27_1000
  - pyopenssl=19.0.0=py27_0
  - pyparsing=2.4.0=py_0
  - pyproj=1.9.6=py27h14380d9_0
  - pyqt=4.11.4=py27_3
  - pyresample=1.12.3=py27hf2d7682_0
  - pyrsistent=0.14.11=py27h7b6447c_0
  - pysal=1.14.4.post1=py27_1
  - pyshp=2.1.0=py_0
  - pysocks=1.7.0=py27_0
  - pyspharm=1.0.9=py27h6091dcd_1003
  - pytest=4.6.2=py27_0
  - python=2.7.16=h9bab390_0
  - python-dateutil=2.8.0=py27_0
  - python-ecmwf_grib=1.14.7=np111py27_0
  - python-editor=1.0.4=py_0
  - python-graphviz=0.10.1=py_0
  - python-magic=0.4.15=py27_1001
  - pytz=2019.1=py_0
  - pytzdata=2019.1=py_0
  - pywavelets=1.0.3=py27hdd07704_1
  - pyyaml=5.1=py27h7b6447c_0
  - pyzmq=18.0.0=py27he6710b0_0
  - qt=4.8.7=2
  - qtconsole=4.5.1=py_0
  - readline=7.0=h7b6447c_5
  - readme_renderer=24.0=py27_0
  - regex=2019.06.05=py27h7b6447c_0
  - regionmask=0.4.0=py_0
  - requests=2.22.0=py27_0
  - requests-toolbelt=0.9.1=py_0
  - retrying=1.3.3=py27_2
  - rtree=0.8.3=py27_0
  - s3transfer=0.2.0=py27_0
  - scandir=1.10.0=py27h7b6447c_0
  - scikit-image=0.13.1=py27_0
  - scikit-learn=0.20.3=py27_blas_openblashebff5e3_0
  - scipy=1.2.1=py27_blas_openblash1522bff_0
  - seaborn=0.9.0=py27_0
  - send2trash=1.5.0=py27_0
  - setuptools=41.0.1=py27_0
  - sh=1.12.14=py27_0
  - shapely=1.6.4=py27h86c5351_0
  - simplegeneric=0.8.1=py27_2
  - singledispatch=3.4.0.3=py27_0
  - sip=4.18=py27_1
  - six=1.12.0=py27_0
  - snappy=1.1.7=hbae5bb6_3
  - snuggs=1.4.6=py_0
  - sortedcontainers=2.1.0=py27_0
  - soupsieve=1.8=py27_0
  - sphinx=1.8.5=py27_0
  - sphinxcontrib=1.0=py27_1
  - sphinxcontrib-websupport=1.1.2=py_0
  - sqlalchemy=1.3.4=py27h7b6447c_0
  - sqlite=3.28.0=h7b6447c_0
  - statsmodels=0.9.0=py27h035aef0_0
  - subprocess32=3.5.4=py27h7b6447c_0
  - sympy=1.4=py27_0
  - tabulate=0.8.3=py27_0
  - tbb=2019.4=hfd86e86_0
  - tblib=1.4.0=py_0
  - terminado=0.8.2=py27_0
  - testpath=0.4.2=py27_0
  - tk=8.6.9=hed695b0_1002
  - tktable=2.10=h14c3975_0
  - toolz=0.9.0=py27_0
  - tornado=5.1.1=py27h7b6447c_0
  - tqdm=4.32.1=py_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27_0
  - trmm_rsl=1.49=3
  - twine=1.13.0=py_1
  - typing=3.6.4=py27_0
  - tzcode=2019a=h516909a_1002
  - tzlocal=1.5.1=py27_0
  - udunits2=2.2.27.6=h4e0c4b3_1001
  - unittest2=1.1.0=py27_0
  - urllib3=1.24.2=py27_0
  - util-linux=2.21=0
  - vtk=7.0.0=py27_1
  - wcwidth=0.1.7=py27_0
  - webencodings=0.5.1=py27_1
  - webob=1.8.5=py27_0
  - wheel=0.33.4=py27_0
  - widgetsnbextension=3.4.2=py27_0
  - windspharm=1.7.0=py27_1000
  - wrapt=1.11.1=py27h7b6447c_0
  - x264=1!152.20180806=h14c3975_0
  - xarray=0.10.7=py27_0
  - xerces-c=3.2.2=h780794e_0
  - xlrd=1.2.0=py27_0
  - xorg-fixesproto=5.0=h14c3975_1002
  - xorg-imake=1.0.7=0
  - xorg-inputproto=2.3.2=h14c3975_1002
  - xorg-kbproto=1.0.7=h14c3975_1002
  - xorg-libice=1.0.9=h516909a_1004
  - xorg-libsm=1.2.3=h84519dc_1000
  - xorg-libx11=1.6.7=h14c3975_1000
  - xorg-libxau=1.0.9=h14c3975_0
  - xorg-libxaw=1.0.13=h14c3975_1002
  - xorg-libxdmcp=1.1.3=h516909a_0
  - xorg-libxext=1.3.4=h516909a_0
  - xorg-libxfixes=5.0.3=h516909a_1004
  - xorg-libxi=1.7.9=h516909a_1002
  - xorg-libxmu=1.1.3=h516909a_0
  - xorg-libxpm=3.5.12=h14c3975_1002
  - xorg-libxrender=0.9.10=h516909a_1002
  - xorg-libxt=1.1.5=h14c3975_1002
  - xorg-makedepend=1.0.6=he1b5a44_1
  - xorg-renderproto=0.11.1=h14c3975_1002
  - xorg-xextproto=7.3.0=h14c3975_1002
  - xorg-xproto=7.0.31=h14c3975_1007
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zarr=2.2.0=py_0
  - zeromq=4.3.1=he6710b0_3
  - zict=0.1.4=py27_0
  - zipp=0.5.1=py_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.3.7=h0b5b093_0
  - pip:
    - mule==2017.8.1
    - um-utils==2017.8.1