Python 2 environment NO Longer Supported - GeoscienceAustralia/hiperseis GitHub Wiki
Note: Python-2 has reached its official End-Of-Life. So we recommend to use Python-3 first - See the instructions.
Python-2 setup
NCI
This section describes setup on the Australian National Computational Infrastructure (NCI) facilities. NCI has two primary high end computational resource platforms - Raijin for batch HPC computation, and VDI for interactive development and analysis. Below are documented the setup steps for Python 2 on each of these platforms.
On both VDI and Raijin, a Python 2.7.11 compatible environment has been established. Note that on both systems the system default is Python 2.6, which will not be usable with hiperseis, so it is essential the user module load a Python 2.7 module -- see instructions below.
You can expect a complete Python virtual environment installation with the necessary libraries to take about 500 MB of disk space, so if your home directory has insufficient space, create a symbolic link from inside the venv folder to a folder on a larger capacity filesystem, such as project file storage.
VDI
Initial setup of Python 2 virtual environment:
Start from a freshly opened terminal window on VDI.
- In home directory,
mkdir venvif this folder does not yet exist. module purgemodule load python/2.7.11 virtualenv/15.0.2-py2.7python -m virtualenv venv/py2.7.11- Create a new file in your VDI home directory named
py2_vdi_setup.shand paste the following into it:
module purge
module load git/2.9.5
module load gcc/4.9.0
module load python/2.7.11
module load virtualenv/py2.7
module load cython/0.24-py2.7
module load mpi4py/2.0.0-ompi-1.8.4-py2.7
module load libspatialindex/1.8.5
module load geos/3.5.0
module load proj/4.9.3
module list
source ~/venv/py2.7.11/bin/activate
- Log out of VDI terminal and log in again.
- Each time you log in, run
source ~/py2_vdi_setup.shto activate your Python 2 virtual environment.
For remainder of setup, follow instructions under Installation of libraries to Python 2 virtual environment below.
Raijin
Initial setup of Python 2 virtual environment:
Start from a fresh login on raijin.nci.org.au
- In home directory,
mkdir venvif this folder does not yet exist. module purgemodule load python/2.7.11pip install --user virtualenvpython -m virtualenv venv/py2.7.11- Create a new file in your Raijin home directory named
py2_raijin_setup.shand paste the following into it:
module purge
module load git/2.9.5
module load gcc/4.9.0
module load python/2.7.11
module load mpi4py/2.0.0
module load geos/3.5.0
module load proj/4.9.3
module list
source ~/venv/py2.7.11/bin/activate
- Log out of raijin and log in again.
- Each time you log in, run
source ~/py2_raijin_setup.shto activate your Python 2 virtual environment.
Installation of libraries to Python 2 virtual environment:
- Make sure you have run
source ~/py2_raijin_setup.shimmediately after login. - It is a good idea to verify you are seeing the correct versions of
pythonandpip. Confirm thatwhich pythonandwhich pipreturn paths to yourvenv/py2.7.11/binfolder. - It is suggested at first to update
pipusingpip install --upgrade pip. - For installing subsequent libraries using
pip, it is highly recommended to always use the--no-cache-diroption to preventpipfrom locally caching library downloads, as this can use up a lot of the very limited user home directory disk quota. - This is not necessarily an exhaustive list of packages to install and the user should check the
requirements.txtfiles in modules of thehiperseislibrary.pip installthe following packages (doing them one or two at a time can make it easier to build up your environment incrementally and isolate installation problems more easily):
pip install --no-cache-dir numpy
pip install --no-cache-dir pandas
pip install --no-cache-dir click scipy
pip install --no-cache-dir Cython
pip install --no-cache-dir pyproj==1.9.6
pip install --no-cache-dir dask decorator joblib distributed
pip install --no-cache-dir fastdtw
pip install --no-cache-dir matplotlib
pip install --no-cache-dir obspy
pip install --no-cache-dir toeplitz
pip install --no-cache-dir netcdf4
pip install --no-cache-dir mpi4py
pip install --no-cache-dir h5py
pip install --no-cache-dir obspyh5
pip install --no-cache-dir pathlib2
pip install --no-cache-dir pyasdf
pip install --no-cache-dir pytz tqdm
pip install --no-cache-dir Rtree
pip install --no-cache-dir requests
pip install --no-cache-dir tables
pip install --no-cache-dir ujson
pip install --no-cache-dir xarray
pip install --no-cache-dir shapely
pip install --no-cache-dir geographiclib
pip install --no-cache-dir scikit-learn
pip install --no-cache-dir cartopy
pip install --no-cache-dir rf
Further libraries are required in order to use the full capabilities of hiperseis, in particular basemap cannot be trivially installed with a single pip install command.
basemap library
@medlin01GA has cloned the relevant version of the basemap git repository and built it at path /g/data/ha3/am7399/dev/basemap on NCI. If you have to rebuild basemap, perform the optional python setup.py build below.
To install basemap to your Python 2 environment:
- Make sure you are in your Python 2 virtual environment set up according to the instructions above.
cd /g/data/ha3/am7399/dev/basemapexport GEOS_DIR=$GEOS_ROOTexport PROJ_DIR=$PROJ_ROOT- [OPTIONAL, if needed]
python setup.py build pip install -e .orpython setup.py install- To test basemap install, change to home directory (
cd ~), and then verify that the following import works in your Python 2 virtual environment:from mpl_toolkits.basemap import Basemap
NCI setup summary
If the above steps succeeded, you should be ready to use Python 2 on NCI with the hiperseis library.
Technical Notes
- A system build of
basemapis installed on VDI, however this is not usable because it overrides numpy>=1.16 with a much older numpy==1.11 which does not meet minimum requirements for other Python libraries. pyprojlibrary version must be kept below version 2.x, as the 2.x series ofpyprojbreakbasemapinteraction with the PROJ4 library.
Last known VDI working library set:
Package Version
---------------------------------- -----------
adjustText 0.7.3
asn1crypto 0.24.0
astroid 1.6.5
atomicwrites 1.3.0
attrs 19.1.0
backports-abc 0.5
backports.functools-lru-cache 1.5
backports.shutil-get-terminal-size 1.0.0
basemap 1.2.0
bleach 3.1.0
Cartopy 0.17.0
certifi 2019.3.9
cffi 1.12.3
cftime 1.0.3.4
chardet 3.0.4
Click 7.0
colorama 0.4.1
colorcet 2.0.1
configparser 3.7.3
contextlib2 0.5.5
coverage 4.5.3
cryptography 2.6.1
cycler 0.10.0
Cython 0.24
decorator 4.4.0
defusedxml 0.5.0
Deprecated 1.2.5
descartes 1.1.0
dill 0.2.9
docopt 0.6.2
entrypoints 0.3
enum34 1.1.6
fastdtw 0.3.2
flake8 3.7.7
funcsigs 1.0.2
functools32 3.2.3.post2
future 0.17.1
futures 3.2.0
geographiclib 1.49
graphviz 0.10.1
h5py 2.9.0
idna 2.8
importlib-metadata 0.15
ipaddress 1.0.22
ipdb 0.11
ipykernel 4.10.0
ipython 5.8.0
ipython-genutils 0.2.0
ipywidgets 7.4.2
isodate 0.6.0
isort 4.3.15
Jinja2 2.10
joblib 0.13.2
jsonschema 3.0.1
jupyter 1.0.0
jupyter-client 5.2.4
jupyter-console 5.2.0
jupyter-core 4.4.0
kiwisolver 1.0.1
lazy-object-proxy 1.3.1
lxml 4.3.2
MarkupSafe 1.1.1
matplotlib 2.2.4
mccabe 0.6.1
mistune 0.8.4
mock 2.0.0
more-itertools 5.0.0
mpi4py 2.0.0
nbconvert 5.4.1
nbformat 4.4.0
netCDF4 1.4.3.2
networkx 2.2
notebook 5.7.6
numexpr 2.6.9
numpy 1.16.2
obspy 1.1.1
obspyh5 0.3.2
pandas 0.24.2
pandocfilters 1.4.2
param 1.9.0
pathlib2 2.3.3
pbr 5.1.3
pep517 0.5.0
pexpect 4.6.0
pickleshare 0.7.5
pip 19.0.3
pluggy 0.12.0
prometheus-client 0.6.0
prompt-toolkit 1.0.15
prov 1.5.3
psutil 5.6.1
ptyprocess 0.6.0
py 1.8.0
pyasdf 0.4.0
pycodestyle 2.5.0
pycparser 2.19
pyct 0.4.6
pyflakes 2.1.1
Pygments 2.3.1
pylint 1.9.4
pyOpenSSL 19.0.0
pyparsing 2.3.1
pyproj 1.9.6
pyrsistent 0.14.11
pyshp 2.1.0
pytest 4.5.0
python-dateutil 2.8.0
pytoml 0.1.20
pytz 2018.9
PyYAML 5.1
pyzmq 18.0.1
qtconsole 4.4.3
rdflib 4.2.2
requests 2.21.0
requests-mock 1.6.0
rf 0.7.0
Rtree 0.8.3
scandir 1.10.0
scikit-learn 0.20.3
scipy 1.2.1
seaborn 0.9.0
Send2Trash 1.5.0
setuptools 21.2.2
Shapely 1.6.4.post2
simplegeneric 0.8.1
singledispatch 3.4.0.3
six 1.12.0
SQLAlchemy 1.3.1
subprocess32 3.5.3
tables 3.5.1
terminado 0.8.1
testpath 0.4.2
toeplitz 0.1.4
tornado 5.1.1
tqdm 4.31.1
traitlets 4.3.2
typing 3.6.6
ujson 1.35
urllib3 1.24.1
virtualenv 15.0.2
wcwidth 0.1.7
webencodings 0.5.1
wheel 0.33.1
widgetsnbextension 3.4.2
wrapt 1.11.1
zipp 0.5.1
Last known Raijin working library set:
Package Version Location
----------------------------- ----------- ------------------------------------
aenum 2.1.2
atomicwrites 1.3.0
attrs 19.1.0
backports-abc 0.5
backports.functools-lru-cache 1.5
basemap 1.2.0 /g/data1a/ha3/am7399/dev/basemap/lib
Cartopy 0.17.0
certifi 2019.6.16
cftime 1.0.3.4
chardet 3.0.4
Click 7.0
cloudpickle 1.2.1
colorama 0.4.1
configparser 3.7.4
contextlib2 0.5.5
cycler 0.10.0
Cython 0.29.12
dask 1.2.2
decorator 4.4.0
dill 0.3.0
distributed 1.28.1
entrypoints 0.3
enum34 1.1.6
fastdtw 0.3.2
flake8 3.7.8
funcsigs 1.0.2
functools32 3.2.3.post2
future 0.17.1
futures 3.3.0
geographiclib 1.49
h5py 2.9.0
HeapDict 1.0.0
idna 2.8
importlib-metadata 0.18
isodate 0.6.0
joblib 0.13.2
kiwisolver 1.1.0
lxml 4.3.4
matplotlib 2.2.4
mccabe 0.6.1
mock 3.0.5
more-itertools 5.0.0
mpi4py 2.0.0
msgpack 0.6.1
netCDF4 1.5.1.2
networkx 2.2
numexpr 2.6.9
numpy 1.16.4
obspy 1.1.1
obspyh5 0.4.1
packaging 19.0
pandas 0.24.2
pathlib2 2.3.4
pip 19.1.1
pluggy 0.12.0
prov 1.5.3
psutil 5.6.3
py 1.8.0
pyasdf 0.4.0
pycodestyle 2.5.0
pyflakes 2.1.1
pyparsing 2.4.0
pyproj 1.9.6
pyshp 2.1.0
pytest 4.6.4
python-dateutil 2.8.0
pytz 2019.1
PyYAML 5.1.1
rdflib 4.2.2
requests 2.22.0
rf 0.7.0
Rtree 0.8.3
scandir 1.10.0
scikit-learn 0.20.3
scipy 1.2.2
setuptools 41.0.1
Shapely 1.6.4.post2
singledispatch 3.4.0.3
six 1.12.0
sortedcontainers 2.1.0
SQLAlchemy 1.3.5
subprocess32 3.5.4
tables 3.5.2
tblib 1.4.0
toeplitz 0.3.0
toolz 0.10.0
tornado 5.1.1
tqdm 4.32.2
typing 3.7.4
ujson 1.35
urllib3 1.25.3
wcwidth 0.1.7
wheel 0.33.4
xarray 0.11.3
zict 1.0.0
zipp 0.5.2