Python - nthu-ioa/cluster GitHub Wiki

Quickstart

Load the python module: module load python Make an environment: conda create --name {env name} python={version} {packages} For example: conda create --name {env name} python=3.6 numpy scipy matplotlib Use the environment: source activate {env name}

Conda

Python with the conda package manager can be loaded with module load python. To see which version is the default and any other options, module keyword python.

This will put python and conda on your path. By default, no conda environment is activated when you load the module -- this may be slightly different to how you used conda, for example, on your own machine.

We recommend creating a new conda environment and installing your own packages in that.

conda create --name mypy python=3.6 numpy

If you see the error CondaValueError: The target prefix is the base prefix. Aborting, it means you forgot --name in the command above (a common mistake!).

Notice the packages that will get installed in your new environment (always a good idea to check this):

    package                    |            build
    ---------------------------|-----------------
    blas-1.0                   |              mkl           6 KB
    ca-certificates-2019.5.15  |                0         126 KB
    certifi-2019.6.16          |           py36_0         150 KB
    intel-openmp-2019.4        |              243         729 KB
    libedit-3.1.20181209       |       hc058e9b_0         163 KB
    libffi-3.2.1               |       hd88cf55_4          40 KB
    libgfortran-ng-7.3.0       |       hdf63c60_0        1006 KB
    mkl-2019.4                 |              243       131.2 MB
    mkl_fft-1.0.12             |   py36ha843d7b_0         155 KB
    mkl_random-1.0.2           |   py36hd81dba3_0         362 KB
    ncurses-6.1                |       he6710b0_1         777 KB
    numpy-1.16.4               |   py36h7e9f1db_0          48 KB
    numpy-base-1.16.4          |   py36hde5b4d6_0         3.5 MB
    openssl-1.1.1c             |       h7b6447c_1         2.5 MB
    pip-19.1.1                 |           py36_0         1.6 MB
    python-3.6.8               |       h0371630_0        30.1 MB
    readline-7.0               |       h7b6447c_5         324 KB
    setuptools-41.0.1          |           py36_0         505 KB
    tk-8.6.8                   |       hbc83047_0         2.8 MB
    wheel-0.33.4               |           py36_0          41 KB
    xz-5.2.4                   |       h14c3975_4         283 KB
    zlib-1.2.11                |       h7b6447c_3         103 KB

Two things to notice:

numpy uses mkl -- conda provides its own copy of mkl, so you don't need to worry about using the system version (no need for module load mkl).
In this example I asked for Python 3.6, even though the version in the base environment of our default python module is 3.7. You could also create an environment for python 2.x, or 3.8. You can fix the version of packages in this way too (for example, you can force astropy=4.2.1 if you need that older version).

Once the packages are installed, you can activate the environment with source activate mypy and start work.

You can create as many environments with different configurations as you like. Full instructions are given in the conda documentation.

Q&A

Q. Conda keeps telling me to use `conda activate` rather than `source activate`. Why?

A. Currently conda activate doesn't work in non-interactive sessions like shell scripts. See also here. We recommend sticking with source activate unless you have a good reason to change.

Q. I want to conda-install packages that rely on MPI or CUDA (e.g. tensorflow, pytorch). Will they be optimised for the hardware on the cluster?

A. Maybe, but you need to be aware of some issues. See the conda-forge page here for MPI and here for CUDA.