Python Installation - shawfdong/hyades GitHub Wiki

We've installed a variety of Python releases on Hyades.

Table of Contents

Python 2.7.8

Install prerequisites: 

# yum install sqlite-devel gdbm-devel

Download the source tar ball from https://www.python.org/downloads/

Unpack the tar ball: 

$ tar xvf Python-2.7.8.tar.xz

Build and install Python 2.7.8 

$ cd /scratch/Python-2.7.8
$ ./configure --prefix=/pfs/sw/python/Python-2.7.8
$ make
$ make install

Install pip

Download get-pip.py from https://pip.pypa.io/en/latest/installing.html, then run: 

$ /pfs/sw/python/Python-2.7.8/bin/python get-pip.py

Install NumPy & other Python packages 

$ module load python/2.7.8
$ pip install numpy

However, we have to manually build and install matplotlib and scipy.

Build and install matplotlib

Download matplotlib from http://matplotlib.org/downloads.html, and unpack the source tar ball: 

$ tar xvfz matplotlib-1.4.0.tar.gz

matplotlib 1.4.0 officially requires freetype2 2.4 or later, but it is known to work with freetype 2.3, which is the version provided by RHEL/CentOS 6. Edit matplotlib-1.4.0/setupext.py, and replace line 945: 

            min_version='2.4', version=version)
with: 
            min_version='2.3', version=version)

There is also a known bug in matplotlib 1.4.0. Edit matplotlib-1.4.0/lib/matplotlib/tight_bbox.py, and replace line 86: 

    r = adjust_bbox(figure, bbox_inches, fixed_dpi)
with: 
    r = adjust_bbox(fig, bbox_inches, fixed_dpi)

Now we are ready to build and install matplotlib 1.4.0: 

$ cd /scratch/matplotlib-1.4.0
$ python setup.py build
$ python setup.py install

Build and install SciPy

Download SciPy from http://sourceforge.net/projects/scipy/files/, and unpack the source tar ball: 

$ tar xvfz scipy-0.14.0.tar.gz

Here we build SciPy with ATLAS and LAPACK[1]: 

$ export BLAS=/pfs/sw/serial/gcc/atlas-3.10.1/lib/libf77blas.a
$ export LAPACK=/pfs/sw/serial/gcc/atlas-3.10.1/lib/liblapack.a
$ export ATLAS=/pfs/sw/serial/gcc/atlas-3.10.1/lib/libatlas.a
$ cd /scratch/scipy-0.14.0
$ python setup.py build
$ python setup.py install

Python 3.4.1

Download the source tar ball from https://www.python.org/downloads/

Unpack the tar ball: 

$ tar xvf Python-3.4.1.tar.xz

Build and install Python 3.4.1 

$ cd /scratch/Python-3.4.1
$ ./configure --prefix=/pfs/sw/python/Python-3.4.1
$ make
$ make install

Install pip

Download get-pip.py from https://pip.pypa.io/en/latest/installing.html, then run: 

$ /pfs/sw/python/Python-3.4.1/bin/python get-pip.py

Install NumPy & other Python packages 

$ module load python/3.4.1
$ pip3 install numpy

Build and install matplotlib

Modify the source of matplotlib 1.4.0 as described in section Python 2.7.8, then build and install it: 

$ cd /scratch/matplotlib-1.4.0
$ python3 setup.py build
$ python3 setup.py install

Build and install SciPy 

$ export BLAS=/pfs/sw/serial/gcc/atlas-3.10.1/lib/libf77blas.a
$ export LAPACK=/pfs/sw/serial/gcc/atlas-3.10.1/lib/liblapack.a
$ export ATLAS=/pfs/sw/serial/gcc/atlas-3.10.1/lib/libatlas.a
$ cd /scratch/scipy-0.14.0
$ python3 setup.py build
$ python3 setup.py install

NOTE there appears to be a bug in the python-dateutil package: 

$ pip3 install python-dateutil
$ ls -l /pfs/sw/python/Python-3.4.1/lib/python3.4/site-packages/python_dateutil-2.2-py3.4.egg/EGG-INFO
-rw------- 1 dong dong   1 Sep 23 14:57 dependency_links.txt
-rw------- 1 dong dong   1 Sep 23 14:57 not-zip-safe
-rw------- 1 dong dong 970 Sep 23 14:57 PKG-INFO
-rw------- 1 dong dong   3 Sep 23 14:57 requires.txt
-rw------- 1 dong dong 563 Sep 23 14:57 SOURCES.txt
-rw------- 1 dong dong   9 Sep 23 14:57 top_level.txt

Let's fix it™! 

$ chmod 664 /pfs/sw/python/Python-3.4.1/lib/python3.4/site-packages/python_dateutil-2.2-py3.4.egg/EGG-INFO/*

PyPy 2.4.0

Download the source tar ball of PyPy 2.40 from https://bitbucket.org/pypy/pypy/downloads

Unpack the tar ball: 

$ tar xvfj pypy-2.4.0-src.tar.bz2

Install build-time dependencies[2]: 

# yum install libffi-devel
$ module load python/2.7.8
$ pip install Sphinx
$ pip install greenlet

Translate the PyPy Python interpreter: 

$ cd /scratch/pypy-2.4.0-src
$ make

Install PyPy 

$ mkdir /pfs/sw/python/pypy-2.4.0
$ cp -r include/ lib* pypy-c site-packages /pfs/sw/python/pypy-2.4.0/
$ cd /pfs/sw/python/pypy-2.4.0
$ mkdir bin
$ cd bin
$ ln -s ../pypy-c pypy

Install pip 

$ module load python/pypy-2.4.0
$ pypy get-pip.py

Install NumPy[3] 

$ pip install git+https://bitbucket.org/pypy/numpy.git

PyPy3 2.3.1

Download the source tar ball of PyPy3 2.3.1 from http://pypy.org/download.html; and unpack the tar ball: 

$ tar xvfj pypy3-2.3.1-src.tar.bz2

Translate the PyPy3 Python interpreter

NOTE it might not be clear in the documentation; but we must use Python 2 or PyPy 2 to translate PyPy3 (because rpython in the source is a Python 2 program)! This admittedly is a bit confusing and counter-intuitive. Let's use PyPy 2 to do the translation, which will be significantly faster than Python 2. 

$ module load python/pypy-2.4.0
$ cd /scratch/pypy3-2.3.1-src
$ make

Install PyPy3 

$ mkdir /pfs/sw/python/pypy3-2.3.1
$ cp -r include/ lib* pypy-c site-packages /pfs/sw/python/pypy3-2.3.1/
$ cd /pfs/sw/python/pypy3-2.3.1
$ mkdir bin
$ cd bin
$ ln -s ../pypy-c pypy

Install pip 

$ /pfs/sw/python/pypy3-2.3.1/bin/pypy get-pip.py

Install NumPy 

$ /pfs/sw/python/pypy3-2.3.1/bin/pip install git+https://bitbucket.org/pypy/numpy.git

yt

We'll install yt as complete Python distributions – the alternative is to install yt as a Python package in an existing Python distribution[4].

Download installation scripts for yt

  • yt Legacy (2.x)
$ mkdir -p /pfs/sw/python/yt-2.x
$ cd /pfs/sw/python/yt-2.x
$ wget http://hg.yt-project.org/yt/raw/yt-2.x/doc/install_script.sh
  • yt Stable
$ mkdir -p /pfs/sw/python/yt
$ cd /pfs/sw/python/yt
$ wget http://hg.yt-project.org/yt/raw/stable/doc/install_script.sh
  • yt Development
$ mkdir -p /pfs/sw/python/yt-devel
$ cd /pfs/sw/python/yt-devel
$ wget http://hg.yt-project.org/yt/raw/yt/doc/install_script.sh

Enable the installation of SciPy. Edit each install_script.sh, and replace the line: 

INST_SCIPY=0    # Install scipy?
with 
INST_SCIPY=1    # Install scipy?

Install all 3 versions of yt 

$ cd /pfs/sw/python/yt-2.x
$ bash install_script.sh
$ cd /pfs/sw/python/yt
$ bash install_script.sh
$ cd /pfs/sw/python/yt-devel
$ bash install_script.sh

Fix the permission bug of python-dateutil: 

$ chmod 664  /pfs/sw/python/yt*/yt-x86_64/lib/python2.7/site-packages/python_dateutil-2.2-py2.7.egg/EGG-INFO/*

Note SciPy in yt is built with the generic (unoptimized) BLAS and LAPACK libraries: 

    export BLAS=$PWD/BLAS/libfblas.a
    export LAPACK=$PWD/$LAPACK/liblapack.a
It will be interesting to compare the performance of various SciPy builds.

Python 2.7.3

The Python roll of Rocks 6.1 provides Python 2.7.3 and 3.2.1 (in /opt/python/). We'll add a few popular Python packages to each version.

Install pip[5] 

$ mkdir -p /pfs/sw/python/Python-2.7.3
$ export PYTHONUSERBASE=/pfs/sw/python/Python-2.7.3
$ /opt/python/bin/python get-pip.py --user

Now new Python packages can be installed with either 

$ module load python/2.7.3
$ pip install --install-option="--prefix=/pfs/sw/python/Python-2.7.3" nose
or 
$ module load python/2.7.3
$ export PYTHONUSERBASE=/pfs/sw/python/Python-2.7.3
$ pip install --user python-dateutil

Build and install NumPy & SciPy with Intel MKL (Math Kernel Library)[6]

Change directory to numpy.1.9.0, and create site.cfg as follows: 

[mkl]
library_dirs = /opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64
include_dirs = /opt/intel/composer_xe_2013_sp1.1.106/mkl/include
mkl_libs = mkl_rt
lapack_libs =

Replace line 14 of numpy/distutils/intelccompiler.py: 

         self.cc_exe = 'icc -fPIC'
with: 
         self.cc_exe = 'icc -O3 -g -fPIC -fp-model strict -fomit-frame-pointer -openmp -xhost -DMKL_ILP64'

And replace lines 52–54 of numpy/distutils/fcompiler/intel.py 

    def get_flags_opt(self):
         #return ['-i8 -xhost -openmp -fp-model strict']
         return ['-xhost -openmp -fp-model strict']
with: 
    def get_flags_opt(self):
         return ['-i8 -xhost -openmp -fp-model strict']
         # return ['-xhost -openmp -fp-model strict']

NOTE here we use ILP64[7] interface of Intel MKL.

Compile and install NumPy with the Intel compiler: 

$ cd /scratch/numpy-1.9.0
$ python setup.py config --compiler=intelem build_clib \
    --compiler=intelem build_ext \
    --compiler=intelem install --prefix=/pfs/sw/python/Python-2.7.3

Compile and install SciPy with the Intel compiler: 

$ cd /scratch/scipy-0.14.0
$ python setup.py config --compiler=intelem --fcompiler=intelem build_clib \
    --compiler=intelem --fcompiler=intelem build_ext \
    --compiler=intelem --fcompiler=intelem install --prefix=/pfs/sw/python/Python-2.7.3

Python 3.2.1

Install pip 

$ mkdir -p /pfs/sw/python/Python-3.2.1
$ export PYTHONUSERBASE=/pfs/sw/python/Python-3.2.1
$ /opt/python/bin/python3 get-pip.py --user

Now new Python packages can be installed with either 

$ module load python/3.2.1
$ pip3 install --install-option="--prefix=/pfs/sw/python/Python-3.2.1" nose
or 
$ module load python/3.2.1
$ export PYTHONUSERBASE=/pfs/sw/python/Python-3.2.1
$ pip3 install --user python-dateutil

Build and install NumPy & SciPy with Intel MKL

First modify the source of NumPy as described in section Python 2.7.3. Then compile and install NumPy with the Intel compiler: 

$ cd /scratch/numpy-1.9.0
$ python3 setup.py config --compiler=intelem build_clib \
    --compiler=intelem build_ext \
    --compiler=intelem install --prefix=/pfs/sw/python/Python-3.2.1
and compile and install SciPy with the Intel compiler: 
$ cd /scratch/scipy-0.14.0
$ python3 setup.py config --compiler=intelem --fcompiler=intelem build_clib \
    --compiler=intelem --fcompiler=intelem build_ext \
    --compiler=intelem --fcompiler=intelem install --prefix=/pfs/sw/python/Python-3.2.1

Python 2.6.6

For completeness, we also add a few popular Python packages to Python 2.6.6, which is the stock Python offered by RHEL/CentOS 6.

Install pip 

$ mkdir -p /pfs/sw/python/Python-2.6.6
$ export PYTHONUSERBASE=/pfs/sw/python/Python-2.6.6
$ /usr/bin/python get-pip.py --user

Build and install NumPy & SciPy with Intel MKL

First modify the source of NumPy as described in section Python 2.7.3. Then compile and install NumPy with the Intel compiler: 

$ cd /scratch/numpy-1.9.0
$ /usr/bin/python setup.py config --compiler=intelem build_clib \
    --compiler=intelem build_ext \
    --compiler=intelem install --prefix=/pfs/sw/python/Python-3.2.1
and compile and install SciPy with the Intel compiler: 
$ cd /scratch/scipy-0.14.0
$ /usr/bin/python setup.py config --compiler=intelem --fcompiler=intelem build_clib \
    --compiler=intelem --fcompiler=intelem build_ext \
    --compiler=intelem --fcompiler=intelem install --prefix=/pfs/sw/python/Python-3.2.1

References

  1. ^ Building SciPy From Source on Linux
  2. ^ Getting Started with PyPy's Python Interpreter
  3. ^ PyPy - Installing NumPy
  4. ^ Get yt
  5. ^ pip - User Installs
  6. ^ Numpy/Scipy with Intel MKL
  7. ^ 64-bit computing
⚠️ **GitHub.com Fallback** ⚠️