Python Installation - shawfdong/hyades GitHub Wiki
We've installed a variety of Python releases on Hyades.
Install prerequisites:
# yum install sqlite-devel gdbm-devel
Download the source tar ball from https://www.python.org/downloads/
Unpack the tar ball:
$ tar xvf Python-2.7.8.tar.xz
Build and install Python 2.7.8
$ cd /scratch/Python-2.7.8 $ ./configure --prefix=/pfs/sw/python/Python-2.7.8 $ make $ make install
Install pip
Download get-pip.py from https://pip.pypa.io/en/latest/installing.html, then run:
$ /pfs/sw/python/Python-2.7.8/bin/python get-pip.py
Install NumPy & other Python packages
$ module load python/2.7.8 $ pip install numpy
However, we have to manually build and install matplotlib and scipy.
Build and install matplotlib
Download matplotlib from http://matplotlib.org/downloads.html, and unpack the source tar ball:
$ tar xvfz matplotlib-1.4.0.tar.gz
matplotlib 1.4.0 officially requires freetype2 2.4 or later, but it is known to work with freetype 2.3, which is the version provided by RHEL/CentOS 6. Edit matplotlib-1.4.0/setupext.py, and replace line 945:
min_version='2.4', version=version)with:
min_version='2.3', version=version)
There is also a known bug in matplotlib 1.4.0. Edit matplotlib-1.4.0/lib/matplotlib/tight_bbox.py, and replace line 86:
r = adjust_bbox(figure, bbox_inches, fixed_dpi)with:
r = adjust_bbox(fig, bbox_inches, fixed_dpi)
Now we are ready to build and install matplotlib 1.4.0:
$ cd /scratch/matplotlib-1.4.0 $ python setup.py build $ python setup.py install
Build and install SciPy
Download SciPy from http://sourceforge.net/projects/scipy/files/, and unpack the source tar ball:
$ tar xvfz scipy-0.14.0.tar.gz
Here we build SciPy with ATLAS and LAPACK[1]:
$ export BLAS=/pfs/sw/serial/gcc/atlas-3.10.1/lib/libf77blas.a $ export LAPACK=/pfs/sw/serial/gcc/atlas-3.10.1/lib/liblapack.a $ export ATLAS=/pfs/sw/serial/gcc/atlas-3.10.1/lib/libatlas.a $ cd /scratch/scipy-0.14.0 $ python setup.py build $ python setup.py install
Download the source tar ball from https://www.python.org/downloads/
Unpack the tar ball:
$ tar xvf Python-3.4.1.tar.xz
Build and install Python 3.4.1
$ cd /scratch/Python-3.4.1 $ ./configure --prefix=/pfs/sw/python/Python-3.4.1 $ make $ make install
Install pip
Download get-pip.py from https://pip.pypa.io/en/latest/installing.html, then run:
$ /pfs/sw/python/Python-3.4.1/bin/python get-pip.py
Install NumPy & other Python packages
$ module load python/3.4.1 $ pip3 install numpy
Build and install matplotlib
Modify the source of matplotlib 1.4.0 as described in section Python 2.7.8, then build and install it:
$ cd /scratch/matplotlib-1.4.0 $ python3 setup.py build $ python3 setup.py install
Build and install SciPy
$ export BLAS=/pfs/sw/serial/gcc/atlas-3.10.1/lib/libf77blas.a $ export LAPACK=/pfs/sw/serial/gcc/atlas-3.10.1/lib/liblapack.a $ export ATLAS=/pfs/sw/serial/gcc/atlas-3.10.1/lib/libatlas.a $ cd /scratch/scipy-0.14.0 $ python3 setup.py build $ python3 setup.py install
NOTE there appears to be a bug in the python-dateutil package:
$ pip3 install python-dateutil $ ls -l /pfs/sw/python/Python-3.4.1/lib/python3.4/site-packages/python_dateutil-2.2-py3.4.egg/EGG-INFO -rw------- 1 dong dong 1 Sep 23 14:57 dependency_links.txt -rw------- 1 dong dong 1 Sep 23 14:57 not-zip-safe -rw------- 1 dong dong 970 Sep 23 14:57 PKG-INFO -rw------- 1 dong dong 3 Sep 23 14:57 requires.txt -rw------- 1 dong dong 563 Sep 23 14:57 SOURCES.txt -rw------- 1 dong dong 9 Sep 23 14:57 top_level.txt
Let's fix it™!
$ chmod 664 /pfs/sw/python/Python-3.4.1/lib/python3.4/site-packages/python_dateutil-2.2-py3.4.egg/EGG-INFO/*
Download the source tar ball of PyPy 2.40 from https://bitbucket.org/pypy/pypy/downloads
Unpack the tar ball:
$ tar xvfj pypy-2.4.0-src.tar.bz2
Install build-time dependencies[2]:
# yum install libffi-devel $ module load python/2.7.8 $ pip install Sphinx $ pip install greenlet
Translate the PyPy Python interpreter:
$ cd /scratch/pypy-2.4.0-src $ make
Install PyPy
$ mkdir /pfs/sw/python/pypy-2.4.0 $ cp -r include/ lib* pypy-c site-packages /pfs/sw/python/pypy-2.4.0/ $ cd /pfs/sw/python/pypy-2.4.0 $ mkdir bin $ cd bin $ ln -s ../pypy-c pypy
Install pip
$ module load python/pypy-2.4.0 $ pypy get-pip.py
Install NumPy[3]
$ pip install git+https://bitbucket.org/pypy/numpy.git
Download the source tar ball of PyPy3 2.3.1 from http://pypy.org/download.html; and unpack the tar ball:
$ tar xvfj pypy3-2.3.1-src.tar.bz2
Translate the PyPy3 Python interpreter
NOTE it might not be clear in the documentation; but we must use Python 2 or PyPy 2 to translate PyPy3 (because rpython in the source is a Python 2 program)! This admittedly is a bit confusing and counter-intuitive. Let's use PyPy 2 to do the translation, which will be significantly faster than Python 2.
$ module load python/pypy-2.4.0 $ cd /scratch/pypy3-2.3.1-src $ make
Install PyPy3
$ mkdir /pfs/sw/python/pypy3-2.3.1 $ cp -r include/ lib* pypy-c site-packages /pfs/sw/python/pypy3-2.3.1/ $ cd /pfs/sw/python/pypy3-2.3.1 $ mkdir bin $ cd bin $ ln -s ../pypy-c pypy
Install pip
$ /pfs/sw/python/pypy3-2.3.1/bin/pypy get-pip.py
Install NumPy
$ /pfs/sw/python/pypy3-2.3.1/bin/pip install git+https://bitbucket.org/pypy/numpy.git
We'll install yt as complete Python distributions – the alternative is to install yt as a Python package in an existing Python distribution[4].
Download installation scripts for yt
- yt Legacy (2.x)
$ mkdir -p /pfs/sw/python/yt-2.x $ cd /pfs/sw/python/yt-2.x $ wget http://hg.yt-project.org/yt/raw/yt-2.x/doc/install_script.sh
- yt Stable
$ mkdir -p /pfs/sw/python/yt $ cd /pfs/sw/python/yt $ wget http://hg.yt-project.org/yt/raw/stable/doc/install_script.sh
- yt Development
$ mkdir -p /pfs/sw/python/yt-devel $ cd /pfs/sw/python/yt-devel $ wget http://hg.yt-project.org/yt/raw/yt/doc/install_script.sh
Enable the installation of SciPy. Edit each install_script.sh, and replace the line:
INST_SCIPY=0 # Install scipy?with
INST_SCIPY=1 # Install scipy?
Install all 3 versions of yt
$ cd /pfs/sw/python/yt-2.x $ bash install_script.sh $ cd /pfs/sw/python/yt $ bash install_script.sh $ cd /pfs/sw/python/yt-devel $ bash install_script.sh
Fix the permission bug of python-dateutil:
$ chmod 664 /pfs/sw/python/yt*/yt-x86_64/lib/python2.7/site-packages/python_dateutil-2.2-py2.7.egg/EGG-INFO/*
Note SciPy in yt is built with the generic (unoptimized) BLAS and LAPACK libraries:
export BLAS=$PWD/BLAS/libfblas.a export LAPACK=$PWD/$LAPACK/liblapack.aIt will be interesting to compare the performance of various SciPy builds.
The Python roll of Rocks 6.1 provides Python 2.7.3 and 3.2.1 (in /opt/python/). We'll add a few popular Python packages to each version.
Install pip[5]
$ mkdir -p /pfs/sw/python/Python-2.7.3 $ export PYTHONUSERBASE=/pfs/sw/python/Python-2.7.3 $ /opt/python/bin/python get-pip.py --user
Now new Python packages can be installed with either
$ module load python/2.7.3 $ pip install --install-option="--prefix=/pfs/sw/python/Python-2.7.3" noseor
$ module load python/2.7.3 $ export PYTHONUSERBASE=/pfs/sw/python/Python-2.7.3 $ pip install --user python-dateutil
Build and install NumPy & SciPy with Intel MKL (Math Kernel Library)[6]
Change directory to numpy.1.9.0, and create site.cfg as follows:
[mkl] library_dirs = /opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64 include_dirs = /opt/intel/composer_xe_2013_sp1.1.106/mkl/include mkl_libs = mkl_rt lapack_libs =
Replace line 14 of numpy/distutils/intelccompiler.py:
self.cc_exe = 'icc -fPIC'with:
self.cc_exe = 'icc -O3 -g -fPIC -fp-model strict -fomit-frame-pointer -openmp -xhost -DMKL_ILP64'
And replace lines 52–54 of numpy/distutils/fcompiler/intel.py
def get_flags_opt(self): #return ['-i8 -xhost -openmp -fp-model strict'] return ['-xhost -openmp -fp-model strict']with:
def get_flags_opt(self): return ['-i8 -xhost -openmp -fp-model strict'] # return ['-xhost -openmp -fp-model strict']
NOTE here we use ILP64[7] interface of Intel MKL.
Compile and install NumPy with the Intel compiler:
$ cd /scratch/numpy-1.9.0 $ python setup.py config --compiler=intelem build_clib \ --compiler=intelem build_ext \ --compiler=intelem install --prefix=/pfs/sw/python/Python-2.7.3
Compile and install SciPy with the Intel compiler:
$ cd /scratch/scipy-0.14.0 $ python setup.py config --compiler=intelem --fcompiler=intelem build_clib \ --compiler=intelem --fcompiler=intelem build_ext \ --compiler=intelem --fcompiler=intelem install --prefix=/pfs/sw/python/Python-2.7.3
Install pip
$ mkdir -p /pfs/sw/python/Python-3.2.1 $ export PYTHONUSERBASE=/pfs/sw/python/Python-3.2.1 $ /opt/python/bin/python3 get-pip.py --user
Now new Python packages can be installed with either
$ module load python/3.2.1 $ pip3 install --install-option="--prefix=/pfs/sw/python/Python-3.2.1" noseor
$ module load python/3.2.1 $ export PYTHONUSERBASE=/pfs/sw/python/Python-3.2.1 $ pip3 install --user python-dateutil
Build and install NumPy & SciPy with Intel MKL
First modify the source of NumPy as described in section Python 2.7.3. Then compile and install NumPy with the Intel compiler:
$ cd /scratch/numpy-1.9.0 $ python3 setup.py config --compiler=intelem build_clib \ --compiler=intelem build_ext \ --compiler=intelem install --prefix=/pfs/sw/python/Python-3.2.1and compile and install SciPy with the Intel compiler:
$ cd /scratch/scipy-0.14.0 $ python3 setup.py config --compiler=intelem --fcompiler=intelem build_clib \ --compiler=intelem --fcompiler=intelem build_ext \ --compiler=intelem --fcompiler=intelem install --prefix=/pfs/sw/python/Python-3.2.1
For completeness, we also add a few popular Python packages to Python 2.6.6, which is the stock Python offered by RHEL/CentOS 6.
Install pip
$ mkdir -p /pfs/sw/python/Python-2.6.6 $ export PYTHONUSERBASE=/pfs/sw/python/Python-2.6.6 $ /usr/bin/python get-pip.py --user
Build and install NumPy & SciPy with Intel MKL
First modify the source of NumPy as described in section Python 2.7.3. Then compile and install NumPy with the Intel compiler:
$ cd /scratch/numpy-1.9.0 $ /usr/bin/python setup.py config --compiler=intelem build_clib \ --compiler=intelem build_ext \ --compiler=intelem install --prefix=/pfs/sw/python/Python-3.2.1and compile and install SciPy with the Intel compiler:
$ cd /scratch/scipy-0.14.0 $ /usr/bin/python setup.py config --compiler=intelem --fcompiler=intelem build_clib \ --compiler=intelem --fcompiler=intelem build_ext \ --compiler=intelem --fcompiler=intelem install --prefix=/pfs/sw/python/Python-3.2.1