Installation FAQ (pip, setup.py) - GalSim-developers/GalSim GitHub Wiki
This page covers questions about installing GalSim with pip or setup.py. For questions about installing with SCons, go here.
To start, see the installation instructions here. If your problem is not yet listed on this FAQ, feel free to post an issue.
-
Dependencies
-
Installing with pip
-
Installing with setup.py
-
Installing with conda
-
Importing galsim after installation
When running pip install galsim, you may get the message:
****
Installation requires setuptools version >= 38.
Please upgrade or install with pip install -U setuptools
****
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/tmp/pip-build-a04km4cq/galsim/setup.py", line 30, in <module>
from setuptools.command.build_clib import build_clib
ImportError: No module named 'setuptools.command.build_clib'
The suggestion to run pip install -U setuptools is normally effective to get this working.
Sometimes you may run into an SSL certificate verify error. E.g. the following output:
Collecting galsim
Downloading https://files.pythonhosted.org/packages/f5/c9/eae469e73476b208701a7cb21a22daec61ec16cfb121e1abb328a9344e9f/GalSim-2.0.0rc1.tar.gz (36.4MB)
100% |████████████████████████████████| 36.4MB 33kB/s
Complete output from command python setup.py egg_info:
Download error on https://pypi.org/simple/eigency/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600) -- Some packages may not be found!
Couldn't find index page for 'eigency' (maybe misspelled?)
Download error on https://pypi.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600) -- Some packages may not be found!
No local packages or working download links found for eigency>=1.77
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/tmp/pip-build-7cpz0c4h/galsim/setup.py", line 864, in <module>
zip_safe=False,
File "/sw/lib/python3.4/site-packages/setuptools/__init__.py", line 128, in setup
_install_setup_requires(attrs)
File "/sw/lib/python3.4/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
dist.fetch_build_eggs(dist.setup_requires)
File "/sw/lib/python3.4/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
replace_conflicting=True,
File "/sw/lib/python3.4/site-packages/pkg_resources/__init__.py", line 770, in resolve
replace_conflicting=replace_conflicting
File "/sw/lib/python3.4/site-packages/pkg_resources/__init__.py", line 1053, in best_match
return self.obtain(req, installer)
File "/sw/lib/python3.4/site-packages/pkg_resources/__init__.py", line 1065, in obtain
return installer(requirement)
File "/sw/lib/python3.4/site-packages/setuptools/dist.py", line 581, in fetch_build_egg
return cmd.easy_install(req)
File "/sw/lib/python3.4/site-packages/setuptools/command/easy_install.py", line 667, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('eigency>=1.77')
Using setuptools version 39.2.0
Python version = 3.4.3 (default, Sep 26 2015, 00:09:24)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
Adding eigency to build_dep
GalSim version is 2.0.0rc1
The result is that pip cannot automatically install the dependencies for you. So the solution is to just install each one by hand. In the above case, just pip install -U eigency. Then, in my case, that also failed to find its dependency, setuptools_scm. So I had to do pip install -U setuptools_scm. Etc.
There may be a different solution to fix the underlying problem with the SSL certificate, but manual installation of the dependencies worked, so I didn't investigate further.
It seems that pip sometimes cannot see the locations where the dependencies are hosted. With pip install galsim, this may result in the error:
Could not find a version that satisfies the requirement eigency>=1.77 (from versions: )
No matching distribution found for eigency>=1.77
Getting more information with pip install -v galsim shows many lines like
Analyzing links from page https://pypi.org/simple/setuptools/
Skipping link https://files.pythonhosted.org/packages/77/38/90cae8fd99c5dc402d25930e1c91f5fcf6de5c8e982fd152a5a061e3df56/setuptools-0.6b1-py2.3.egg#sha256=ae0a6ec6090a92d08fe7f3dbf9f1b2ce889bce2a3d7724b62322a29b92cf93f0 (from https://pypi.org/simple/setuptools/); unsupported archive format: .egg
Skipping link https://files.pythonhosted.org/packages/1f/ab/1ed6131b5112d1d47cb89e1cbea08f783f0ed49d6fc795ed2ff63f84d20c/setuptools-0.6b1-py2.4.egg#sha256=377ff1ce6d8e24842fbc5f7fdf1c64875f7fe9a1c4db9e7b7479cbce1377da86 (from https://pypi.org/simple/setuptools/); unsupported archive format: .egg
[... snip ...]
Skipping link https://files.pythonhosted.org/packages/bc/bf/030ed2136c49a7753a9cc11be7454e16de1977f23ce1cff5fe5427579c2e/eigency-1.76.tar.gz#sha256=b5008d8b502d0ecdafbc6c288a98c24d098e594e3cb06a260fe070accf40a148 (from https://pypi.org/simple/eigency/); No sources permitted for eigency
Skipping link https://files.pythonhosted.org/packages/fb/6e/bc4359fbfb0bb0b588ec328251b0d0836bdd7c0a4c568959ea06df023e18/eigency-1.77.tar.gz#sha256=13cf96e00020fd900aa5f8905036488afd27c89212bfc07b60ac5412073ea7d7 (from https://pypi.org/simple/eigency/); No sources permitted for eigency
Could not find a version that satisfies the requirement eigency>=1.77 (from versions: )
Cleaning up...
Removing source in /private/var/folders/h8/0h3_h20d4jdg8pk8fzwxmr380000gn/T/pip-req-build-3y5anoty
No matching distribution found for eigency>=1.77
The solution that seems to work is to upgrade to pip version 18+:
pip install -U pip
and try again.
Bryan Gillis reported the following error when trying to build GalSim: (The error actually happened when compiling the dependency, LSSTDESC/Coord, but the error is unrelated to the module being compiled.)
gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Iinclude -I/cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.0/usr/include/python3.6m -c src/Angle.cpp -o build/temp.linux-x86_64-3.6/src/Angle.o
g++ -pthread -shared -Wl,-z,relro build/temp.linux-x86_64-3.6/src/Angle.o -L/usr/lib64 -lpython3.6m -o build/lib.linux-x86_64-3.6/coord/_coord.cpython-36m-x86_64-linux-gnu.so
/usr/bin/ld: cannot find -lpython3.6m
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1
The problem was that the library libpython3.6m.so was not in his LIBRARY_PATH. The directory with this library needs to be in both LIBRARY_PATH and LD_LIBRARY_PATH. The former is where the linker looks at build time, and the latter is where the system looks at run time. So you need to have both of these system variables set properly.
The solution is to make sure both of the following are set properly, usually in your .bash_profile. In his case, this looked like:
export LIBRARY_PATH=$LIBARY_PATH:/cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.0/usr/lib64/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.0/usr/lib64/
Obviously, you should replace the directory name abouve with wherever your system has the libpython*.so library file.
If you build with sudo and then later build without sudo, this can happen. The easiest resolution is probably to sudo rm -r ~/.ccache to blow away the current cache (with sudo) and start over.
Erin Sheldon reported that after installing GalSim with python setup.py install, the installed file is an .egg directory rather than a flat galsim directory. This is normally not a problem, but if you previously installed GalSim with pip, it installs a wheel, which does include the flat galsim directory and a .dist-info file.
The problem is that import galsim will favor the wheel installation over the newer egg installation. This kind of thing is why many people gripe about egg installations and why the newer wheel installation is gaining favor. However, I haven't been able to figure out how to make python setup.py install install either a wheel or the older distutils-style egg installation with a flat directory and an .egg-info file.
The two workarounds for this issue are:
a. Delete all older galsim installations before using python setup.py install. The easiest way to do so is to run pip uninstall galsim repeatedly until it says that galsim is not installed.
b. Use pip install . to install galsim rather than python setup.py install. This will install a wheel, which will properly supersede previous versions. The downside is that this takes a bit longer to run than running setup.py install, since it makes sure to resolve all dependencies each time, which during development is just wasted cycles.
Installing with setup.py python install on Mac OS Mojave (10.14) may lead to the error:
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
... fatal error: 'cmath' file not found
This appears to be a problem with the default MACOSX_DEPLOYMENT_TARGET environment variable on Mac OS Mojave (10.14). The solution is to execute
export MACOSX_DEPLOYMENT_TARGET=10.9
before executing setup.py.
(You may wish to add this to your .bashrc or equivalent shell startup script).
David Kirkby reported the following problem using pip install galsim with Anaconda Python (See also #1035):
Collecting galsim
Downloading https://files.pythonhosted.org/packages/96/a1/4da265850d10f13b0aa99fa20db13a5ee81865fb8e07e7544fd69a3d0c5a/GalSim-2.0.2.tar.gz (36.8MB)
100% |████████████████████████████████| 36.8MB 737kB/s
Missing build time requirements in pyproject.toml for galsim from https://files.pythonhosted.org/packages/96/a1/4da265850d10f13b0aa99fa20db13a5ee81865fb8e07e7544fd69a3d0c5a/GalSim-2.0.2.tar.gz#sha256=659ba3d3b0c4fe2b77361822ce89ee1ed6a454fe1a8ed14f7a3e752b52304996: 'wheel'.
This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
Installing build dependencies ... done
Complete output from command python setup.py egg_info:
warning: no files found matching 'eigency/*.pyd'
warning: no previously-included files found matching 'eigency/*.cpp'
warning: no previously-included files matching 'CMakeLists.txt' found under directory 'eigency/eigen_3.2.8/Eigen'
...snip...
distutils.errors.DistutilsError: Setup script exited with error: SandboxViolation: mkdir('/private/var/var', 511) {}
The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.
This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand. Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.
Using setuptools version 40.0.0
Python version = 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:14:59)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
Adding eigency to build_dep
GalSim version is 2.0.2
This seems to be an error either in eigency or setuptools wherein eigency tries to write some temporary files in the normal place (e.g. either /tmp or /private/var), and setuptools doesn't allow it.
One solution is to install eigen separately first. E.g. if you have conda, then you can do
conda install eigen
to use its Eigen distribution rather than relying on eigency. Then run pip install galsim again.
Another solution is to use pip to install the dependencies separately before trying to install GalSim. For instance, if you have the GalSim repository cloned, then you can do
pip install -r requirements.txt
or just install eigency, since that's the problematic one:
pip install rmjarvis.eigency>=1.77.1
David Kirkby reported the following problem using pip install galsim with Anaconda Python:
Installing build dependencies ... done
Complete output from command python setup.py egg_info:
Using setuptools version 40.0.0
Python version = 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:14:59)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
GalSim version is 2.0.2
running egg_info
creating pip-egg-info/GalSim.egg-info
writing pip-egg-info/GalSim.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/GalSim.egg-info/dependency_links.txt
writing entry points to pip-egg-info/GalSim.egg-info/entry_points.txt
writing requirements to pip-egg-info/GalSim.egg-info/requires.txt
writing top-level names to pip-egg-info/GalSim.egg-info/top_level.txt
writing manifest file 'pip-egg-info/GalSim.egg-info/SOURCES.txt'
error: Could not find PyBind11
Include directory for fftw3 is /usr/local/include
PyBind11 is version 2.2.3
Looking for pybind11 header files:
/Users/david/.local/include/python3.6m (no)
/Users/david/anaconda3/envs/desi/include/python3.6m (no)
Could not find pybind11 header files.
They should have been in one of the following two locations:
/Users/david/.local/include/python3.6m
/Users/david/anaconda3/envs/desi/include/python3.6m
I'm not sure why pybind11 doesn't report its own location properly sometimes when installed with pip in a conda distribution. However, using the conda-forge installation instead usually seems to work:
conda install -c conda-forge pybind11
Then repeat pip install galsim
David Kirkby reported the following problem importing galsim after installing with pip install galsim with Anaconda Python:
ImportError: dlopen(/Users/david/anaconda3/envs/desi/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so, 2): Symbol not found: __ZN3tmv14BandMatrixViewINSt3__17complexIdEELi0EE13conjugateSelfEv
Referenced from: /usr/local/lib/libgalsim.1.3.dylib
Expected in: /usr/local/lib/libtmv_symband.0.dylib
in /usr/local/lib/libgalsim.1.3.dylib
This can happen when you have previously installed GalSim 1.x with the old SCons installation and are now installing GalSim 2.x with pip or setup.py. If the old GalSim libraries are in your LIBRARY_PATH, then they can sometimes be linked rather than the new 2.0 library.
The easiest solution is to manually remove these obsolete libraries and reinstall. E.g.
sudo rm /usr/local/lib/libgalsim*
pip uninstall galsim
pip install --no-cache-dir galsim
Note the --no-cache-dir option to make sure that pip actually rebuilds everything rather than just reusing the previous library with the bad linkage.
a) Are you doing import galsim in the GalSim source directory?
If so, you might see something like the following:
>>> import galsim
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/Mike/GalSim/galsim/__init__.py", line 96, in <module>
from .position import Position, PositionI, PositionD
File "/Users/Mike/GalSim/galsim/position.py", line 22, in <module>
from . import _galsim
ImportError: cannot import name '_galsim'
Python looks in the current directory first when importing something. So in this case, it sees the galsim directory and tries to import that rather than from the installed location. But the build process doesn't build the _galsim.so library there. (It's in a subdirectory of the build directory.) So you get this error.
The solution is to change to a different directory and running Python from there.
b) It may be finding the wrong installation of galsim.
Sometimes you can end up with multiple GalSim versions installed on your machine, and python can get confused about which one is the right one to use. Especially if you have switched among the pip, SCons, and setup.py installation methods.
Normally you can check which version of something in imported by checking e.g. galsim.__file__. But if it
cannot import, this method doesn't work. Instead, you need to do the following:
>>> import imp
>>> imp.find_module('galsim')
(None, '/Users/Mike/anaconda3/lib/python3.6/site-packages/galsim', ('', '', 5))
That second item in the tuple is the location that it is trying to import from when you do import galsim. If that doesn't match the location that you think galsim is installed, then try deleting that installation and trying again.
One easy way to make sure you only have one version of galsim installed is to do pip uninstall galsim repeatedly until it says there is no version of galsim installed. Then reinstall it once.
a) This is another possible error if you have multiple versions of galsim installed and it is finding the wrong one at run time.
E.g. Niall MacCrann reported the following error:
>> import galsim
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/global/project/projectdirs/des/maccrann/py3/lib/python3.6/site-packages/galsim/__init__.py", line 94, in <module>
from .position import PositionI, PositionD
File "/global/project/projectdirs/des/maccrann/py3/lib/python3.6/site-packages/galsim/position.py", line 22, in <module>
from . import _galsim
ImportError: /global/project/projectdirs/des/maccrann/py3/lib/python3.6/site-packages/galsim/_galsim.so: undefined symbol: _ZN6galsim7Silicon10accumulateIfEEdRKNS_11PhotonArrayENS_14UniformDeviateENS_9ImageViewIT_EENS_8PositionIiEE
He found that deleting the files in this directory and doing a clean install fixed it.
Sometimes, especially when switching from a SCons-based installation to a pip or setup.py installation, not all of the previous version's files are correctly removed or updated, so manual deleting may be required. Further updates should work fine if you keep using the same installation method.
b) Another possibility is that the _galsim.so file is correct, but it is linked to the wrong libgalsim.so library.
E.g. the following error:
>>> import galsim
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/__init__.py", line 96, in <module>
from .position import Position, PositionI, PositionD
File "/Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/position.py", line 22, in <module>
from . import _galsim
ImportError: dlopen(/Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so, 2): Symbol not found: __ZN6galsim11invertImageISt7complexIdEEEvNS_9ImageViewIT_EE
Referenced from: /Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so
Expected in: flat namespace
in /Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so
>>>
This probably means that there is some libgalsim.so file in your LIBRARY_PATH, which was found at compile time, probably from an older SCons installation procedure. If setup.py finds that at link time, it can link to that older library, rather than the libgalsim.a library that it just compiled.
The way to check is to use either ldd (on Unix) or otool -L (on Mac) to check what external libraries _galsim.so is linking to:
$ otool -L /Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so
/Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so:
/sw/lib/libgalsim.2.0.dylib (compatibility version 2.0.0, current version 2.0.0)
/sw/lib/libfftw3.3.dylib (compatibility version 7.0.0, current version 7.2.0)
@rpath/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.19.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/Users/Mike/anaconda3/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
The above is an example of an erroneous build. The libgalsim line should not be there, since we now statically link all the C++ code into the _galsim.so library.
The solution is to delete the file in the above location (/Users/Mike/lib/libgalsim.2.0.dylib in the above case), remove the GalSim/build directory, and rerun python setup.py install.
If you used pip to install, it might still have linked to this wrong library. To force pip to reinstall, you can do
pip uninstall galsim [possibly multiple times to make sure all versions are removed]
pip --no-cache-dir install galsim
That should then rebuild and hopefully link properly.
In the above example, after I removed the file /sw/lib/libgalsim.2.0.dylib, the output of otool is:
$ otool -L build/*/galsim/_galsim.cpython-36m-darwin.so
build/lib.macosx-10.7-x86_64-3.6/galsim/_galsim.cpython-36m-darwin.so:
/sw/lib/libfftw3.3.dylib (compatibility version 7.0.0, current version 7.2.0)
@rpath/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.19.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/Users/Mike/anaconda3/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
Obviously, there are many possibilities here, but so far the main reason seems to be similar to the above ImportError problem. There may be some incompatibility between the _galsim library and the back-end libgalsim library. It may be that import galsim doesn't notice the incompatibility, but when the code tries to do something that uses the back-end C++ code, there is an incompatibility and you get a segmentation fault.
The solution is the same. Delete the old, now invalid libgalsim library, and rebuild GalSim.