Developing on SourceXtractor in Conda - embray/SourceXtractorPlusPlus GitHub Wiki

Developing on SourceXtractor++ in Conda

SourceXtractor++ was developed as part of the software suite for the Euclid mission. It was originally developed within a rather rigid and complex development environment built around a C++ build tool based on cmake called Elements. I say this not as a criticism, because I'm sure it made sense at the time, but I found it incredibly difficult to set up an environment for doing development on the project on my own system, outside whatever rigid assumptions the project makes by default (use of Fedora, a specific non-standard installation layout, etc.)

The project does provide conda packages for all their releases, so for the end user this is not a major stumbling block. In principle the conda build recipes could also suggest a path towards setting up a development environment. I did find this a little helpful, though not altogether transparent either.

So here are some minimal steps that at least worked for me, though YMMV:

Setup

  • Install and set up a recent copy of the Miniforge conda distribution, which also includes mamba (faster dependency resolver for conda packages).

  • Create a new conda environment for developing in, I called mine sourcex-dev:

    $ mamba create -n sourcex-dev -c astrorama \
    alexandria=2.32.0 \
    gcc_linux-64=13.3.0 \
    gxx_linux-64=13.3.0 \
    fftw \
    gsl \
    lxml \
    wcslib

    Here -c astrorama selects to install packages from the astrorama conda release channel--this is needed for Alexandria, the a C++ utility library used by the other astrorama projects. I found specifically that version 2.32.0 works with the current development branch of SE++. I also found at least GCC >=13 was needed. Other dependencies including FFTW, GSL, lxml, and WCSLIB seemed sufficient to build a working version.

  • Now activate the environment:

    $ mamba activate sourcex-dev

  • It's also necessary to set the CC and CXX flags to point to the compilers installed in the conda environment:

    export C=${CONDA_PREFIX}/bin/x86_64-conda-linux-gnu-gcc
export CXX=${CONDA_PREFIX}/bin/x86_64-conda-linux-gnu-g++

Building

  • Clone the SourceXtractor++ git repository, either the official one or my fork for example:

    $ git clone [email protected]:embray/SourceXtractorPlusPlus.git
$ cd SourceXtractorPlusPlus

  • As is common practice for cmake-based builds, it's highly advisable to create a build directory into which it will place build artifacts:

    $ mkdir build/
$ cd build/

  • And this is the cmake incantation that worked well enough for me:

    $ cmake -DCMAKE_PREFIX_PATH=$CONDA_PREFIX \
        -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX \
        -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
        -DSGS_SYSTEM=x86_64-linux-gcc133 \
        -Wno-dev ..

  • That should run fairly quickly, and assuming it succeeds you can then simply build:

    $ make -j8

    Use of -jN if you have a number of CPUs at your disposal is highly recommended.

Running

You could install system-wide at this point or into your conda environment, though I have not braved it.

Inside the build/ directory you'll find a shell script called run which as far as I can tell does some tricks to set up the runtime environment correctly, and a bin/ directory containing a bunch of test executables (I think passing -DELEMENTS_BUILD_TESTS:BOOL=OFF to cmake will disable this, but I haven't tried it), as well as the sourcextractor++ executable itself. For some reason this does not work properly if you try to run it without going through the run script, so run it as:

$ ./run bin/sourcextractor++

Possibly this could be bypassed by configuring your environment as suggested in the generated SourceXtractorPlusPlusBuildEnvironment.xml file (which is actually read in by the run script). But it's straightforward enough so probably not worth the hassle.

Loading the runtime environment variables into your environment

As previously mentioned, the ElementsProject build system outputs a script called run through which you have to call executables built by its build system (at least in build/debug mode). This turns out to actually be a wrapper around a Python script which:

  1. Sets up the environment by reading an XML file (!!), the aforementioned SourceXtractorPlusPlusBuildEnvironment.xml.
  2. It passes this environment to and runs your command through a naïve call to subprocess.Popen in the Python script.

This means that if you want run your build outputs through this in anything that uses e.g. readline (say, debugging it in gdb) it will break readline and make a mess of your terminal to boot (I found this especially aggravating while working in tmux).

The same Python script that sets up the runtime environment can also be used to just dump the environment variables as follows. Here, assuming we're in the Elements project conda environment you can run:

$ python "${CONDA_PREFIX}/lib/cmake/ElementsProject/scripts/env.py" --sh --xml SourceXtractorPlusPlusBuildEnvironment.xml

This dumps the environment variables in shell syntax, with this strange leading marker .=. which is not valid, so we have to trim it:

$ eval $(python "${CONDA_PREFIX}/lib/cmake/ElementsProject/scripts/env.py" --sh --xml SourceXtractorPlusPlusBuildEnvironment.xml | tail -n +2)

...and this should work.

I use direnv (recommended) and put that into an .envrc file for my build directory, like:

source ~/.local/opt/miniforge3/etc/profile.d/conda.sh
conda activate sourcex-dev
eval "$(python "${CONDA_PREFIX}/lib/cmake/ElementsProject/scripts/env.py" --sh --xml SourceXtractorPlusPlusBuildEnvironment.xml | tail -n +2)"

No more having to pass build artifact commands through run.

Coda (building without conda?)

I initially tried setting up the build environment without even conda, thinking I could just merrily install dependencies into my ~/.local/` prefix until it worked. This meant also building and installing the Elements build system itself, and Alexandria, along with all the build dependencies which included a chunk of boost:

  • libboost-dev
  • libboost-filesystem-dev
  • libboost-program-options-dev
  • libboost-regex-dev
  • libboost-test-dev
  • libboost-thread-dev
  • liblog4cpp5-dev
  • libccfits-dev
  • libopenblas64-openmp-dev
  • wcslib-dev

But I gave up on this along the way. In particular it wanted some Python 2.7 bindings for boost that are not even available in my system's package repository anymore. Installing Alexandria from astrorama's conda package for it brought in most everything else needed for SE++.