Conference call notes 20170329 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 73rd EasyBuild conference call, Wednesday March 29th 2017 (5pm - 6pm CET)

Attendees

Alphabetical list of attendees (5):

  • Damian Alvarez (JSC, Germany)
  • Davide Vanzo (Vanderbilt University)
  • Markus Geimer (JSC, Germany)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Bart Oldeman (McGill University, Canada)

Agenda

  • report on benchmarking of Python/numpy built with GCC/Intel compilers + Intel MKL by Damian
  • boegelbot reporting broken unit tests in PRs
  • moderator for EB conf calls on April 12th & 26th?
  • Q&A

Notes

Benchmarking of Python/numpy
  • JSC is testing the performance of a Python installation in GCCcore, but a numpy library in the top level of the toolchain hierarchy (e.g. with Intel compilers, in a bundle with other scipy-stack libraries).
    • For most Python related things, it didn't seem to matter whether Python was built with GCC or other compilers
      • PGI doesn't built Python properly...
      • so (minimal version of) Python was installed with GCCcore to provide it everywhere
    • The benchmarking was done using https://github.com/serge-sans-paille/numpy-benchmarks
    • Most benchmarks perform satisfactorily
      • Significant slowdown in functions that rely on symbols defined by libm
    • What happened before (interpreter compiled with icc):
      • Everything was compiled with icc, so libimf was used instead of libm
    • What happened after having a base python in GCCcore:
      • Even though numpy itself was compiled with icc, when launching the interpreter (compiled with gcc), libm was loaded, and the exp and related symbols are picked from there, instead of the faster libimf
      • can be worked out/verified by preloading libimf
    • Workaround at the moment:
      • The SciPy-Stack bundle installs a python interpreter compiled with icc, that gets precedence over the one loaded in GCCcore
        • 2 (well 3) pythons available via $PATH...
      • The Python module defines PYTHONPATH (to use all the modules installed there) since different python is being used... images/numpy_benchmarks.png
    • some differences may be explained by comparing different numpy versions
    • largest differences (~2x slowdown) explained by libm vs libimf
    • see vibr_energy, log_likelihood and arc_distance
    • See https://github.com/numpy/numpy/issues/8823
    • Is statically linking libimf into numpy a solution?
    • Bart: will also check whether they are hitting this with Python provided via Nix (together with wrapper module) + scipy stack compiled with EasyBuild with Intel compiler
    • related to https://github.com/hpcugent/easybuild-easyblocks/issues/1136?
      • this needed to be fixed, but not related to performance issues
boegelbot reporting broken unit tests in PRs
Q&A

  • Davide: ncurses --with-termlib issue: https://github.com/hpcugent/easybuild-easyconfigs/issues/4049

    • mainly triggered for Python 3 which requires XZ, to dance around cyclic dep between XZ-gettext-ncurses
    • why was --with-termlib added to ncurses? because of libreadline? cfr. https://github.com/hpcugent/easybuild-easyconfigs/pull/3545
    • should figure out why ncurses is being picked up from Core/ in the first place...
      • --minimal-toolchains is enabled, but --add-dummy-to-minimal-toolchains was not
      • Lmod is loading ncurses/6.0 from Core while it shouldn't? wrong $MODULEPATH?
    • possible solutions:
      • add another 'configure' loop iteration without --with-termlib to installed => libncurses will not have termlib symbols stripped out, but libtinfo will be available...
      • give ncurses with dummy a specific versionsuffix to avoid it being picked up as ncurses/6.0

  • Davide: plan to enhance MATLAB easyblock to fix problems with recent versions

  • Markus: feedback on extra parameters related to documentation?

  • Markus: Clang as toolchain compiler without Fortran support

    • is Fortran support strictly required by the EasyBuild toolchain mechanism?
      • if it is, then we could make it more flexible
    • only for tools/apps that don't require Fortran support
    • unclear error message blocked progress on this, Markus will come back to it

  • Markus: update on EasyBuild tutorial proposal for SC17?