Conference call notes 20190403 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 122nd EasyBuild conference call, Wednesday Apr 3rd 2019 (17:00 - 18:00 CEST)

Attendees

Alphabetical list of attendees (7):

• Damian Alvarez (JSC, Germany)
• Fotis Georgatos (SDSC, Switzerland)
• Bart Oldeman (ComputeCanada)
• Victor Holanda (CSCS)
• Kenneth Hoste (HPC-UGent)
• Åke Sandgren (Umeå University, Sweden)
• Davide Vanzo (Vanderbilt University)

Agenda

• outlook to EasyBuild v3.9.0
• update on progress towards support for "eb --remove"
• the road towards EasyBuild 4.0
• Q&A

Outlook to EasyBuild v3.9.0

• support for multi_deps is merged into framework

  • see https://github.com/easybuilders/easybuild-framework/pull/2813
  • used by SciPy-bundle easyconfigs to install for both Python 2.7.15 & 3.7.2 in a single directory
  • also required changes to PythonPackage & PythonBundle
    • cfr. https://github.com/easybuilders/easybuild-easyblocks/pull/1664 & https://github.com/easybuilders/easybuild-easyblocks/pull/1671
  • multi_deps support broke --check-conflicts and --dep-graph checks in easyconfig PRs
    • fixed soon after that popped up:
      • https://github.com/easybuilders/easybuild-framework/pull/2826
      • https://github.com/easybuilders/easybuild-framework/pull/2827

• pending/blocking release:

  • framework:

    • https://github.com/easybuilders/easybuild-framework/milestone/65
    • loading first version listed in multi_deps by default
      • cfr. https://github.com/easybuilders/easybuild-framework/pull/2828
      • required to limit impact on end users for Python/SciPy-bundle split
        • since SciPy-bundle requires Python to be loaded to actually work...
      • Davide: remark wrt order of versions in multi_deps
      • depends_on should be guarded as well (+ or mode == unload)
    • minor bugs for which PRs are already open can hopefully be squashed before the release as well...
      • Damian fix for FFTW libs: incomplete, does not cover $FFT_* env vars (Kenneth will take a look)
        • https://github.com/easybuilders/easybuild-framework/pull/2822
      • Alan's fix: fixes reported issue with --software (Kenneth will take a look)
        • https://github.com/easybuilders/easybuild-framework/pull/2732
      • Kenneth's fix for --new-pr/--update-pr w.r.t. finding where patch belongs (Davide will take a look)
        • https://github.com/easybuilders/easybuild-framework/pull/2803

  • easyblocks

    • https://github.com/easybuilders/easybuild-easyblocks/milestone/56
    • no blockers?
    • maybe change to CMakeMake w.r.t. preventing that Boost is picked up from the system can be included
    • https://github.com/easybuilders/easybuild-easyblocks/pull/1618
    • after tackling Alan's remark
    • Åke: small fix for MesonNinja easyblock: https://github.com/easybuilders/easybuild-easyblocks/pull/1673

  • easyconfigs

    • https://github.com/easybuilders/easybuild-easyconfigs/milestone/59
    • would be nice to have an easyconfig for TensorFlow 1.13.1 using foss/2019a
      • to prove that Python/SciPy-bundle split & multi-Python installations work as intended...
    • any important (open) bug fix PRs that should be bumped to 3.9.0 milestone?

Update on progress towards support for "eb --remove"

• first step for a (safe) implementation of "eb --remove" is to be able to determine reverse dependencies

  • cfr. https://github.com/easybuilders/easybuild-framework/pull/2783

  • turns out that's not so trivial if you consider both flat & hierarchical modules names schemes...

  • eb --remove should take easyconfig filename, not module filename

    • that helps in HMNS context to know where to check in hierarchy...

• will not be ready for EasyBuild v3.9.0 (hopefully by EasyBuild 4.0)

The road to EasyBuild v4.0

• overview of planned changes & current status on progress: https://github.com/easybuilders/easybuild/issues/447

• porting framework/easyblocks/easyconfigs repos to also support Python 3 is considered done (no known issues)

  • only in separate 4.x branch, not (yet) in develop (which is still for 3.x for now)
  • cfr. https://github.com/easybuilders/easybuild-framework/issues?q=label%3Apython3+is%3Aclosed
  • Victor: NAMD, GROMACS, Amber with Cray/foss toolchains works with EB 4.x branch on top of Python 2/3
  • Åke: more to check: TensorFlow, Qt5, ... (big easyblocks)
  • Kenneth will try to run a (partial) regtest with eb on top of Python 3 (after EasyBuild v3.9.0 release)

• next up is getting rid of setuptools as required dependency...

  • WIP by Kenneth
  • actually less of a problem than first anticipated
  • cfr. https://github.com/easybuilders/easybuild-framework/compare/4.x...boegel:pkgutil_namespace?expand=1
  • needs to be revisited/updated/synced with current develop/4.x...

• other planned 'major' changes:

  • rename dummy toolchain to system toolchain
    • trickier than it sounds, lots of small things to take into account
    • using two names for one thing (because we only want to deprecate 'dummy', not stop supporting it) doesn't help
  • easyblock for OpenMPI
    • more sensible default configuration w.r.t. things like Slurm integration, etc.
    • should not get in the way of sites who already heavily tweak OpenMPI configuration (via easyconfigs, hooks, ...)
  • (hopefully) stable --containerize support (https://easybuild.readthedocs.io/en/latest/Containers.html)

• ETA for EasyBuild 4.0 is (still) by summer 2019...

Q&A

• Bart: how many packages use MPI version of FFTW?

  • only PETSc?
  • Åke: also VASP
  • Victor: GROMACS too? (Bart: no)
  • FFTW wrappers in MKL requires MPI
  • could consider removing FFTW MPI from toolchain?
    • separate FFTW-MPI as dependency where needed
  • worth considering for 2019b generation of easyconfigs?
    • which would be first common toolchain version in EasyBuild v4.0
  • should check whether this has impact on framework w.r.t. toolchain build env prepare

• Victor: flex/Bison deps for binutils is a bit of a mess

  • different versions, no longer needed since they're only build deps for build deps
  • unclear why they are needed, pollute user environment
  • Kenneth: done because of bootstrapping, to shield ourselves from the OS
  • Åke: could introduce the concept of bootstrap-only build dependency that are trashed after use

• Victor: support for containerize to have layered builds

  • helps a lot with getting minimal container images
  • not supported yet in eb --containerize, but since now both Singularity & Docker has this, it's interesting...

• issues will be opened in framework repo for each of these to follow up...