Conference call notes 20240911 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 254th EasyBuild conference call, Wednesday 11 Sept 2024 (08:00 UTC)

Attendees

List of attendees (15):

  • Simon Branford (Univ. of Birmingham, UK)
  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Davide Grassano (CECAM, Switzerland)
  • Jasper Grimm (University of York, UK)
  • Alexander Grund (ZIH, Dresden, Germany)
  • Leonardo Honfi Camilo (Univ. of Wageningen, Netherlands)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Georgios Kafanas (University of Luxembourg)
  • Sébastien Moretti (SIB, Switzerland)
  • Jure Pečar (EMBL, Heidelberg, Germany
  • Jan Reuter (JSC, Germany)
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Alexandre Strube (JSC, Germany)
  • Cintia Willemyns (Vrije Universiteit Brussel, Belgium)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024a/2024b update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.2 (12 June 2024)
  • next EasyBuild release:
    • likely EasyBuild 4.9.3 very soon (days)
      • status:
        • framework: no more changes planned, see 4.9.3 milestone
        • easyblocks: small set of PRs to be merged, see 4.9.3 milestone
        • easyconfigs: only bug fix PRs (and whatever gets merged before 4.9.x branch is updated with current develop to prepare for release), see 4.9.3 milestone
  • EasyBuild v5.0.0
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • allow for case where homepage = None when generating the docs (PR #4626)
    • enhancements
      • add support for --extra-source-urls to fetch sources from additional URLs (PR #4079)
      • only call _sanity_check_step_extensions if --skip-extensions is not set (PR #4620)
      • Add support for --software-commit and an associated template %(software_commit)s (PR #4628)
    • changes
      • ...
    • code cleanup
      • ...
    • EasyBuild 5.0 (separate 5.0.x branch)
      • enable --module-extensions by default (+ resolve template values used in extension version) (PR #4501)
      • Use default value $XDG_CONFIG_DIRS from XDG basedir spec: /etc/xdg (instead of /etc) (PR #4591)
        • warning is printed if /etc/easybuild.d/*.cfg is found
      • convert template constant lists to dicts and export the constants by name (PR #4595)
      • move verifying of checksums from source to fetch step, to include it with --fetch (PR #4624)
      • changes to support for using Tcl-based Environment Modules as modules tool (instead of Lmod):
        • drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
        • Run unit tests on an updated versions of Modules: v4.5.3 + v5.3.1 #4415 (PR #4415)
        • add check_group support for module files in Tcl syntax (PR #4418)
        • bump minimum required Tmod (4.x) to 4.3.0 (PR #4425)
        • Use getenv modulefile command with EnvironmentModules >= 4.2.0 (PR #4614)
        • Add module cache build support on EnvironmentModules (PR #4615)
        • derive EnvironmentModules class directly from ModulesTool rather than from to be deprecated EnvironmentModulesTcl (PR #4625)

  • easyblocks (merged PRs)

    • bug fixes
      • ...
    • enhancements
      • update custom easyblock for PETSc to consider include/suitesparse subdirectory for SuiteSparse headers (PR #3391)
      • enhance custom easyblock for GCC to use with-arch option for nvptx with 13.1+ (PR #3396)
      • disable the version check of pip in Python packages by defining $PIP_DISABLE_PIP_VERSION_CHECK (PR #3427)
    • updates
      • ...
    • changes
      • ...
    • new easyblocks
      • ...
    • code cleanup
      • ...
    • EasyBuild 5.0 (separate 5.0.x branch)
      • use run_shell_cmd in custom easyblock for ABAQUS (PR #3414)
      • use PYPI_SOURCE constant in generic PythonPackage easyblock + custom easyblock for Python (PR #3410)
      • remove (20) unused software-specific easyblocks (PR #3423)
      • remove unused easyblock for BamTools (PR #3424)
      • add check to make sure that easyblocks do not use run_cmd or run_cmd_qa (PR #3426)

  • easyconfigs (merged PRs)

    • ~60 easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • disable test step for WIEN2k 23.2 because files required by it can no longer be downloaded (PR #20969)
      • add patch for HDF5 v1.14.3 to suppress fp exceptions (PR #21280)
      • consistently fix homepage + source URL for HDF + h4toh5 (PR #21134)
      • correct extension order in R-bundle-CRAN 2024.06 (PR #21285)
      • update easyconfig for dorado 0.7.3 to properly use openssl from EB and not install external libraries into its own lib directory (PR #21297)
      • use source tarballs from GitHub for recent libdap easyconfigs (PR #21334)
      • remove Highway build dependency in Brunsli easyconfigs, since it's not actually required at all (PR #21366)
    • enhancements
      • ...
    • (noteworthy) new software
      • ...
    • noteworthy software updates
      • ...
    • changes
      • use stand-alone absl-py as dependency for jax w/ gfbf/2023a + CUDA 12.1.1 (PR #21039)
      • reorganize Flax/JAX stack in 2023a: scvi-tools, Flax, Optax, jax, absl-py (PRs #21038)
    • EasyBuild 5.0 (separate 5.0.x branch)
      • remove unnecessary use of an easyblock for BamTools (PR #21263)

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

    • bug fixes
      • ...
    • enhancements
      • Copy build log and artifacts to a permanent location after failures (WIP PR #4601)
    • changes
      • ...
    • EasyBuild 5.0 (separate 5.0.x branch)
      • project board: https://github.com/orgs/easybuilders/projects/18/views/2
      • Enable module-depends-on by default (PR #4500)
        • TODO: only use depends_on when using Lmod as modules tool, module load is basically equivalent for recent versions of Environment Modules
      • More granular exit codes (PR #4534)
      • prepend to $PYTHONPATH or $EBPYTHONPREFIXES in generated module files by automatically scanning for python site package directories (PR #4539)
      • Problem using $CPATH in modulefiles overwriting system paths (issue #3331)
        • we'll probably handle this by introducing an extra EasyBuild configuration option like --c-cxx-header-path, which will be CPATH by default, but can be set to INCLUDE_PATH which implies not updating $CPATH and updating $C_INCLUDE_PATH, $CXX_INCLUDE_PATH and $INCLUDE_PATH instead;
      • Rename source step (issue #4627)

  • easyblocks (open PRs + issues)

    • bug fixes
      • ...
    • enhancements
      • Set CMake hints for path to Python installation if it's direct or indirect dependency (when using CMake >= 3.12) (PR #3282)
      • enhance LLVM easyblock for compilation of clang/flang + other llvm-projects (PR #3373)
      • Update NCL easyblock for intel (PR #3409)
      • enhance OpenFOAM easyblock to also build the plugins for OpenFOAM >= v2406 (PR #3436)
      • allow Python bundles and packages to specify a maximum Python version for the system toolchain (PR #3431)
      • enhance binutils easyblock to explicitely pass msgpack configure option (PR #3438)
    • updates
      • update sanity check in OpenFOAM easyblock, since there's no modifyMesh in OpenFOAM 12 (PR #3434)
      • fix sanity check for Geant4 >= v11.2 (PR #3439)
      • update custom easyblock for CP2K >= v2024 (PR #3433)
      • update AOCC easyblock to work with LLVM 16 and above (PR #3437)
    • changes
      • ...
    • new
      • ...
    • EasyBuild 5.0 (separate 5.0.x branch)
      • Use prefer_ebpythonprefixes flag to use EBPYTHONPREFIXES (PR #3343)
      • enhance CMakeMake easyblock to check whether correct Python installation was picked up by CMake (PR #3399)
      • run pip check only once for PythonBundle (PR #3428 + PR #3432)

  • easyconfigs (open PRs + issues)

    • bug fixes/reports
      • add alternative checksum for bold 1.3.0 extension in R-bundle-CRAN (PR #21370)
    • enhancements
      • add the plugins to OpenFOAM v2406 (PR #21332)
    • (noteworthy) new software
      • ...
    • software updates
    • changes
      • unbundle tensorboard from TensorFlow v2.15.1 (PR #21337)
    • EasyBuild 5.0 (separate 5.0.x branch)
      • ...

2024b common toolchains

  • GCC 14.2 as a base (see easyconfigs PR #21114)
  • we should define candidate toolchains using latest version of all components
    • are newer versions of Intel components compared to intel/2024a already available?
  • should we consider replacing intel with Clang-based toolchain as 2nd common toolchain next to foss?
    • Fortran frontend in Clang is becoming mature enough?

Q&A / others

  • Alexandre: how does EasyBuild deal with source tarballs being changed in-place upstream (without version bumps)
    • there's support for "alternative checksums" in case the newly served tarball is benign (no code changes), see for example easyconfig PR #21370
    • sources that have disappeared only can be made available via https://sources.easybuild.io (no automation in place there to add stuff, ask @boegel)
      • but this doesn't help with source tarballs that are suddenly different upstream, only for failing downloads since it's used as a last resort;
    • we also report problems with upstream changes in source tarballs, but sometimes without much success, see for example upstream issue for libxc
  • EasyBuild 5.0
    • last bit of effort will focus on extending the documentation covering the changes
      • https://docs.easybuild.io/easybuild-v5 will become a one-page overview with breaking changes, etc. and links to other pages with more details
      • a webinar will be planned shortly after the release of EasyBuild 5.0 in which we'll present in detail what has changed, and how it may affect your EasyBuild setup (or not)