Conference call notes 20241120 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 259th EasyBuild conference call, Wednesday 20 Nov 2024 (16:00 UTC / 17:00 CET)

Attendees

List of attendees (9):

  • Eesaan Atluri (University of Alabama at Birmingham, USA)
  • Leonardo Honfi Camilo (Univ. of Wageningen, Netherlands)
  • Georgios Kafanas (University of Luxembourg)
  • Sebastien Moretti (SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland)
  • Bart Oldeman (McGill University, Digital Research Alliance of Canada)
  • Jure Pečar (EMBL)
  • Jan Reuter (JSC, Germany)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Roberto Scipioni (Red Oak Consulting UK)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024b update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.4 (22 Sept 2024)
  • next EasyBuild release:
    • very likely EasyBuild v5.0.0 :fire:
    • additional EasyBuild 4.9.x versions could still be done via 4.9.x branches, but none are planned currently
  • EasyBuild v5.0.0
    • project board: https://github.com/orgs/easybuilders/projects/18/views/2
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • ...
    • enhancements
      • [5.0.x] Allow use of custom delimiter for paths in module generator (PR #4687)
    • changes
      • [develop] Added possibility to call amend/try-amend multiple times from easystack (PR #4667)
      • [5.0.x] mark support for installing extensions in parallel as being mature, since it's no longer experimental (PR #4672)
      • [5.0.x] mark easystack support as being mature, since it's no longer experimental (PR #4673)
    • code cleanup
      • ...

  • easyblocks (merged PRs)

    • bug fixes
      • [develop] revert changes from PR #3472, seems to be leading to an infinite loop in easyconfigs test suite?(PR #3504)
    • enhancements
      • [develop] fix setting of CMAKE_OPTIONS for SuiteSparse versions between 5.1.2 and 6.0.0 (PR #3471)
      • [develop] enhance generic Bundle EasyBlock to transfer module requirements of components (PR #3472)
      • [develop] enhance cargo easyblock for sources from git repositories (PR #3483)
      • [develop] gromacs: make it possible to disable single precision as build target. (PR #3501)
    • updates
      • [develop] Enhance NAMD easyblock to support NAMD v3.0 (PR #3494)
    • changes
      • ...
    • new easyblocks
      • ...
    • code cleanup
      • ...

  • easyconfigs (merged PRs)

    • ~80 easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • [develop] strip Suppressor out of ResistanceGA (PR #20106)
      • [develop] use skani rather than FastANI as dependency for GTDB-Tk v2.4.0 (PR #21518)
      • [develop] add patch for SciPy-bundle 2023.07 + 2023.11 to fix vectorization bug in scipy 1.11.1 (PR #21693)
      • [develop] Change tagbot workflow to run on pull_request_target and comment on security (PR #21779)
      • [develop] Fix tagbot concurrency group (PR #21793)
      • [develop] Actually perform temporary merge in tagbot workflow to determine PR changes (PR #21795)
      • [develop] add path to Faiss shared libraries to $LD_LIBRARY_PATH (PR #21799)
      • [develop] add patch to fix vectorization bug in scipy 1.11.1 also to SciPy-bundle 2023.07 w/ iimkl/2023a (PR #21805)
      • [develop] add missing hatchling build dependency for epiScanpy 0.4.0 w/ foss/2023a (PR #21809)
      • [develop] add missing hatchling build dependency for scArches 0.6.1 w/ foss/2023a (PR #21844)
    • (noteworthy) new software
      • ...
    • noteworthy software updates
      • ...
    • cleanup
      • ...
    • changes
      • ...

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

    • bug fixes
      • [5.0.x] improve portability of reproducible tarballs by replacing external tar command with tarfile module (PR #4660)
      • [develop] fix dry-run output when using multi_deps (PR #4678)
      • [5.0.x] make LooseVersion('1.0') == LooseVersion('1') (PR #4691)
    • enhancements
      • [develop] Copy build log and artifacts to a permanent location after failures (WIP PR #4601)
      • [develop] allow extensions to add text to the module file (PR #4652)
      • Problem using $CPATH in modulefiles overwriting system paths (issue #3331)
        • [5.0.x] add --search-path-cpp-headers configuration option to control how EasyBuild sets paths to headers at build time (PR #4645)
        • [5.0.x] refactor generation of required environment variables in module files (PR #4653)
          • test was added to verify that making ModuleLoadEnvironment a singleton causes trouble
          • needs more love, additional test is broken
          • easyblocks will need to be updated accordingly to not use make_module_req_guess anymore (since it was deprecated)
        • [5.0.x] add module-search-path-headers configuration option to control how modules set search paths to header files (PR #4655)
      • [develop] add support for injecting checksums for cargo crates (PR #4661)
      • [5.0.x] Add function to modify path-like environment variable in a context (PR #4681)
      • [5.0.x] Add global option to set default toolchain option of "debug" (PR #4688)
      • [5.0.x] add a CUDA device code sanity check (PR #4692)
      • [5.0.x] add support for developer mode (WIP PR #4693)
        • allows to specify a particular prepared unpacked source directory rather than unpacking source tarball
      • [5.0.x] add option search-path-linker to control linker options at build time (PR #4697)
      • [develop] enhance get_software_libdir to return full paths if requested (PR #4699)
    • code cleanup
      • [5.0.x] simplify code for determining the $PYTHONPATH module entries (PR #4686)
      • [5.0.x] simplify loop in collect_exts_file_info (PR #4689)
    • changes
      • [5.0.x] Rename SOURCE_STEP to EXTRACT_STEP (PR #4629)
      • [5.0.x] Deprecate use of parallel easyconfig parameter and fix updating the template value (PR #4580)
      • [5.0.x] Deprecate post_install_step, rename it to post_processing_step (issue #4656)
      • [5.0.x] proposed changest to add a timeout to run_shell_cmd (PR #4665)
      • [5.0.x] Let jobs retweak easyconfigs themselves (draft PR #4669)
      • [5.0.x] Refactor make_extension_string (PR #4690)
      • Honor SOURCELOWER_TAR_GZ in source_tmpl of extensions (issue #4695)
      • Utility for --check-patches (issue #4696)
      • [5.0.x] Add dashes directly in compiler optimization flags (PR #4698)
      • [develop] filter the installed EC to remove the statistics and built with version XX (PR #4702)
      • [5.0.x] change semantics of --dry-run, so it doesn't imply --robot (PR #4704)
      • Trigger rebuild on change of easybuild config (issue #4705)
      • Extensions can should not be allowed to impact global environment during builds (issue #4706)
      • [develop] EasyBlock: Allow to pass logfile from other EasyBlock (PR #4707)
      • Integrate easy_update capabilities into EasyBuild (issue #4708)
      • add an option to run eb with a debugger such as pdb (issue #4709)

  • easyblocks (open PRs + issues)

    • bug fixes
      • [5.0.x] run pip check only once for PythonBundle (PR #3432)
      • [develop] Avoid failure in Python sanity check when PIP_REQUIRE_VIRTUALENV is set (PR #3460)
      • [develop] take into account that f90wrap_version may be None in MetalWalls easyblock (PR #3468)
      • ORCA Easyblock does not install packaged executable in ORCA 6.0.0 (issue #3479)
      • [develop] tweak libpaths in TensorFlow easyblock by adding directory containing libnccl.so.2 (PR #3497)
      • [develop] Unset PMIX variables when configuring OpenMPI (PR #3511)
      • [develop] Minor changes to make hotfixes 2441 and 2424 working (PR #3512)
    • enhancements
      • [develop] ignore user python packages by using -s option when running Python code via "python -c" (PR #3440)
      • [5.0.x] PythonPackage fix python shebangs by default (PR #3499)
      • [5.0.x] add support for optarch to GoPackage easyblock (PR #3503)
      • [develop] QuantumESPRESSO: Let internal EasyBlock not create a log file (PR #3505)
      • [develop] Reintroduce #3472 (transfer Bundle requirements to module) but do not create logfile in components (PR #3509)
    • updates
      • [develop] update custom easyblock for CP2K >= v2024 (PR #3433)
      • Update numpy easyblock for numpy>=2.0 (issue #3508)
      • [develop] Add support for STAR-CCM+ installs with .aol installer (PR #3510)
    • changes
      • Put our Python sitecustomize.py into the site-packages folder (issue #3493)
      • Should fix_python_shebang_for="bin/*" be default in PythonPackage? (issue #3498)
      • Sanity check python package versions (issue #3507)
    • new
      • [develop] add custom easyblock for Bandicoot (PR #3380)
      • [develop] custom easyblock for DeepSpeed (PR #3450)

  • easyconfigs (open PRs + issues)

    • bug fixes/reports
      • [develop -> 5.0.x] switch to CargoPythonBundle easyblock for orjson 3.9.15 (PR #21392)
        • requires being able to produce reproducible source tarballs from Git repository (WIP in 5.0.x branch of EasyBuild framework)
      • [5.0.x] promote pybind11 to runtime dependency for CPPE (to fix pip check failure) (PR #21564)
      • [develop] bugfix of HTSplotter-2.11 (PR #21812)
      • [develop] add missing dependencies (TSEBRA, compleasm) + use more recent AUGUSTUS version for BRAKER v3.0.8 (draft PR #21837)
      • [develop] build CORSIKA v77550 in installation directory (PR #21868)
      • [develop] Fix confliction platformdirs dependency in Pylint-3.2.5-GCCcore-13.2.0 (PR #21873)
    • enhancements
      • [develop] ignore user-installed packages when running python in sanitycheck (PR #21362)
      • [develop] Update workflows to 24.04 + update actions (PR #21826)
      • [develop] Convert jupyter-resource-usage ECs to PythonPackage (PR #21863)
      • [develop] detect use of deprecated behavior in test runs (PR #21885)
    • (noteworthy) new software
      • ...
    • software updates
      • [develop] PyTorch v2.3.0 (PR #20489)
        • blocked by changes in how results of PyTorch test suite are reported
    • changes
      • [5.0.x] Replace parallel by maxparallel (PR #19375)

2024b common toolchains

  • GCC 14.2 as a base (see easyconfigs PR #21114)
  • we should define candidate toolchains using latest version of all components
    • are newer versions of Intel components compared to intel/2024a already available?
    • 2025.0 version removes ifort (classic Fortran compiler), replaced with ifx (oneAPI Fortran compiler)
  • should we consider replacing intel with Clang-based toolchain as 2nd common toolchain next to foss?
    • Fortran frontend in Clang is becoming mature enough?

Q&A / others

  • EUM'25 will be 25-27 March 2025 at Jülich Supercomputing Centre
  • Jorg: Intel compiler 2025 on Ubuntu 18, glibc too old, should we check in easyblock?
  • Leonardo: how to check for updates, which easyconfigs have updated between versions? Can check with reprod folder, release notes from Kenneth.
  • Roberto: Python package installation, challenges with GUIs. Suggested to use Slack and GitHub issues.