Conference call notes 20240228 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 241th EasyBuild conference call, Wednesday 28 Feb 2024 (09:00 UTC)

Attendees

List of attendees (19):

Alex Domingo (Vrije Universiteit Brussel, Belgium)
Davide Grassano (CECAM, Switzerland)
Jasper Grimm (University of York, UK)
Alexander Grund (TU Dresden, Germany)
Jorge Guerra (Universidad Politécnica de Madrid, Spain)
Sven Hansen (RWTH, Germany)
Thomas Hoffman (EMBL, Heidelberg, Germany)
Kenneth Hoste (HPC-UGent, Belgium)
Adam Huffman (Big Data Institute, Oxford, UK)
Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
Sam Moors (Vrije Universiteit Brussel, Belgium)
Sébastien Moretti (SIB, Switzerland)
Mikael Öhman (Chalmers University of Technology, Sweden)
Jure Pečar (EMBL, Heidelberg, Germany)
Jörg Saßmannshausen (Imperial College London, UK)
Åke Sandgren (Umeå University, Sweden)
Alexandre Strube (JSC, Germany)
Alain van Hoof (TU Eindhoven, Netherlands)
Lars Viklund (Umeå University, Sweden)

Agenda

overview of recent developments
Q&A

Recent developments

latest EasyBuild release: 4.9.0 (30 Dec 2023)
- see release notes + highlights
next EasyBuild release: hopefully EasyBuild v5.0.0
- there will only be an EasyBuild v4.9.1 if there's a clear need for it, or if getting EasyBuild 5.0 is taking too long (way beyond 8 weeks after v4.9.0 release)

ETA for EasyBuild 5.0 release: hopefully first EasyBuild release of 2024

started doing short sprint meetings, each Monday at 15:00 CE(S)T to set next 5 goals to tackle that week
monthly sync meetings every 1st Wed of the month at 14:00 CE(S)T: next today Wed 7 Feb'24 at 14:00 CET
see also https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA

to test development version of EasyBuild 5.0:

# set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version

recent changes
- docs (merged PRs)
  - ...
- framework (merged PRs)
  - bug fixes
    - use cp -dR instead of cp -a for shell script extraction (PR #4465)
  - enhancements
    - ...
  - changes
    - Auto-enable Fortran oneAPI compilers (issue #4455)
  - code cleanup
    - ...
  - EasyBuild 5.0 (to 5.0.x branch)
    - bug fixes
      - ...
    - changes
      - ...
    - enhancements
      - add support for running shell commands asynchronously with run_shell_cmd (PR #4444)
        
        there's still a bug there somewhere, test_toy_exts_parallel sometimes fails in CI (more often on Python 3.6)
      - allow only alphanumeric characters in the output filename used in run_shell_cmd (PR #4454)
      - enhance download instructions by mentioning active source path (PR #4459)
      - add cuda compute capability integer format templates (PR #4463)
- easyblocks (merged PRs)
  - bug fixes
    - ...
  - enhancements
    - honor pretestopts & testopts in PerlModule (PR #3231)
  - updates
    - ...
  - changes
    - ...
  - new easyblocks
    - ...
  - EasyBuild 5.0 (to 5.0.x branch)
    - bug fixes:
      - ...
    - changes:
      - update run_async methods in RPackage, Rmpi, and Rserve easyblocks to use submit run_shell_cmd call into thread pool (PR #3163)
- easyconfigs (merged PRs)
  - over 100 easyconfig PRs were merged since last conf call
  - bug fixes/reports
    - remove numa configure option from hwloc 2+ (PR #19833)
    - add patch to fix MS_RDONLY error in OpenMPI 4.0.x (PR #19837, PR #19944)
    - Fix test failures of Perl-bundle-CPAN-5.38 in non-English locales (PR #19971)
    - add patch for adding a write memory barrier to all OpenMPI 4.1.x easyconfigs (PR #19940)
    - stop using non-existent --disable-libdeflate option for LibTIFF 4.1.0 (PR #19951)
    - add patch to remove git version check and fix test command for molmod 1.4.8 (PR #19952)
    - stop using non-existing --with-doc configure option for groff 1.23 (PR #19969)
    - add patch to fix tree-optimization bug in GCC 13.2.0 for ARM architectures with SVE support (PR #19974)
  - enhancements
    - ...
  - (noteworthy) new software
    - ...
  - noteworthy software updates
    - PyTorch v2.1.2 w/ foss/2023a + CUDA 12.1.1 (PR #19666)
      - max_failed_tests sets to 50
      - common causes of failures in PyTorch test suite
        
        running with random seed
        
        use of hypothesis by generating random inputs for tests
        
        actual bugs in compiler, OpenMPI, OpenBLAS, ...
      - can we reach out to PyTorch team to give feedback on this?
      - ideally there would be a subset of tests that should definitely never fail, more reliable way of checking the installation
      - follow-up PRs that depends on PyTorch, should introduce PyTorch-bundle first (see easyconfigs PR #19987)
        
        Alex: older ignite version required for MONAI, can still depend on PyTorch-bundle + add older ignite version as extension to MONAI easyconfig
  - changes
    - ...
  - EasyBuild 5.0 (to 5.0.x branch)
    - add download_instructions to various easyconfigs (~40 PRs)
    - detect easyconfigs where missing information about downloading sources (PR #19881)
- docs (open PRs + issues)
  - project board to perform yearly review cycle of all documentation pages: https://github.com/orgs/easybuilders/projects/17/views/1
- framework (open PRs + issues)
  - reported bugs / bug fixes
    - Run unit tests on an updated version of Modules 4 (PR #4415)
  - enhancements
    - add terse support to --missing-modules (PR #4407)
    - add new 'authors' field to detailed JSON software listing (PR #4460)
  - changes
    - ...
  - EasyBuild 5.0 (to 5.0.x branch)
    - eb command fails outside of framework source folder (issue #4451)
    - active PRs
      - change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
        
        Alex has some idea on how to ensure that tarball created for sources obtained via git_config are identical across Linux & macOS
      - rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400)
        
        problems fixed by Alex, ready for review/merge
      - implement support for running interactive commands with run_shell_cmd (WIP PR #4453)
        
        implementation is slowly progressing...
      - refactor easyconfig parameters and template constants to use a consistent naming style (issue #4464)
    - dormant issues/PRs (for now)
      - deprecate failure to resolve a template value (PR #3285)
      - drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
        
        we should also enable depends_on by default when using Lmod as modules tool in EasyBuild 5.0
      - replace parallel easyconfig parameter by maxparallel (PR #4398)
      - bump minimum required Tmod (4.x) to 4.3.0 (PR #4425)
        
        requires that PR #4415 is merged first
      - deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP) (PR #4433)
        
        requires that implementation of run_shell_cmd is complete, incl. support for asynchronously running commands and running interactive shell commands (WIP)
      - support uninstalling software + module (issue #590)
      - clean up SLURM/Batchsystem environment before doing builds (issue #4434)
        
        unset all $SLURM_* environment variables to avoid trouble with srun/MPI
        
        should be done opt-in (--clean-slurm-env configuration option)
- easyblocks (open PRs + issues)
  - bug reports/fixes
    - remove trailing slash in SLEPC_DIR (PR #3229)
    - fix OpenFOAM easyblock to correctly have is_dot_org as True when v >=11 and fix motorBike example for v >=11 (PR #3232)
  - enhancements
    - Support GROMACS builds with SYCL (AdaptiveCpp implementation) (PR #3077)
    - use proxy in sanity check of EB_OpenSSL_wrapper (PR #2922)
    - Score-P easyblock: Select the oneAPI compiler suite when enabled in the Intel compiler toolchain (PR #3228)
    - enhance CMakeMake easyblock to check whether correct Python installation was picked up by cmake (PR #3233)
    - Make OpenBLAS easyblock aware to --optarch=GENERIC (issue #3230)
  - updates
    - ...
  - new easyblocks
    - ...
  - changes
    - Always set $EBPYTHONPREFIXES instead of $PYTHONPATH (issue #2887)
  - EasyBuild 5.0 (to 5.0.x branch)
    - fix extension filter for Perl packages (PR #2699)
    - update ConfigureMake easyblock to error out on unknown configure args (PR #3025 + #3026)
    - rename run method to install_extension, and likewise for prerun to pre_install_extension, postrun to post_install_extension, and run_async to install_extension_async (PR #3064)
      - companion for framework PR #4400
    - set CMake installation LIBDIR to lib by default in CMakeMake easyblock (PR #3227)
    - ~15 open PRs to replace run_cmd with run_shell_cmd in custom easyblocks
      - see also overview in issue #3089
      - all help to migrate easyblocks to run_shell_cmd is welcome!
    - ~5 open PRs to remove unused custom easyblocks
- easyconfigs (open PRs + issues)
  - bug fixes/reports
    - ...
  - enhancements
    - ...
  - new software
    - ...
  - noteworthy software updates
    - Qt5 v5.15.11 w/ GCCcore/13.2.0 (for foss/2023b) (WIP PR #19320)
  - changes
    - ...
  - EasyBuild 5.0 (to 5.0.x branch)
    - replace parallel by maxparallel (PR #19375)
    - detect easyconfigs where missing information about downloading sources (PR #19881)
  - closed PRs
    - ...

Q&A / others

EasyBuild User Meeting 2024 - agenda
- https://easybuild.io/eum
- suggestions for talk welcome!
- draft agenda is being puzzled together behind the scenes
- suggestions/ideas for talks still welcome!
We may be a bit too aggressive with compiler version w.r.t. being able to combine it with latest CUDA/NVCC
- CUDA 12.3.0 is not compatible with GCC 13.2
- cfr. easyconfigs PR #19799
Jörg: some software requires citing the use of it in paper
- can we try honoring this
- for example if there's a CITATION file in the source tarballs