Conference call notes 20241218 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 261th EasyBuild conference call, Wednesday 18 Dec 2024 (16:00 UTC / 17:00 CET)

Attendees

List of attendees (15):

  • Simon Branford (Univ. of Birmingham, UK)
  • Em Dragowsky (Case Western Reserve University, Ohio, US)
  • Thomas Hoffman (EMBL, Germany)
  • Jasper Grimm (University of York, UK)
  • Alexander Grund (ZIH, Dresden, Germany)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Georgios Kafanas (University of Luxembourg)
  • Bart Oldeman (McGill University, Digital Research Alliance of Canada)
  • Jurij Pečar (EMBL, Germany)
  • Andrea Pisercia (E4 consultancy company, Italy)
  • Jan Reuter (JSC, Germany)
  • Ãke Sandgren (Umeå University, Sweden)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Roberto Scipioni (Red Oak Consulting, UK)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024b update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: v4.9.4 (22 Sept 2024)
  • next EasyBuild release: 5.0.0beta1 :fire:
  • next (stable) EasyBuild release:
    • very likely EasyBuild v5.0.0...
    • additional EasyBuild 4.9.x versions could still be done via 4.9.x branches, but none are planned currently
    • as it stands currently, --install-latest-eb-release would install 5.0.0beta1 when easyconfigs PR #22049 gets merged
  • EasyBuild v5.0.0
    • project board: https://github.com/orgs/easybuilders/projects/18/views/2
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • [5.0.x] tweak setup.py to fall back to importing setup from setuptools if distutils is not available (PR #4478 )
      • [5.0.x] improve portability of reproducible tarballs by replacing external tar command with tarfile module from Python standard library (PR #4660)
      • [develop] fix dry-run output when using multi_deps (PR #4678)
      • [5.0.x] make sure "type module" is run during a dry run (fixes use of eb -x) (PR #4721)
    • enhancements
      • [5.0.x] add --search-path-cpp-headers configuration option to control how EasyBuild sets paths to headers at build time (PR #4645)
      • [develop] Add resolve_template method to EasyConfig class (PR #4677)
      • [develop] Allow templates in custom_paths & custom_commands sanity-check arguments (PR #4679)
      • [5.0.x] add --keep-debug-symbols configuration option to set default value of 'debug' toolchain option, and enable it by default so -g is included in $CXXFLAGS & co (PR #4688)
      • [5.0.x] add --search-path-linker option to control linker options at build time (PR #4697)
      • [5.0.x] enhance get_software_libdir to return full paths if requested (PR #4699)
    • changes
      • [5.0.x] don't allow unresolved templates in easyconfig parameters by default + add support for --allow-unresolved-templates configuration option (PR #4516)
      • [5.0.x] Rename 'source' step to 'extract' (affects skipsteps easyconfig parameter + --stop option) (PR #4629)
      • [5.0.x] Refactor make_extension_string (PR #4690)
      • [5.0.x] change Toolchain.get_flag so it doesn't automatically prepend a dash (-) to compiler flags, add deprecation warning for optarch value without leading dash, rename Compiler.COMPILER*_FLAGS to Compiler.COMPILER*_OPTIONS (PR #4698)
      • [5.0.x] Rename post_install_step to post_processing_step + deprecate use of post_install_step (PR #4715)
      • [5.0.x] run sanity checks commands from an empty temporary directory (rather than the software install directory) (PR #4723)
    • code cleanup
      • ...

  • easyblocks (merged PRs)

    • bug fixes
      • [develop] fix detection of math library in numpy build (PR #3520)
      • [5.0.x] retain custom easyconfig parameters supported by MesonNinja easyblock in custom easyblock for scipy (PR #3526)
    • enhancements
      • [5.0.x] enhance ConfigureMake easyblock to error out on unknown configure options (PR #3025)
    • updates
      • [develop] pass netCDF-Fortran path via $NETCDFF_DIR in WPS easyblock (PR #3522)
    • changes
      • [5.0.x] let PythonPackage easyblock fix python shebangs by default (PR #3499)
      • [5.0.x] update Python easyblock to move sitecustomize.py into site-packages (PR #3514)
    • new easyblocks
      • ...
    • code cleanup
      • [5.0.x] remove unused custom easyblock for TAU + clean up custom easyblock for Paraver, only support Paravar >= v4.7 (PR #3535)

  • easyconfigs (merged PRs)

    • ~XXX easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • ...
    • enhancements
      • ...
    • (noteworthy) new software
      • ...
    • noteworthy software updates
      • ...
    • cleanup
      • ...
    • changes
      • ...

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

    • bug fixes
      • [develop] Allow nesting values in checksum dicts (PR #4711)
      • [develop] fix handling of quotes in rpath wrapper (PR #4722)
    • enhancements
      • Problem using $CPATH in modulefiles overwriting system paths (issue #3331)
        • [5.0.x] refactor generation of required environment variables in module files + deprecate make_module_req_guess method in EasyBlock class #4653 (PR #4653)
          • see also (initial) companion PR for easyblocks: PR #3513
        • [5.0.x] add module-search-path-headers configuration option to control how modules set search paths to header files (PR #4655)
      • initial work towards integrating easy_update functionality (PR #4714)
    • code cleanup
      • [develop] use enumerate where applicable fixing _generate_multi_deps_list (PR #4720)
    • changes
      • [5.0.x] Deprecate use of parallel easyconfig parameter and fix updating the template value (PR #4580)

  • easyblocks (open PRs + issues)

    • bug fixes
      • [develop] let internal easyblock not create a log file in QuantumESPRESSO easyblock (PR #3505)
      • [develop] fix enhance_sanity_check for PythonPackage (PR #3477)
    • enhancements
      • [develop -> 5.0.x] enhance LLVM easyblock for compilation of clang/flang + other llvm-projects (PR #3373)
      • [develop] enhance generic Bundle easyblock to transfer module requirements of components, but do not create logfile in components (PR #3509)
      • [develop] enable hwloc and HeFFTe support in GROMACS easyblock (PR #3531)
    • updates
      • ...
    • changes
      • updates to easyblock required because make_module_req_guess is being deprecated:
        • adopt module_load_environment: Bundle (PR #3513)
        • adopt module_load_environment in PerlModule + custom easyblock for FlexiBLAS (PR #3529)
        • adopt module_load_environment: PythonPackage (PR #3530)
    • code cleanup
      • [5.0.x] Remove code supporting obsolete versions from IntelBase family of easyblocks + remove unused custom easyblock for IPP (PR #3533)
    • new
      • ...

  • easyconfigs (open PRs + issues)

    • over 1,000 open easyconfig PRs...
    • bug fixes/reports
      • ...
    • enhancements
      • ...
    • (noteworthy) new software
      • ...
    • software updates
      • [develop] PyTorch v2.3.0 (PR #20489)
        • blocked by changes in how results of PyTorch test suite are reported...
        • custom easyblock for PyTorch will need to be updated, Flamefire has started looking into it
      • [develop] AlphaFold 3 (PR #21864)
        • Jörg is involved in meeting soon with DeepMind on their parameter DB
          • only ~1GB of model files, so feasible to let users download themselves
          • public database of data (couple of TBs) they also need can be installed centrally
          • Jurij is interested in contacts related to this
        • Thomas is actively looking into it
          • jax builds (see open PR)
          • Triton works (requires specific commit of LLVM...)
          • dm-haiku (requires tensorstore), also ready (see open PR)
          • non-working easyconfig for AlphaFold 3 available (missing deps)
    • changes
      • ...

2024b 2025a common toolchains

  • GCC 14.2 as a base (see easyconfigs PR #21114)
  • we should define candidate toolchains using latest version of all components
  • to discuss: skip 2024b, jump straight from 2024a to 2025a to "catch up" on release cycle
    • EasyBuild v5.0 is ideal excuse for that break in continuity...
  • Mikael is looked into Perl (see merged PR)
  • Python 3.13.1 seems likely candidate for this toolchain...

Q&A / others

  • (Roberto) OS dependencies
    • checked as RPM
    • what about software installed in another way?
  • (Roberto) support for a feature like --continue?
  • Merry Christmas!