Conference call notes 20240131 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 239th EasyBuild conference call, Wednesday 31 Jan 2024 (09:00 UTC)

Attendees

List of attendees (14):

• Alex Domingo (Vrije Universiteit Brussel, Belgium)
• Jasper Grimm (University of York, UK)
• Kenneth Hoste (HPC-UGent, Belgium)
• Adam Huffman (Big Data Institute, Oxford, UK)
• Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
• Sam Moors (Vrije Universiteit Brussel, Belgium)
• Sébastien Moretti (SIB, Switzerland)
• Gizo Nanava (Leibniz Universit ät Hannover, Germany)
• Mikael Öhman (Chalmers University of Technology, Sweden)
• Jure Pečar (EMBL, Heidelberg, Germany)
• Åke Sandgren (Umeå University, Sweden)
• Jörg Saßmannshausen (Imperial College London, UK)
• Alain van Hoof (TU Eindhoven, Netherlands)
• Lars Viklund (Umeå University, Sweden)

Agenda

• overview of recent developments
• quick status update on development for EasyBuild 5.0
• Q&A

Recent developments

• latest EasyBuild release: 4.9.0 (30 Dec 2023)
  • see release notes + highlights
• next EasyBuild release: hopefully EasyBuild v5.0.0
  • there will only be an EasyBuild v4.9.1 if there's a clear need for it, or if getting EasyBuild 5.0 is taking too long (way beyond 8 weeks after v4.9.0 release)
• ETA for EasyBuild 5.0 release: hopefully first EasyBuild release of 2024
  • started doing short sprint meetings, each Monday at 15:00 CE(S)T to set next 5 goals to tackle that week
  • monthly sync meetings every 1st Wed of the month at 14:00 CE(S)T: next today Wed 7 Feb'24 at 14:00 CET
  • see also https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
• recent changes
  • docs (merged PRs)
    • document how to transition from run_cmd to run_shell_cmd (PR #238)
      • see https://docs.easybuild.io/easybuild-v5/run_shell_cmd/#transitioning-from-run_cmd-to-run_shell_cmd
  • framework (merged PRs)
    • bug fixes
      • ...
    • enhancements
      • add support for appending to path environment variables via modextrapaths_append (and add corresponding allow_append_abs_path) (PR #4436)
      • improve output produced by --check-github (PR #4437)
      • show error when multiple PR options are passed (PR #4440)
    • changes
      • ...
    • code cleanup
      • ...
    • EasyBuild 5.0 (to 5.0.x branch)
      • bug fixes
        • don't hardcode /bin/bash in eb script, RPATH wrapper script, and run_shell_cmd (PR #4441)
        • fix error reporting when test step fails (PR #4443)
      • changes
        • deprecate EnvironmentModulesC and EnvironmentModulesTcl (PR #4439)
      • enhancements
        • ...
  • easyblocks (merged PRs)
    • bug fixes
      • pick Tcl/Tk lib directory correctly in R easyblock, may be either lib or lib64 (PR #3072)
      • use (=|:=) in regex because OpenFOAM rule files use := instead of = since v2212 (PR #3073)
      • improve generate_crate_list + parsing of crates in Cargo easyblock (PR #3084)
      • remove trailing slash in PETSC_DIR set by custom easyblock for PETSc (PR #3086)
      • apply CMake policy to select python based on $PATH order, instead of highest Python version (PR #3088)
      • use correct TCL configure options for Python 3.11+ (PR #3087)
    • enhancements
      • also support OpenSSL 3 in OpenSSL wrapper easyblock (PR #3081)
    • updates
      • ...
    • changes
      • ...
    • new easyblocks
      • ...
    • EasyBuild 5.0 (to 5.0.x branch)
      • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (PR #3022)
      • disable use_pip by default for PyTorch, except for recent versions (>= 2.0) (PR #3079)
      • don't change installopts easyconfig parameter value in-place in PythonPackage easyblock (PR #3080)
      • remove unused easyblocks (ARB, ATLAS, BLACS, CHARMM, cppcheck, Doxygen, ESPResSo) (PRs #3096, #3103, #3104, #3108, #3111, #3114, #3117)
      • replace run_cmd with run_shell_cmd in custom easyblock for ADF, Amber, Anaconda, ant, ANSYS, Bazel, binutils, Boost, bzip2, Chimera, COMSOL, cryptography, ScaLAPACK, wxPython, Xmipp (PRs #3090, #3091, #3092, #3093, #3094, #3098, #3099, #3100, #3101, #3102, #3105, #3106, #3107, #3109, #3110)
        • see also overview in issue #3089
    • drop support for versions < 2.0 and replace run_cmd with run_shell_cmd in custom easyblock for ScaLAPACK (PR #3106)
  • easyconfigs (merged PRs)
    • ~over 100 easyconfig PRs were merged since last conf call
    • bug fixes
      • add missing KronaTools dep to q2-krona (PR #19633)
      • add missing dm-tree dependency for dm-reverb 0.2.0 (PR #19653)
      • add libyaml import check to PyYAML and fix builds since split to minimal Python (PR #19662)
      • demote meson-python to a build dependency in recent matplotlib easyconfigs (PR #19670)
      • demote poetry to build dependency in easyconfig for expecttest v0.1.5 (PR #19675)
      • fix missing spaces in Graphviz configure options (PR #19687)
      • use separate bcrypt easyconfig (using CargoPythonPackage) as dependency for JupyterHub v4.0.2 (PR #19703)
      • use maturin easyconfig using CargoPythonPackage as a dependency for fastparquet v2023.4.0 (PR #19704)
      • add patches for Perl-bundle-CPAN v5.36.1 to fix installation of Sys::Info::Driver::Linux* extensions on Debian Bookworm (PR #19727)
      • explicitly specify Fortran compiler for recent CDO versions to fix issues on non-x86_64 architectures (PR #19735)
      • add Python build dependency to Doxygen v1.9.x (PR #19743)
    • enhancements
      • add tidybayes + spdep to R-bundle-CRAN v2023.12 (PRs #19712 + #19729)
    • (noteworthy) new software
      • ...
    • noteworthy software updates
      • SciPy-bundle v2023.07 w/ iimkl/2023a (PR #18875)
        • problems with failing tests dealt with by:
          • adding patches to disable selected tests
          • enabling toolchain options lowopt (-O1) + strict (stricter precision-related compiler flags)
        • see overview of test platforms at https://github.com/easybuilders/easybuild-easyconfigs/pull/18875#issuecomment-1912508980
    • changes
      • stop using custom easyblock for Doxygen (PR #19742)
    • EasyBuild 5.0 (to 5.0.x branch)
      • archive EasyBuild 3.x easyconfigs (PR #19656)
      • move setuptools_scm from hatchling to Python easyconfig (PR #19651)
      • replace usage of easybuild.tools.py2vs3 in easyconfigs test suite (PR #19744)
• work-in-progress
  • docs (open PRs + issues)
    • project board to perform yearly review cycle of all documentation pages: https://github.com/orgs/easybuilders/projects/17/views/1
  • framework (open PRs + issues)
    • reported bugs / bug fixes
      • Run unit tests on an updated version of Modules 4 (PR #4415)
      • treat colon as special path in modextrapath (WIP PR #4442)
    • enhancements
      • ...
    • changes
      • drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
      • use more performant and concise dict construction (PR #4410)
      • clean up & speed up environment checks (PR #4409)
    • EasyBuild 5.0 (to 5.0.x branch)
      • deprecate failure to resolve a template value (PR #3285)
      • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
      • replace parallel easyconfig parameter by maxparallel (PR #4398)
      • rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400)
      • bump minimum required Tmod (4.x) to 4.3.0 (PR #4425)
        • requires that PR #4415 is merged first
      • deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP) (PR #4433)
        • requires that implementation of run_shell_cmd is complete, incl. support for asynchronously running commands and running interactive shell commands (WIP)
      • create lib/lib64 symlink before running postinstallcmds (PR #4435)
      • support uninstalling software + module (issue #590)
      • clean up SLURM/Batchsystem environment before doing builds (issue #4434)
        • unset all $SLURM_* environment variables to avoid trouble with srun/MPI
      • add support for running shell commands asynchronously with run_shell_cmd (WIP) (PR #4444)
  • easyblocks (open PRs + issues)
    • bug reports/fixes
      • fix extension filter for Perl packages (PR #2699)
        • easyconfigs PR #18789 is a big step towards merging this, but there are other easyconfigs that needs to be tested first, like XML-LibXML, worker, GD, ...
      • update PyTorch easyblock to avoid configure warnings by disabling some options (PR #3070)
    • enhancements
      • update KOKKOS_CPU_MAPPING for AArch64 in LAMMPS (PR #3036)
      • refactor GAMESS-US easyblock to directly write install.info (v2) (PR #3047)
        • currently only parallel tests are being run
        • we should run the sequential tests, see GAMESS_MPI_TEST_BLACKLIST
      • Support GROMACS builds with SYCL (AdaptiveCpp implementation) (PR #3077)
      • 4 Conda Fixes to allow Conda.py to work with Micromamba (PR #3082)
      • Improve error checking in PyTorch easyblock (PR #3085)
    • updates
      • update Clang-AOMP easyblock for ROCm 5.6 (PR #2958)
        • we maybe should look into merging the Clang and Clang-AOMP easyblocks
    • new easyblocks
      • ...
    • changes
      • ...
    • EasyBuild 5.0 (to 5.0.x branch)
      • update ConfigureMake easyblock to error out on unknown configure args (PR #3025 + #3026)
      • rename run method to install_extension, and likewise for prerun to pre_install_extension, postrun to post_install_extension, and run_async to install_extension_async (PR #3064)
      • replace run_cmd with run_shell_cmd in custom easyblock for CP2K, CUDAcompat, ESMF (PRs #3112, #3113, #3115)
        • see also overview in issue #3089
        • all help to migrate easyblocks to run_shell_cmd is welcome!
  • easyconfigs (open PRs + issues)
    • bug fixes/reports
      • fix JULIA_DEPOT_PATH for Julia (PR #19650)
        • Alex: maybe alternative can be to play with JULIA_LOAD_PATH instead?
    • enhancements
      • ...
    • new software
      • ...
    • noteworthy software updates
      • Qt5 v5.15.11 w/ GCCcore/13.2.0 (for foss/2023b) (WIP PR #19320)
      • PyTorch v2.1.2 w/ foss/2023a + CUDA 12.1.1 (PR #19666)
        • test reports look pretty good, only a handful of failing tests on a bunch of GPUs
    • changes
      • ...
    • EasyBuild 5.0 (to 5.0.x branch)
      • clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail (PR #19265)
      • replace parallel by maxparallel (PR #19375)
    • closed PRs
      • ...

Q&A / others

• EasyBuild User Meeting 2024 - agenda
  • https://easybuild.io/eum
  • suggestions for talk welcome!
  • ideas:
    • deep-dive into EasyBuild 5.0
• Jure: hitting PATH_MAX for $LD_LIBRARY_PATH
  • hitting this with R-bundle-Bioconductor on Rocky 8.x
    • loads ~125 modules...
  • value of $LD_LIBRARY_PATH seems to ignored at 4096 characters?
  • @Jure: please open issue with instructions on how to reproduce
  • fixes/workarounds:
    • shorter install paths (via symlinks, if needed)
    • use RPATH (because then $LD_LIBRARY_PATH is only used as a fallback)
    • symlink farm (maybe even as needed, via Lmod hook?)
  • paths that don't have libraries should not be added to $LD_LIBRARY_PATH
    • see also https://github.com/easybuilders/easybuild-framework/issues/3504
• Jörg is looking into warpX
  • https://github.com/ECP-WarpX/WarpX
  • test jobs require additional Python packages
  • need to inject some subdirectories on $PYTHONPATH to run examples