Conference call notes 20240214 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 240th EasyBuild conference call, Wednesday 14 Feb 2024 (16:00 UTC)

Attendees

List of attendees (10):

  • Sebastian Achilles (JSC, Germany)
  • Simon Branford (Univ. of Birmingham, UK)
  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Thomas Hoffman (EMBL, Heidelberg, Germany)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Sébastien Moretti (SIB, Switzerland)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Jure Pečar (EMBL, Heidelberg, Germany)
  • Lars Viklund (Umeå University, Sweden)

Agenda

  • overview of recent developments
  • quick status update on development for EasyBuild 5.0
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.0 (30 Dec 2023)
  • next EasyBuild release: hopefully EasyBuild v5.0.0
    • there will only be an EasyBuild v4.9.1 if there's a clear need for it, or if getting EasyBuild 5.0 is taking too long (way beyond 8 weeks after v4.9.0 release)
  • ETA for EasyBuild 5.0 release: hopefully first EasyBuild release of 2024
    • started doing short sprint meetings, each Monday at 15:00 CE(S)T to set next 5 goals to tackle that week
    • monthly sync meetings every 1st Wed of the month at 14:00 CE(S)T: next today Wed 7 Feb'24 at 14:00 CET
    • see also https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version
  • recent changes
    • docs (merged PRs)
      • RPATH is enabled by default in EasyBuild v5.0 (PR #241)
    • framework (merged PRs)
      • bug fixes
        • clean up log file of EasyBlock instance in check_sha256_checksums (PR #4452)
      • enhancements
        • ...
      • changes
        • ...
      • code cleanup
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • bug fixes
          • ...
        • changes
          • create lib -> lib64 symlink (or vice versa) before running postinstallcmds (PR #4435)
          • enable RPATH linking by default (PR #4448)
        • enhancements
          • ...
    • easyblocks (merged PRs)
      • bug fixes
        • update PyTorch easyblock to avoid configure warnings by disabling some options (PR #3070)
        • update Cargo easyblock to extract crates into a vendor subdir and overwrite git repo URLs with local paths (PR #3118)
        • only add --with-ucc for OpenMPI 4.1.4+ (PR #3223)
      • enhancements
        • Improve error checking in PyTorch easyblock (PR #3085)
      • updates
        • update Clang-AOMP easyblock for ROCm 5.6 + consider both $EBROOTGCC and $EBROOTGCCCORE to specify -DGCC_INSTALL_PREFIX (PR #2958)
        • enhance LAMMPS easyblock dynamically add ARMV81 and A64FX to Kokkos CPU mapping based on LAMMPS version + fix installation of Python bindings for LAMMPS >= 2Aug2023 + fix sanity check by doing MPI_Finalize (PR #3036)
        • only install Gurobi Python bindings via setup.py for Gurobi v10.x and older (PR #3225)
      • changes
        • ...
      • new easyblocks
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • bug fixes:
          • fail on non-zero exit code in command to run Perl test suite (PR #3170)
          • enable use_pip for extensions that are installed along with Python (PR #3221)
        • changes:
          • remove ~22 unused custom easyblocks (see PR overview)
          • replace run_cmd with run_shell_cmd in ~52 custom easyblocks (see PR overview)
            • see also overview in issue #3089
            • 16 easyblocks left to migrate to run_shell_cmd (see open PRs)
    • easyconfigs (merged PRs)
      • ~70 easyconfig PRs were merged since last conf call
      • bug fixes/reports
        • make sure that Brunsli libraries are installed into /lib (PR #19805)
        • avoid spurious test failure when enabling RPATH for Perl (PR #19835)
        • add missing dependencies for fslpy in easyconfig for FSL v6.0.5.1 (PR #19829)
        • 2 different versions of Compress::Zlib in Perl/5.34.1-GCCcore-11.3.0 (issue #19854)
      • enhancements
        • ...
      • (noteworthy) new software
        • ...
      • noteworthy software updates
        • ...
      • changes
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail (PR #19265)
        • replace run_cmd with run_shell_cmd in easyconfigs testsuite (PR #19818)
        • archive or delete old patches (PR #19827)
        • add check to make sure that download_dep_fail, sanity_pip_check, use_pip are not explicitly set to True in easyconfigs (PR #19830)
  • work-in-progress
    • docs (open PRs + issues)
    • framework (open PRs + issues)
      • reported bugs / bug fixes
        • Run unit tests on an updated version of Modules 4 (PR #4415)
      • enhancements
        • More granular eb exit codes (return codes) (issue #4426)
        • Download instructions should tell user exactly what files to put where (issue #4449)
      • changes
        • use more performant and concise dict construction (PR #4410)
        • clean up & speed up environment checks (PR #4409)
      • EasyBuild 5.0 (to 5.0.x branch)
        • eb command fails outside of framework source folder (issue #4451)
        • active PRs
          • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
            • Alex has some idea on how to ensure that tarball created for sources obtained via git_config are identical across Linux & macOS
          • rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400)
            • problems fixed by Alex, ready for review/merge
          • add support for running shell commands asynchronously with run_shell_cmd (PR #4444)
          • implement support for running interactive commands with run_shell_cmd (WIP PR #4453)
            • initial working implementation, but very much still WIP...
        • dormant issues/PRs (for now)
          • deprecate failure to resolve a template value (PR #3285)
          • drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
            • we should also enable depends_on by default when using Lmod as modules tool in EasyBuild 5.0
          • replace parallel easyconfig parameter by maxparallel (PR #4398)
          • bump minimum required Tmod (4.x) to 4.3.0 (PR #4425)
            • requires that PR #4415 is merged first
          • deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP) (PR #4433)
            • requires that implementation of run_shell_cmd is complete, incl. support for asynchronously running commands and running interactive shell commands (WIP)
          • support uninstalling software + module (issue #590)
          • clean up SLURM/Batchsystem environment before doing builds (issue #4434)
            • unset all $SLURM_* environment variables to avoid trouble with srun/MPI
            • should be done opt-in (--clean-slurm-env configuration option)
    • easyblocks (open PRs + issues)
      • bug reports/fixes
        • ...
      • enhancements
        • refactor GAMESS-US easyblock to directly write install.info (v2) (PR #3047)
        • Support GROMACS builds with SYCL (AdaptiveCpp implementation) (PR #3077)
        • 4 Conda Fixes to allow Conda.py to work with Micromamba (PR #3082)
      • updates
        • ...
      • new easyblocks
        • ...
      • changes
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • fix extension filter for Perl packages (PR #2699)
        • update ConfigureMake easyblock to error out on unknown configure args (PR #3025 + #3026)
        • rename run method to install_extension, and likewise for prerun to pre_install_extension, postrun to post_install_extension, and run_async to install_extension_async (PR #3064)
        • update run_async methods in RPackage, Rmpi, and Rserve easyblocks to use submit run_shell_cmd call into thread pool (PR #3163)
        • ~16 open PRs to replace run_cmd with run_shell_cmd in custom easyblocks
          • see also overview in issue #3089
          • all help to migrate easyblocks to run_shell_cmd is welcome!
        • ~5 open PRs to remove unused custom easyblocks
    • easyconfigs (open PRs + issues)
      • bug fixes/reports
        • fix JULIA_DEPOT_PATH for Julia (PR #19650)
          • Alex: maybe alternative can be to play with JULIA_LOAD_PATH instead?
      • enhancements
        • ...
      • new software
        • ...
      • noteworthy software updates
        • Qt5 v5.15.11 w/ GCCcore/13.2.0 (for foss/2023b) (WIP PR #19320)
        • PyTorch v2.1.2 w/ foss/2023a + CUDA 12.1.1 (PR #19666)
          • test reports look pretty good, only a handful of failing tests on a bunch of GPUs
      • changes
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • replace parallel by maxparallel (PR #19375)
      • closed PRs
        • ...

Q&A / others

  • EasyBuild User Meeting 2024 - agenda
    • https://easybuild.io/eum
    • suggestions for talk welcome!
    • draft agenda is being puzzled together behind the scenes
    • suggestions/ideas for talks still welcome!
  • Jörg looking into installation of Dynare
    • requires path to MATLAB installation
    • Jörg's comment: 
      # That is what I am currently doing:
preconfigopts = 'module add MATLAB/%s && export MATLABROOT=${EBROOTMATLAB} && ' % (local_mlver)
preconfigopts += 'module del MATLAB/%s && ml; ' % (local_mlver)
configopts = "-Dprefer_static=true -Dmatlab_path=${MATLABROOT} -Dbuildtype=release "
      slightly better: 
      local_mlver = '2023a_Update_3'
versionsuffix = '-MATLAB-%s' % local_mlver

      builddependencies = [('MATLAB', local_mlver)]

preconfigopts = 'export MATLABROOT=${EBROOTMATLAB} && ' % (local_mlver)
preconfigopts += 'module unload MATLAB && '
configopts = '...'
    • can the code be patched to use $EBROOTMATLAB rather than using a hardcoded path?
      • not really, because during installation time it's also building stuff on top of that MATLAB
    • Alex has done a similar workaround for other things (like MATLAB-Engine)
    • Thomas: another option could be to simply strip out the subdirectory that contains libstdc++ in MATLAB installation dir (glx...)
      • this was used to make old MATLAB version (2016) work on CentOS 8, in combination with a GCC/foss toolchain controlled by EasyBuild