Conference call notes 20200513 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 147th EasyBuild conference call, Wednesday May 13th 2020 (8:00 UTC - 10:00 CEST)

Attendees

Alphabetical list of attendees (9):

• Kenneth Hoste (HPC-UGent, Belgium)
• Åke Sandgren (Umeå University, Sweden)
• Mikael Öhman (Chalmers University of Technology, Sweden)
• Henrik Nagel (NTNU, Norway)
• Alexandre Strube (JSC, Germany)
• Alan O'Cais (JSC, Germany)
• Lars Viklund (Umeå University, Sweden)
• Thekla Loizou (Cyprus Institute)
• Adam Huffman (Big Data Institute, Oxford, UK)

Agenda

• overview of recent changes to be included with upcoming EasyBuild v4.2.1
• update on progress on 2020a generation of easyconfigs
• Q&A

Overview of recent changes

• project in GitHub with overview of things to be done for EasyBuild v4.2.1
  • maintainers: if any issues/PRs pop up that really must be included in next release, assign them to the 4.2.1 project!
• ETA for EasyBuild v4.2.1:
  • get everything ready by the end of this week
  • regression test over the weekend
  • release early next week (Mon?)

framework: merged PRs in 4.2.1 milestone

• recently merged PRs:
  • automatic changing of directory by extract_file is deprecated (PR #3292)
    • also fixed in easyblocks (PR #2051)
  • proper fix for PyPI download problem, parse page served by PyPI's /simple API as HTML (not XML) (PR #3329)
    • it looks like the cause of the problems in EasyBuild will be fixed on the PyPI side too (see here)
  • cleaning up locks on signals (PR #3291)
    • this is not enough to auto-cleanup locks for EasyBuild sessions run in Slurm jobs that get killed though... (issue #3280)
    • requires looking into fcntl style looks; looks doable, maybe even should be the default locking mechanism?
  • stop using distutils.dir_util in copy_dir (PR #3310)
  • make ModulesTool.exist more robust w.r.t. module wrappers & co (PR #3216)
    • this introduced a regression (issue #3335)
  • fix problems with parsing easyconfigs using a Cray toolchain when there are no actual external modules (PR #3319)
    • this fixed --list-installed-software crash when using HMNS
  • add constants for common OS dependencies (PR #3309, PR #3334)
    • OS_PKG_IBVERBS_DEV, OS_PKG_OPENSSL_DEV, OS_PKG_PAM_DEV, ...
    • Åke: OS_PKG_IBVERBS_DEV should be renamed?
  • bug fixes for --try-update-deps experimental feature (PR #3325), (PR #3326), (PR #3330),
  • cleaned up output for --list-toolchains --output-format=rst (PR #3246)
    • see result in docs (develop only for now)
  • add --disable-job in 'eb' command used in jobs, to prevent infinite job cycle (PR #3328)
  • avoid empty entries in $LD_LIBRARY_PATH and other path-like environment variables (PR #3333)
• open PRs/issues that should be merged/fixed for EasyBuild 4.2.1 release:
  • bugs to fix:
    • Java/11 module wrapper is not seen as installed when using HierarchicalMNS (issue #3335, introduced by PR #3216)
  • bugfix PRs to review/merge:
    • expand glob paths when checking for multiple inclusion of easyblocks (bugfix for --include-easyblocks*) (PR #3311)
  • new features:
    • allow copying of tweaked easyconfigs when using --try-* (PR #3332)

easyblocks: merged PRs in 4.2.1 milestone

• recently merged PRs:
  • fixes/enhancements for PETSc easyblock (PR #2048)
  • use config.guess for GCC builds (PR #2033)
  • impi: execute minimal test in sanity check commands (PR #2045)
  • fix for ROOT easyblock (PR #2047)
  • include own lib dir in RPATH before system lib dirs in binutils easyblock (PR #2044)
• open PRs/issues that should be merged/fixed for EasyBuild 4.2.1 release:
  • correct download url for test data in WPS easyblock (PR #2055)
  • use updated config.guess for installing R packages (PR #1949)
  • fix for CMakeMake easyblock w.r.t. switch to building outside source tree (PR#2032)
  • rewrite GROMACS easyblock to build all four variations in the same build (PR #1991)
  • generically find the arch folder and add top-level bin/lib symlinks in PDT & TAU easyblocks (PR #2029)
  • new easyblock to install XALT (PR #1942)
  • fix problem with forced reinstallation of ROOT (issue #2023)

easyconfigs merged PRs in 4.2.1 milestone

• recently merged PRs:
  • patch for h5py with 2020a toolchain to avoid workaround for MPI_Init at import h5py (PR #10626, original workaround in PR #10246)
    • will need more work, current approach of disabling auto-init by mpi4py isn't working out well
    • see also https://github.com/h5py/h5py/pull/1544
    • Alan: god way forward could be to roll back patch for h5py with foss/2020a and pull out h5py with intel/2020a from develop, to avoid blocking 4.2.1 release over this
  • fix for R 3.6.2 + R 3.6.3 on POWER (PR #9830, PR #10531)
• open issues/PRs:
  • fix Mako easyconfigs using a 2016 & 2017 toolchain (PR #10627)
  • TensorFlow 2.2.0 w/ fosscuda/2019b (PR #10600)
    • Mikael: discussion with Jack Perdue in this PR w.r.t. handling of CUDA compute capabilities
    • silly typo to fix in TensorFlow easyblock: "may not optional" -> "may not be optimal"
  • updated/more extensions for R 4.0.0 & Bioconductor 3.11 (PR #10614, PR #10621)
  • bugfix for Arlequin (PR #10620)

Update on 2020a common toolchains

• gradually collapsing easyconfigs prepared in 2020a branch into develop
• TODO:
  • LAMMPS (blocked byMPI_Init issues with h5py) (PR #10191, PR #10537)
  • GROMACS (PR #10026)
  • clean up h5py with foss/2020a (no patch) + remove h5py with intel/2020a to avoid blocking release
• CUDA compatible with GCC 9.3.0 (for fosscuda/2020a) is coming soon?

Q&A

• Henrik: Java/11 wrapper doesn't seem to work for us in HMNS context
  • could be an issue with old Lmod version not picking up .modulerc.lua?
    • Lmod 7.8
  • fix could be to install Java/11 wrapper with Tcl module syntax (because older Lmod versions don't pick up on .modulerc.lua, only .modulerc)
• Alexandre: issues with updating PyTorch to 1.5.0 with fosscuda toolchain, CMake build dep is not being found?
  • easier to discuss in Slack, CMake using GCCcore easyconfig should be picked up when using fosscuda
• Alexandre: starting point for implementing --new-draft-pr?
  • see easybuild/tools/github.py
  • add option to new_pr_from_branch function, act in it after figuring out magic incantation via GitHub PR, expose it via new --new-draft-pr CLI option