Conference call notes 20200916 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 156th EasyBuild conference call, Wednesday September 16th 2020 (15:00 UTC - 17:00 CEST)

Attendees

Alphabetical list of attendees (14):

• Maxime Boissonneault (ComputeCanada)
• Simon Branford (Univ. of Birmingham, UK)
• Kevin Bryan (Univ. of Rhode Island)
• Ben Carr (AstraZeneca)
• Alex Domingo (VUB, Belgium)
• Kenneth Hoste (HPC-UGent, Belgium)
• Adam Huffman (Big Data Institute, Oxford, UK)
• Terje Kvernes (University of Oslo, Norway)
• Alan O'Cais (Jülich Supercomputing Centre, Germany)
• Mikael Öhman (Chalmers University of Technology, Sweden)
• Bart Oldeman (ComputeCanada)
• Jörg Saßmannshausen (NIHR Biomedical Research Centre, UK)
• Davide Vanzo (Microsoft HPC)
• Lars Viklund (Umeå University, Sweden)

Agenda

• recent developments
• updates on fosscuda/2020a
• updates on 2020b common toolchains
• backlog in easyconfig PRs
• Q&A

Recent developments

• EasyBuild v4.3.0 was released on Sept 13th \o/
• ETA next release (probably EasyBuild v4.3.1): mid Oct?
  • project: https://github.com/orgs/easybuilders/projects/9
• recent changes
  • framework
    • (none)
  • easyblocks
    • included in EasyBuild v4.3.0:
      • custom easyblock for PyTorch (PR #2104, PR #2164)
      • enhanced easyblock for TensorFlow to provide more dependencies via EasyBuild (PR #2117, PR #2163, PR #2165)
      • update QuantumESPRESSO easyblock for v6.6 (PR #2112)
      • change easyblock to support installation on Arm: Qt(5) (PR #2160), OpenFOAM (PR #2162)
  • easyconfigs
    • fix Doxygen by promoting libiconv to runtime dependency (PR #11257)
      • Doxygen easyblock was updated to catch this issue (PR #2171)
    • Perl 5.32.0 w/ GCCcore/10.2.0 (PR #11201)
      • ground work for 2020b common toolchains
    • provide more dependencies for TensorFlow via EasyBuild (PR #11109, PR #11233)
    • update PyTorch 1.4 easyconfigs to use custom easyblock (PR #11219)
• to merge soon:
  • framework
    • gcccorecuda toolchain (PR #3385)
    • make apply_regex_substitutions produce warnings when no changes were made (PR #3440)
  • easyblocks
    • fixes for Boost easyblock, related to splitting off Boost MPI libs to separate installation (PR #2167)
    • cleanup/enhancements for TensorFlow easyblock (PR #2166)
    • enhancements to OpenBLAS easyblock to make it more aware of optarch (PR #1946)
  • easyconfigs
    • TensorFlow 2.3 w/ fosscuda/2019b (PR #11040)
    • PyTorch 1.6 w/ foss/2019b (PR #11251)

Updates on fosscuda/2020a

• blocked by gcccorecuda toolchain (framework PR #3385)?
• current status, what's left to do (and who tackles it)?
  • see https://github.com/easybuilders/easybuild-framework/pull/3385#issuecomment-676620435
  • we need a place to park UCX that is CUDA-aware
  • need to figure out which module extends $MODULEPATH with that location
  • only relevant for UCX & libfabric?
  • Bart: should we resort to using versionsuffix to make quicker progress?
• should support be added to configure EasyBuild to use CUDA as a (build) dep for UCX/OpenMPI dynamically?
  • this could allow us to get rid of foss vs fosscuda...
  • but that may be a more long term way forward?
  • Terje: would also be relevant for Arm, were we want to exclude some deps when building on Arm/aarch64 systems
    • see yasm (dep for x265) (https://github.com/easybuilders/easybuild-framework/pull/3429)
    • see libunwind (https://github.com/easybuilders/easybuild-easyblocks/pull/2153)

2020b update of common toolchains

• base: GCC 10.2 + binutils 2.35 (see PR #11056)
• foss:
  • OpenMPI 4.1.0 (RC is available, final release soon) + UCX 1.9.0 (rc6 is out now)
    • still no final 4.1.0 release yet
    • see https://github.com/open-mpi/ompi/milestone/59 for progress
    • should we stick to v4.0.5 instead? using a X.Y.0 is often not a good idea
  • OpenBLAS 0.3.10 (updated from 0.3.9 in 2020a)
    • maybe 0.3.11? see https://github.com/xianyi/OpenBLAS/milestone/30
  • FFTW 3.3.8 (no updates)
• intel:
  • compilers: 2020 update 3 (Aug 21st 2020)
  • MPI: 2019 update 8 (Jul 15th 2020)
  • MKL: 2020 update 3 (Sept 1st 2020)
  • is GCC 10.x supported officially as base compiler?
    • strictly according to release notes: no => "gcc 4.8.5 to gcc 9.0.1" and "binutils 2.27 to binutils 2.31.1", but that seems very conservative
    • Kenneth has contacted Intel guy, no reply yet...
• recent progress:
  • Perl 5.32.0 with GCCcore/10.2.0 (PR #11201)
• Python
  • no more versionsuffix when using Python 3 as a dep?

Backlog in easyconfig PRs

• close to 400 easyconfig PRs in flight currently...
• time for another merge sprint
  • focused on oldest PRs: close or get them merged
  • we should not forget to ensure a fresh CI run is done first!
• Kenneth will set a date for this (Wed in non-conf-call week)

Q&A

• Jörg: any way of detecting similar PRs to what you're working on?
  • maybe we need to add support for --new-draft-pr?
• Ben: first PR -> https://github.com/easybuilders/easybuild-easyconfigs/pull/11291
  • should look into GitHub integration: https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html