Conference call notes 20201028 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 159th EasyBuild conference call, Wednesday October 28th 2020 (09:00 UTC - 10:00 CET)

Attendees

Alphabetical list of attendees (6):

  • Simon Branford (Univ. of Birmingham, UK)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Jörg Saßmannshausen (NIHR Biomedical Research Centre, UK)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Alan O'Cais (Jülich Supercomputing Centre, Germany)
  • Mikael Öhman (Chalmers University of Technology, Sweden)

Agenda

  • recent developments
  • updates on 2020b version of common toolchains (foss, intel)
  • Q&A

Recent developments

  • ETA for EasyBuild v4.3.1 release: tomorrow (Oct 29th 2020)
    • 4.3.x branches are ready to make the release
    • PRs that are merged in develop branches from this point forward will not be included in EaysBuild v4.3.1 release!
  • recent changes
    • framework
      • bug fixes
        • avoid confusing error log message when determining impi version while trying to define value for %(mpi_cmd_prefix)s template (PR #3474)
        • unset $LD_LIBRARY_PATH when checking for OS dependencies with 'rpm' & co (PR #3477)
          • fixes failing OS dependency check because rpm command is broken when XZ module is loaded as dependency...
        • take source_urls, checksums, patches into account when extension source is specified via 'sources' ((PR #3487)
        • don't change directory in download_repo function in tools.github (PR #3486)
      • enhancements
        • check for _VERSION and _PREFIX Cray environment variables with both software and module name (PR #3460)
        • allow including easyblocks from multiple PRs ((PR #3480 + PR #3481)
    • easyblocks
      • bug fixes
        • respect --disable-mpi-tests in Siesta easyblock (PR #2207)
      • enhancements
        • add easyblock for CFDEMcoupling (PR #1439)
        • custom easyblock for NVHPC (PR #2190)
        • add support to RubyGem easyblock for installing zipped gems (PR #2203)
        • update ScaLAPACK easyblock to support installation with CMake for recent versions (>= v2.1.0) (PR #2205)
        • only enable CUDA in LAMMPS if it is an explicit dep + add support for building for NVIDIA Ampere GPUs (PR #2208)
        • update Score-P easyblock to add support for NVHPC toolchain (PR #2206
    • easyconfigs
      • ~100 merged easyconfig PRs in last 2 weeks!
      • bug fixes
        • add patch for OpenMPI 3.1.4 (foss/2019b) adding device parameters for ConnectX-6 (PR #11575)
      • enhancements
        • update Java/1.8 wrapper to also support aarch64 (PR #11545)
      • new software
      • software updates
      • changes
  • to merge soon (won't be included in EasyBuild v4.3.1 anymore!)
    • framework (v4.3.2 milestone)
      • bug fixes
      • enhancements
        • enhance apply_regex_substitutions to allow specifying action to take in case there are no matches (PR #3440)
        • add support for using customized HTTP headers in download_file (PR #3472)
    • easyblocks (v4.3.2 milestone)
      • enhancements
        • enhance OpenBLAS easyblock to make it aware of optarch (PR #1946)
        • improve config checks for Boost.MPI and Boost.Python and fix sanity checks of MT libs (PR #2167)
          • didn't get merged for EasBuild v4.3.1 because it broke the installation of some existing Boost easyconfigs...
        • run motorBike tutorial case for recent (community) OpenFOAM versions (PR #2201)
    • easyconfigs (v4.3.2 milestone)
      • bug fixes
        • added missing space in configopts in ParaView 5.8.0 easyconfigs using 2020a toolchain (PR #10989)
          • missing space breaks OSMESA functionality (so it's basically GLX)
          • Mikael: we need to build 3 different versions of ParaView (3 different backends)
            • OSMESA (no GPU + off-screen rendering) => not compatible with other stuff
            • EGL (off-screen GPU rendering, in theory supports everything)
              • this is the glorious future when X11 is dead and everybody uses Wayland
              • depends on libglvnd
            • GLX (interactive use via VNC, works with VirtuaGL)
            • which one do we use as a dependency for OpenFOAM?!
              • ComputeCanada uses GLX build as dep for OpenFOAM
            • also see issue #10977
            • need to figure out proper versionsuffix
        • use bfd linker for glibc (PR #11331)
        • fix LAMMPS 3Mar2020 easyconfigs using intel toolchain on AMD CPUs by patching out hardcoded -xHost (PR #11577)
      • enhancements
      • new software
      • software updates

2020b update of common toolchains

  • status:
    • candidate toolchains definitions are now ready
    • needs some more testing before setting them in stone
    • 2020b toolchains will not be included in EasyBuild v4.3.1
    • aiming for EasyBuild v4.3.2 release in a couple of weeks that includes foss/2020b and intel/2020b
      • and maybe also fosscuda/2020b?
  • base: GCC 10.2 + binutils 2.35
  • foss/2020b (PR #11489):
  • intel/2020b (PR #11337):
    • compilers and imkl have been updated from "2020 update 3" to "2020 update 4" release since last conf call
    • compilers: 2020 update 4 (Oct 22st 2020)
    • MPI: 2019 update 8 (Jul 15th 2020)
    • MKL: 2020 update 4 (Oct 21st 2020)
    • is GCC 10.x supported officially as base compiler?

      • reply from Kenneth's inside Intel contact:

        GCC 10.2:
Since gcc 10.2 released on July 23, no past releases of 19.1 (like PSXE 2020 update 2) have been tested with it. Upcoming PSXE 2020 update 4 (compiler 19.1.3) is tested with gcc 10.2.  For 2021.1 Beta 09 was tested with gcc 10.2.
We haven't seen major issues for gcc 10.2 vs 10.1 so it would be safe to assume that PSXE 2020 update 2 will work (it was tested against gcc 10.1).

        binutils 2.35:
We haven’t tested any IL0 compiler against binutils 2.35 (released on July 24). We only do testing with binutils that come with OS we      support. The most recent binutils we tested are 2.34 coming with Ubuntu 20.04.
So we cannot claim the compiler will work fine with binutils 2.35.
  • various PRs have been merged adding easyconfigs with GCCcore/10.2.0 as toolchain
  • we're dropping the versionsuffix when using Python 3 as a dep (retain with Python 2)
  • TODO:
    • testing large applications on top of these candidate toolchain definitions! (R, OpenFOAM, GROMACS, ...)

Easyconfigs merge sprint

  • merge sprint was held on Fri Oct 16th 2020
  • goal was to close or merge oldest 100 PRs
  • see https://hackmd.io/ZPMh4H-3SpuXWsHvNMmyNw
  • end result:
    • merged ~35 old PRs
    • closed ~25 old PRs
    • plus more cleanup of old PRs after the sprint
    • significant reduction in total number of open PRs (~400 => ~330)
  • should we plan another sprint like this?

Ideas for next EasyBuild User Meeting

Q&A

  • Mikael: collapse foss and fosscuda?
    • getting rid of unnecessary duplicates (but there are not a lot of them)
    • compiler + BLAS/LAPACK subtoolchains could also help (also beyond foss & fosscuda)
      • effectively moves subtoolchains to diamond DAG
      • could help a lot with moving SciPy-bundle to a non-MPI subtoolchain
    • probably too late for 2020b, start discussion with 2021a in mind?
    • open issue with overview of what could be cleaned up if we make this effort
    • good topic to discuss in EasyBuild User Meeting?