Conference call notes 20210818 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 179th EasyBuild conference call, Wednesday Aug 18th 2021 (15:00 UTC)

Attendees

Alphabetical list of attendees (13):

  • Sebastian Achilles (Jülich Supercomputing Centre, Germany)
  • Simon Branford (Univ. of Birmingham, UK)
  • John Dey (Fred Hutch, Seattle, US)
  • Alex Domingo (VUB, Belgium)
  • Fotis Georgatos (SDSC, Switzerland)
  • Jose Sergio Hleap (SHARCNET, Canada)
  • Victor Holanda Rusu (CSCS, Switzerland)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Mikael Öhman (Chalmers University of Technology, Sweden)
  • Åke Sandgren (Umeå University, Sweden)
  • Jörg Saßmannshausen (NIHR Biomedical Research Centre, UK)
  • Davide Vanzo (Microsoft Azure)

Agenda

  • overview of recent developments
  • update on support for installing AlphaFold
  • 2021b common toolchains
  • Q&A

Recent developments

  • release timeline
    • last release: EasyBuild v4.4.1 (July 6th)
    • ETA next release: by end of August (?)
  • recent changes
    • framework
      • bug fixes
        • fix verify_imports by deleting all imported modules before re-importing them one by one (PR #3780)
          • fixes bug when including multiple easyblocks and running eb on top of Python 2
        • don't add directory that doesn't include any files to $PATH or $LD_LIBRARY_PATH (PR #3769)
        • don't parse patch files as easyconfigs when searching for where patch file is used (PR #3786)
      • enhancements
        • also define $BLAS_SHARED_LIBS & co in build environment (analogous to $BLAS_STATIC_LIBS) (PR #3800)
          • required for fixing installing of GROMACS with foss/2021a (due to FlexiBLAS), see easyblocks issue #2521
          • Victor: what about Cray toolchains?
        • report use of --ignore-test-failure in success message in output (PR #3806)
        • add --filter-ecs options to filter out easyconfigs from set of easyconfigs to install (PR #3796)
      • changes
        • create libtinfo symlinks in easyconfigs for ncurses 6.2 (PR #13660 + #13661)
    • easyblocks
      • bug fixes
        • set $I_MPI_ROOT correctly for 2021.x in impi easyblock (PR #2537)
          • only requires eb --module-only to fix installed impi/2021.* modules
      • enhancements
      • new easyblocks
      • changes
        • (none)
    • easyconfigs
      • ~150 easyconfig PRs merged since last conf call!
      • bug fixes
        • add upstream patch for GCC 9.x, 10.x, 11.x to avoid spurious FPE on avx512 (affects UCX) (PR #13628)
        • enable EGL in recent libepoxy easyconfigs (PR #13684)
      • enhancements
        • (nothing major)
      • new software
        • easyconfigs for over 20 new software tools, incl. jax, DistributedStream, CellRank, GOATOOLS, ...
      • noteworthy software updates
        • ~100 PRs with software updates...
        • lots of easyconfigs with updated software versions using GCCcore/11.2.0 toolchain (prep for 2021b generation of easyconfigs)
  • to merge/fix/tackle soon
    • framework
      • reported bugs / bug fixes
        • make logdir writable also when --stop/--fetch is given (PR #3771)
        • add optimal optimization flags for Intel compilers on AMD CPUs (PR #3793)
          • -xHost results in only using SSE2 on AMD systems...
        • make sure git clone with a tag argument actually downloads a tag (PR #3795)
        • correctly resolve templates for patches in extensions when uploading to GitHub (PR #3805)
      • enhancements
        • avoid using a priority in prepend_module_path (Lmod) to avoid costly module calls (PR #3636)
        • add support for installing extensions in parallel (WIP) (PR #3667)
          • needs more test + a dedicated unit test
          • should be marked experimental at first?
        • finding modules with multiple modulepaths and HMNS (issue #3703)
        • check for recursive symlinks by default before copying a folder (PR #3784)
        • add easybuild.tools.LooseVersion (PR #3794)
          • required because distutils which provides LooseVersion will be removed in Python 3.10+ ...
        • allow optarch values to be partial maps including vector extensions (PR #3797)
      • changes
        • fully drop support code for Python < 2.7 (PR #3788)
        • test suite improvements (PR #3790)
        • performance improvements of tests (PR #3802)
    • easyblocks
      • reported bugs / bug fixes
        • explicitly use only OpenBLAS for PyTorch if MKL is not used (PR #2448)
        • clean up installation of Tkinter (PR #2509)
        • reset runtest in CMakePythonPackage PR #2536
        • fix motorBike test in sanity check for OpenFOAM 9 (PR 2544#)
      • enhancements
        • enhance test and install step of CMakePythonPackage easyblock (PR #2318)
        • add support for installing R extensions in parallel (WIP) (PR #2408)
        • enable make check and sanity check exec for OpenMPI (PR #2444)
        • OpenSSL-1.1.eb does not install corresponding pkg-config file (PR #2525)
      • changes
        • (nothing major)
      • new software
    • easyconfigs

Common toolchains

2021b (WIP!)

  • base: GCCcore/11.2.0
    • what about compatibility with Intel compilers?
  • merged easyconfigs PRs with GCCcore/11.2.0:
  • easyconfig for OpenMPI 4.1.1 with GCCcore/11.2.0 (#13668)
    • same OpenMPI as in foss/2021a
    • should we wait until OpenMPI 4.1.2 is released?
    • is OpenMPI 4.1.1 actually compatible with PMIx 4.x?
    • foss/2021b is important for GPU software (GCC 11.2 is officially supported in latest CUDA)
  • PR for intel/2021.07 (candidate for intel/2021b): #13639
    • unofficially, latest Intel versions should be compatible with GCC 11.2 ...

AlphaFold

  • WIP PR for AlphaFold easyconfig (foss/2020b): #13744

    • but AlphaFold doesn't seem well suited to be run CPU-only...
      • 15x speedup when running on P100 GPU (for the small_bfd dataset)
      • bfd vs small_bfd is 100x in data volume (~1.7TB vs 17GB)
    • easyconfig for required aria2 dependency merged via PR #13709
    • easyconfig for required jax dependency merged via PR #13622

  • custom easyblock for jaxlib (component in jax installation): https://github.com/easybuilders/easybuild-easyblocks/pull/2545

    • required because: uses Bazel as build tool, additional options/work needed when building with CUDA support enabled
    • corresponding (updated) jax easyconfigs (foss/2020b + fosscuda/2020b): https://github.com/easybuilders/easybuild-easyconfigs/pull/13760
      • still some trouble with the test suite
      • several tests fail, even when setting various magic environment variables to steer the tests
        • patch out the broken tests
      • trouble in multi-GPU context
      • severely overloading cores when running tests in parallel (even close to a fork bomb...)

Q&A

  • Mikael: no more releases of Qt 5.15.x
    • we should start looking into Qt6 soon-ish
    • latest bugfix releases of 5.15.x are not public...
  • Mikael: should we start having GTK2 and GTK3?
    • they seem to be compatible to have both active at the same time
  • Fotis: who was looking into GitLab integration for testing EasyBuild?
    • to enable use of --new-pr to a GitLab repository
    • it's possible to combine GitLab CI/CD to test GitHub repositories