Conference call notes 20210303 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 168th EasyBuild conference call, Wednesday March 3rd 2021 (16:00 UTC)

Attendees

Alphabetical list of attendees (14):

  • Sebastian Achilles (Jülich Supercomputing Centre, Germany)
  • Simon Branford (University of Birmingham, UK)
  • Fotis Georgatos (SDSC, Switzerland)
  • Victor Holanda (CSCS, Switzerland)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Terje Kvernes (University of Oslo, Norway)
  • Kurt Lust (Univ. of Antwerp, Belgium + LUMI User Support Team)
  • Robert Mijakovic (LuxProvide)
  • Mikael Öhman (Chalmers University of Technology, Sweden)
  • Bart Oldeman (Compute Canada)
  • Jörg Saßmannshausen (NIHR Biomedical Research Centre, UK)
  • Åke Sandgren (Umeå University, Sweden)
  • Davide Vanzo (Microsoft HPC)

Agenda

  • update on recent developments
  • quick update on evaluation of BLIS for 'foss' toolchain
  • revamp of EasyBuild documentation
  • Q&A

Recent developments

  • EasyBuild v4.3.3 was released last week (Tue Feb 23rd) \o/
  • master branch was renamed to main (which seems to become the standard on GitHub)
  • recent changes
    • framework
      • bug fixes
        • fix UTF-8 encoding errors when running EasyBuild with Python 3.0.x-3.6.x (PR #3565)
        • symlink 'lib' to 'lib64' if it doesn't exist (PR #3580)
          • mostly relevant for SLES (which favors lib64)
        • include %(mpi_cmd_prefix)s and %(cuda_*)s templates in output of --avail-easyconfig-templates (PR #3586)
        • fix $BLAS_LIB_MT for OpenBLAS, ensure -lpthread is included (PR #3584) (((not included yet in EB v4.3.3)))
        • avoid raised exception when getting output from interactive command in run_cmd_qa (PR #3599) (((not included yet in EB v4.3.3)))
      • enhancements
        • add support for using customized HTTP headers in download_file (PR #2472)
        • allow use of alternate envvar(s) to $HOME for user modules (PR #3558)
        • add support for intel-compiler compiler toolchain (>= 2021.x versions, oneAPI) (PR #3581)
          • only for "classic" Intel C/C++/Fortran compiler (icc, icpc, ifort)
          • TODO: oneAPI compilers (icx, icpx, ifx, dpcpp)
        • advise PR labels in --review-pr and add support for --add-pr-labels (PR #3177)
        • add support for filtering dependencies by using False as version (PR #3506 + PR #3600 <= to review/merge) (((not included yet in EB v4.3.3)))
          • use case: dependencies that are not supported on Arm64 or POWER: 
            dependencies = [("not_on_power", {'arch=*': '1.2.3', 'arch=POWER': False})]
        • add "Citing" section to module files (PR #3596) (((not included yet in EB v4.3.3)))
          • controlled via new citing easyconfig parameter
      • changes
        • (none)
    • easyblocks
      • bug fixes
        • make sure the installation of libiberty.a in the binutils easyblock goes into a populated directory (PR #2308)
        • fix for building GCC with --sysroot on ppc64le (PR #2315)
        • fix Boost sanity check on aarch64 (PR #2341)
        • filter out user packages in LAMMPS easyblock if corresponding dependency isn't included + only set -DUSER-INTEL on x86_64 systems (PR #2254)
        • read MATLAB configuration file in binary mode to avoid UTF-8 encoding errors when using Python 3.6 (PR #2340)
      • enhancements
        • new generic easyblock for Intel oneAPI compilers (PR #2305)
          • using intel-compilers as software name, including icc/icpc/ifort (classic compilers) + icx/icpx/ifx/dpcpp (oneAPI compilers)
        • update impi easyblock for impi 2021.x (oneAPI) (PR #2313)
        • add sanity check commands to GCC (including LTO support) (PR #2322)
        • run motorBike tutorial case as sanity check for recent (community) OpenFOAM versions (PR #2201) + add foamMonitor to sanity checks of OpenFOAM (PR #2256)
        • update PETSc easyblock to take into account that ScaLAPACK installation may not have header files + fix building in parallel (PR #2348) (((not included yet in EB v4.3.3)))
        • update CP2K easyblock w.r.t. running regtest for CP2K v8.1 (PR #2350) (((not included yet in EB v4.3.3)))
        • make OpenCV easyblock aware of protobuf, libwebp and OpenEXR dependencies provided via EasyBuild (PR #2346)
      • support for new software
      • changes
        • misleading --accept-eula configuration option and a wildcard proposal (issue #3595)
          • --accept-eula suggests boolean, should be --accept-eula-for...
            • add support for using regular expression patterns rather than only hard strings (--accept-eula-for='.*')
    • easyconfigs
      • over 100 merged easyconfig PRs since last conf call!
      • bug fixes
      • enhancements
        • verify checksum of all patch files in easyconfigs test suite (PR #12221)
      • new software
      • software updates
        • PyTorch 1.7.1 with fosscuda/2020b (PR #12003)
      • changes
        • bump JasPer version to latest 2.0.24 for 2020b generation of easyconfigs + remove easyconfigs for ancient JasPer 2.0.1.4 with GCCcore/10.2.0 (PR #12277)
  • to merge/fix/tackle soon
    • framework
      • bug fixes
        • using -fno-math-errno by default is causing numerical errors (issue #3598)
          • introduced in EasyBuild v3.6.0, see PR #2388
        • performance improvements for easyconfig parsing (PR #3555)
        • avoid module use in Lmod if possible to allow faster execution (PR #3557)
      • enhancements
        • add support for using fallback 'arch=*' key in dependency version specified as arch->version mapping (PR #3600)
            • add test for using False as version in dependencies to filter stuff
        • support additional features in easystack files
    • easyblocks
      • bug fixes
        • correctly determine path to active binutils in TensorFlow easyblock (PR #2218) [EESSI, RPATH]
        • fix taking into account --sysroot when installing/using CMake [EESSI] (PR #2247 or PR #2248, latter is best option?)
        • treat files/directories of unpacked sources equally in PackedBinary generic easyblock (PR #2306)
        • RPackages: Change to using R_LIBS_SITE when installing RPackages (PR #2326)/pull/2325))
        • set $NINJAFLAGS to make sure Ninja doesn't use all visible cores when building Qt5 (PR #2338)
          • we should probably set $NINJAFLAGS more generally (in framework), to ensure we cover other software that uses ninja under the hood?
      • enhancements
        • enhance OpenBLAS easyblock to make it aware of optarch (PR #1946)
          • (still) needs review + testing...
        • enhance test and install step of CMakePythonPackage easyblock (PR #2318)
        • add support for including PTX code in PyTorch (PR #2328)
        • enhance CUDA support in CP2K easyblock (WIP) (PR #2349)
          • needs more work to control which GPU features CP2K is being built with...
      • changes
        • (nothing)
      • new software
        • add custom easyblock for:
    • easyconfigs
      • bug fixes
        • (nothing major?)
      • enhancements
        • (nothing major?)
      • new software
        • (nothing major?)
      • software updates
        • (nothing major?)#

Quick update on evaluation of BLIS for 'foss' toolchain

Revamp of EasyBuild documentation

  • https://docs.easybuild.io
  • current status
    • hosted on readthedocs.io
    • .rst format (rendered by Sphinx)
    • some stuff is missing:
      • using a custom module naming schemes
      • writing easyconfig for (bundles of) Python package(s)
    • some pages in dire need of a refresh/update
      • pip install should be highlighted as recommended way of installing
  • EasyBuild tutorial (https://easybuilders.github.io/easybuild-tutorial) not integrated in current documentation
  • mkdocs is a way better tool
    • easier to work on documentation locally, trivial to get (live) preview of changes
    • also supported by readthedocs.io
    • .md (Markdown) format, so requires "porting" of .rst documentation
  • working group to revamp docs + port to mkdocs?
  • set up a starting point with a couple of pages + auto-ported .rst to .md (with pandoc)
  • involve people who are quite new to EasyBuild to help out with this

Q&A

  • Mikael: updates on merging of foss and fosscuda toolchains?
    • can we "hot swap" in a UCX with CUDA support?
    • make it configurable in EasyBuild which UCX is picked up (with or without CUDA)?
      • add support to configure EasyBuild to it knows it's installing software for a GPU system?
    • Åke: or just rebuild UCX (on GPU systems) once we have a compatible CUDA available?
      • that's perhaps a bit painful to do in production...
    • Bart: UCX has good plugin support
      • we could also install the GPU-specific part of UCX as a separate module, which sets the right $UCX_* environment variables?
    • let's make sure we merge foss and fosscuda for the 2021a common toolchains...