Conference call notes 20210623 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 175th EasyBuild conference call, Wednesday June 23rd 2021 (15:00 UTC)

Attendees

Alphabetical list of attendees (14):

• Sebastian Achilles (Jülich Supercomputing Centre, Germany)
• Simon Branford (Univ. of Birmingham, UK)
• Alex Domingo (Vrije Universiteit Brussel, Belgium)
• Alexander Grund (TU Dresden, Germany)
• Kenneth Hoste (HPC-UGent, Belgium)
• Adam Huffman (Big Data Institute, Oxford, UK)
• Kurt Lust (Univ. of Antwerp, Belgium + LUMI User Support Team)
• Sam Moors (Vrije Universiteit Brussel, Belgium)
• Alan O'Cais (Jülich Supercomputing Centre, Germany)
• Mikael Öhman (Chalmers University of Technology, Sweden)
• Bart Oldeman (Compute Canada)
• Jörg Saßmannshausen (NIHR Biomedical Research Centre, UK)
• Åke Sandgren (Umeå University, Sweden)
• Davide Vanzo (Microsoft)

Agenda

• overview of recent developments
• support for metadata (checksums, etc.) outside of easyconfig files
• Q&A

Recent developments

• last release: EasyBuild v4.4.0 (June 2nd)
• ETA next release: early July
• recent changes
  • framework
    • bug fixes
      • fix crash in get_config_dict when copying modules that were imported in easyconfig file (like 'import os') (PR #3729
      • fixes for Fujitsu compiler toolchain (PR #3721, PR #3731)
      • minor fixes to output of test reports when using multiple PRs (PR #3741)
      • fix location for modules installed with intel-compilers toolchain in HierarchicalMNS by always checking toolchain compiler name against template map (PR #3745)
    • enhancements
      • enhance test for post_pr_test_report to cover combination of --from-pr and --include-easyblocks-from-pr (PR #3726)
      • add module_write hook (PR #3728)
        • incl. docs update (PR #714, PR #715, PR #716)
      • add per-step timing information (PR #3716)
      • enhance detection of patch files supplied to 'eb' command with better error messages (PR #3709)
    • changes
      • make sure that tests requiring a github token have 'github' in the test name so that they can be easily filtered (PR #3730)
      • deprecate EasyBuild bootstrap script (PR #3742)
  • easyblocks
    • bug fixes
      • disable installation of bundled BioPerl and HTSLib if they are dependencies of VEP (PR #2468)
      • enhance intel_compiler easyblock to include multiarch include dir in $CPATH (PR #2477)
    • enhancements
      • enhance sitecfg to support overriding core Python packages (PR #2458)
      • enhance ConfigureMake generic easyblock to add support for building multiple build targets (PR #2449, PR #2480)
        • custom easyconfig parameter build_targets was renamed to build_cmd_targets to avoid clash with Clang easyblock (PR #2479)
      • enhance Amber easyblock to support installing Amber via CMake (PR #2445)
      • add file prefix option to XALT easyblock (PR #2463)
      • add custom sanity check to AOMP easyblock (PR #2464)
      • enhance Python easyblock to explicitly disable installing core-pip when install_pip is not True (PR #2476)
      • adjust Perl easyblock to only check for a "man" directory if groff is a dependency (PR #2474)
      • support arbitrary version strings in OpenSSL wrapper and add minimum_openssl_version option (PR #2475)
      • update Xmipp easyblock for new version (v3.20.07) (PR #2486)
    • new easyblocks
      • custom easyblock for torchvision (PR #2467)
    • changes
      • (none)
  • easyconfigs
    • over 100 easyconfig PRs merged since last conf call!
    • bug fixes
      • add patch for GCC 10.2.0 to fix internal compiler error on aarch64 (PR #13086)
      • add patch to fix uniq operation in TensorFlow 2.4.1 (PR #13181)
      • disable own avx detection of libfabric v1.12.1 (PR #13215)
        • only relevant for systems that don't have AVX2 support
      • add GCCcore-1[0-9].x to test_dep_versions_per_toolchain_generation (PR #13243)
    • enhancements
      • groff requires makeinfo as a build dep, which requires Perl, which requires groff. Break cyclic redundancy by adding Perl-minimal and makeinfo-minimal (PR #13163, PR #13164, PR #13165)
        • -minimal should be -bare?
    • new software
      • AOMP (PR #12909)
    • noteworthy software updates
      • TensorFlow 2.5.0 (PR #12906)
    • noteworthy changes
      • switch to NCCL 2.8.3 built from source for CuPy, Horovod, libgpuarray, PyTorch and TensorFlow using fosscuda/2020b (+ add missing astor extension for TensorFlow) (PR #13103)
      • set minimal OpenSSL version to 1.1.1 for OpenSSL v1.1 wrapper (PR #13188)
  • documentation
    • revamp EasyBuild installation docs: recommend pip or (manual) 3-stage bootstrap with pip + EasyBuild (PR #718, PR #719)
• to merge/fix/tackle soon
  • framework
    • reported bugs / bug fixes
      • (nothing major?)
    • enhancements
      • support additional features in easystack files
        • support for filtering via labels (PR #3620)
      • avoid using a priority in prepend_module_path (Lmod) to avoid costly module calls (PR #3636)
      • add support for installing extensions in parallel (WIP) (PR #3667)
      • add option to ignore failing test step (--ignore-test-failure) (PR #3732)
      • finding modules with multiple modulepaths and HMNS (issue #3703)
      • add support for NVHPC-based toolchains (PR #3735, PR #3736)
      • warn about generic milestone in --review-pr and --merge-pr (PR #3751)
      • add --review-pr-max and --review-pr-filter options to limit easyconfigs shown in multi-diff (WIP) (PR #3754)
    • changes
      • also enable static analysis for Python 2.7 (PR #3725)
        • catches accidentally overwriting local variables in list comprehensions (like --from-pr --include-easyblocks-from-pr bug)
      • add per-extension timing (PR #3734)
      • raise error for patches without .patch extension and no destination given (WIP) PR #3738)
      • error out when passing a list to run_cmd (PR #3737)
  • easyblocks
    • bug fixes
      • treat files/directories of unpacked sources equally in PackedBinary (PR #2306)
      • --module-only doesn't always work as expected
        • we need a better way of catching this in tests
        • problem is that you typically need an actual installation to catch these problems, so can't be done in easyconfigs or easyblocks test suite run in CI
        • test installations done on generoso via boegelbot could be enhanced to catch problems with --module-only?
      • explicitly use only OpenBLAS for PyTorch if MKL is not used (PR #2448)
      • Fix CPU-only runtime for dpcpp-generated executables in custom easyblock for intel-compilers (oneAPI) (PR #2457)
      • fix detection of installed Python packages in TensorFlow (PR #2469)
      • enhance sitecustomize.py script installed by Python easyblock to fix support for virtualenv (PR #2483)
      • change Bundle easyblcok to collect altroot and altversion later so that they appear in the module when running --module-only (PR #2485)
    • enhancements
      • enhance test and install step of CMakePythonPackage easyblock (PR #2318)
      • add support for installing R extensions in parallel (WIP) (PR #2408)
        • requires https://github.com/easybuilders/easybuild-framework/pull/3667
      • allow for Perl modules being part of other, already installed Perl modules (PR #2386)
      • including FlexiBLAS as the default BLAS in foss will require easyblock changes (issue #2421)
      • should set BLA_VENDOR in CMakeMake easyblock if BLAS is in the toolchain (PR #2420)
      • enable make check and sanity check exec for OpenMPI (PR #2444)
      • update custom easyblock for Boost to always build single and multi threaded versions (PR #2456)
        • included CMake modules are broken (and skipped) because we install Boost libraries multiple times
      • update CMakeMake to handle old and new Boost/Boost.Python builds using custom easyblock for Boost (PR #2461)
      • enhance Java easyblock to define %(jdkarch)s template (PR #2472)
      • enhance Hypre easyblock to enable CUDA support (PR #2482)
    • changes
      • (nothing major)
    • new software
      • custom easyblocks for RepeatModeler (PR #2470), crispr-dav (PR #2487)
      • Mikael: PR coming up for UCX-CUDA
  • easyconfigs
    • bug fixes
      • improve check for multi-variant dependencies per generation of easyconfigs (PR #12687)
    • enhancements
      • (nothing major)
    • new software
      • (nothing major?)
    • software updates
      • SciPy-bundle with intel/2021a (PR #12964)
        • need to look into handful of failing tests...

2021a update of common toolchains

• TODO: fosscuda/2021a
  • collapsing foss and fosscuda toolchains
  • see https://github.com/easybuilders/easybuild-easyconfigs/issues/12484
  • status? (Mikael)
    • see easyconfigs PR #12866
  • we wouldn't have fosscuda/2021a anymore, just depend on an extra dependency (UCX-CUDA) for GPU Direct RDMA support
  • CUDA bundle that depends on CUDAcore + UCX-CUDA to use as dependency?
  • CUDA support should be reflected in versionsuffix?
  • Intel MPI probably doesn't even support GPU Direct?
  • upstream UCX issue opened, developers were open to suggestion of environment variable
  • currently hardcoded in UCX-CUDA easyconfig PR, could be better to have a custom easyblock for this?
  • CUDA 11.1.3 is not supported with GCC 10.3
    • known issues with GCC 10.3
      • see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100102
      • patches already available for a compiler error (C++ template parsing error)
    • CUDA 11 is only compatible with GCC 9.x officially on x86_64

Support for metadata outside of easyconfig files

• cfr. Maxime's issue + PR on checksums.json
  • and related issue by Alexander
    • orthogonal idea, support for both could be implemented, and could make sense for different use cases
  • checksums are not for humans, and hence don't really belong in easyconfigs?
  • a separate checksums.json could clean up easyconfigs, avoid duplication of checksums across different easyconfigs
  • could also cause trouble?
    • merge conflicts on checksums.json (but should be rare and easy to resolve)
    • "split-brain" situation: just an easyconfig file is not enough anymore
      • already the case today with patch files!
      • eb could have support for generating a stand-alone easyconfig file
  • checksums.json idea could be extended to metadata.json
    • to specify parameter values that typically don't change across versions (homepage, description, source URL, source filename template, ...)
    • more intrusive than for checksums, since checksums are optional... (unless EasyBuild is configured with --enforce-checksums)

Q&A

• separate repository for EasyBuild documentation? easybuild-docs
  • migrate from .rst/Sphinx to .md/mkdocs to lower the bar of contributing to the documentation
  • Simon almost got put off updating the Installation docs because of .rst...
  • Kenneth plans to set up an empty EasyBuild documentation site as a starting point
    • with empty pages corresponding to the current documentation
    • hope is that several people can then look into migrating one or a couple of pages...
  • Simon: docs sprint to speed up migration?