Conference call notes 20210120 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 165th EasyBuild conference call, Wednesday January 20 2021 (09:00 UTC)

Attendees

Alphabetical list of attendees (10):

Simon Branford (University of Birmingham, UK)
Alex Domingo (Vrije Universiteit Brussel, Belgium)
Alexander Grund (TU Dresden, Germany)
Kenneth Hoste (HPC-UGent, Belgium)
Adam Huffman (Big Data Institute, Oxford, UK)
Christian Kniep (AWS)
Terje Kvernes (University of Oslo, Norway)
Robert Mijakovic (LuxProvide)
Mikael Öhman (Chalmers University of Technology, Sweden)
Jörg Saßmannshausen (NIHR Biomedical Research Centre, UK)

Agenda

update on recent developments + outlook to next release
Q&A

Recent developments

next release (v4.3.3): ~~before EUM'21?~~ during EUM week?
recent changes
- framework
  - bug fixes
    - don't clean up imported modules after verifying imports of included Python modules (PR #3544)
      - fix for obscure reported issue related to Python garbage collection (issue #3542)
  - enhancements
    - detect 'SYSTEM' toolchain as special case in easystack files (PR #3543)
    - enhance extract_cmd function to use 'cp -a' for shell scripts (.sh) (PR #3545)
      - we should also do this for *.whl "source" files (avoids the need for unpack_sources = False)
    - add support for specifying custom name to use when checking accepted EULA (PR #3546)
      - useful for Intel oneAPI components: can use 'Intel-oneAPI' to accept EULA for Intel compilers + impi + imkl
- easyblocks
  - bug fixes
    - set $PYTHONNOUSERSITE in PythonBundle.extensions_step (PR #2289)
      - to avoid picking up Python packages installed in $HOME/.local (with pip install --user) when skipping/installing extensions
    - fix Boost sanity check on POWER (PR #2291)
    - unify handling of pylibdirs and don't add duplicated $PYTHONPATH in PythonBundle (PR #2281)
    - enhance Amber easyblock to fix running of update_amber script when 'python' command is not available in OS (PR #2282)
  - enhancements
    - update Clang easyblock to add support for building extra tools + leveraging hwloc and Z3 as optional dependencies (PR #2310)
  - new software
    - (none)
- easyconfigs
  - over 50 merged easyconfig PRs since last conf call
  - bug fixes
    - added missing space in configopts in ParaView 5.8.0 easyconfigs using 2020a toolchain (PR #10989)
    - fix source URL in expat easyconfigs (and consistently add custom sanity_check_paths) (PR #11960)
    - add patch for Boost 1.74.0 to fix missing include file (PR #12007)
  - enhancements
    - (nothing major)
  - new software
    - (nothing major)
  - software updates
    - PyTorch 1.7.1 with fosscuda/2019b (PR #11636) and foss/2020b (PR #12008)
      - fosscuda/2020b is work-in-progress (PR #12003), failing tests?
      - issue with NCCL is fixed (multi-GPU test was running & failing on single GPU system)
    - SciPy-bundle v2020.03 with Python 2.7.18 and {foss,intel}/2020a (PR #11957)
    - ReFrame v3.3 (PR #11787)
  - changes
    - use libpng provided by EasyBuild in VTK to fix build issue on ppc64le (PR #11990)
to merge/fix/tackle soon
- framework (v4.3.3 milestone)
  - support additional features in easystack files (see issues #3468, #3512, #3513, #3516)
  - add toolchain deps to template resolution (PR #3541)
    - added so %(cudaver)s template is defined when using fosscuda toolchain
    - not merged yet because it results in the Toolchain instance being created a lot earlier
    - also, it triggers a search for the easyconfig files for the toolchain...
    - Kenneth will try to find time to look into alternative approach based on $EBVERSIONCUDA...
    - or maybe we just need to be a bit more careful w.r.t. taking toolchain dependencies into account when defining the templates?
  - symlink lib to lib64 if only lib64 exists (issue #3549)
    - other way around is already done (see PR #3401)
    - relevant for OpenSUSE, see easyblocks issue #2309 + easyconfigs issue #11976
- easyblocks (v4.3.3 milestone)
  - bug fixes
    - correctly determine path to active binutils in TensorFlow easyblock (PR #2218)
    - fix taking into account --sysroot when installing/using CMake
      - two options:
        
        use toolchain file in CMakeMake (PR #2247)
        
        patch CMake installation (PR #2248)
    - treat files/directories of unpacked sources equally in PackedBinary generic easyblock (PR #2306)
    - make sure the installation of libiberty.a in the binutils easyblock goes into a populated directory (PR #2308)
  - enhancements
    - enhance OpenBLAS easyblock to make it aware of optarch (PR #1946)
      - needs testing...
      - problem with numpy installed on Arm 64-bit with --optarch=GENERIC may be related, see https://github.com/EESSI/software-layer/issues/55
    - add support for statically linking Bazel (PR #2272)
    - improve Bazel easyblock: add support for running tests, enabl static linking, use build dir rather than tmpdir, verbose output (PR #2285)
    - add support for skipping steps in Python packages installed as extension (PR #2290)
    - update NEURON easyblock to use CMakeMake for recent versions (PR #2304)
    - add support for running TensorFlow CPU and GPU tests separately and enhance test failure reporting (PR #2312)
    - update impi easyblock for impi 2021.x (oneAPI) (PR #2313)
  - changes
    - create less temporary directories for TensorFlow by (only) using --output_user_root (PR #2293)
    - check for accepted EULA in custom easyblock for NVHPC (PR #2311)
  - new software
    - add custom easyblock for:
      - ADIOS (PR #2070)
      - JAX (PR #2262)
      - RELION (PR #2274)
        
        does adding an EasyBuild configuration option to specify the system job scheduler used make sense?
        
        yes, if it can help easyblocks to make proper decisions
        
        can also be useful for OpenMPI to opt-in to Slurm integration
    - new generic easyblock for Intel oneAPI compilers (PR #2305)
      - using intel-compilers as software name, including icc/icpc/ifort (classic compilers) + icx/icpx/ifx/dpcpp (oneAPI compilers)
- easyconfigs (v4.3.3 milestone)
  - bug fixes
    - fix for "error: 'runtime_error' is not a member of 'std'" in qtlocation (Qt5) (PR #12012)
  - enhancements
    - enable tests for most recent Bazel versions (PR #11894)
  - new software
    - Intel oneAPI
      - there will be no further updates to Intel Parallel Studio (apparently)
      - intel-compilers v2020.1 (PR #11982)
  - software updates
    - TensorFlow 2.4.0 (PR #11637)
    - Bazel v3.7.1 with GCCcore/10.2.0 (PR #11810)
      - requires patch?
      - see Alexander's patch for Bazel v3.7.1 included in (PR #11637)
    - OpenFOAM 8 with intel/2020b (PR #11975)
    - iomkl/2020b (PR #12009)
    - impi v2021.1 (PR #12026)

Q&A

Alex: Bazel patch to fix problem with new unzip command refusing to unpack old *.jar used by old Bazel versions
- see https://github.com/lexming/easybuild-easyconfigs/commit/b22b7506137716f87204f200518b1a35165ee5a4
- runtime patch in Bazel easyblock will be tricky/ugly
- so let's go with proper patch file for Bazel easyconfigs (but not too far back)
- should also try to contribute back the patch (which uses jar rather than unzip to unpack *.jar files)
Mikael's flatbuffers PR (PR #11804)
- included patch is required, but doesn't actually work yet
- Robert has a working patch
Robert has a bunch of PRs (~90) to contribute back: TensorFlow (fosscuda/2020b), PyTorch, Horovod, flatbuffers (with fixed patch), ...
Robert: any experiences with building OpenMPI with Clang?
- should ask Markus Geimer for feedback on this (@geimer in GitHub)