Conference call notes 20191113 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 136th EasyBuild conference call, Wednesday Nov 13th 2019 (17:00 - 18:00 CET)

Attendees

Alphabetical list of attendees (7):

Fotis Georgatos (SDSC, Switzerland)
Kenneth Hoste (HPC-UGent, Belgium)
Adam Huffman (Big Data Institute, Oxford)
Eirini Koutsaniti (CSCS, Switzerland)
Mikael Öhman (Chalmers University of Technology, Sweden)
Davide Vanzo (UT Southwestern)
Victor Holanda Rusu (CSCS, Switzerland)

Agenda

update on recent developments for next release (EasyBuild v4.1.0)
pros & cons of GitHub Actions vs Travis CI for running test suite
EasyBuild @ SC19
EUM'20
Q&A

Update on recent developments

next release (EasyBuild v4.1.0) planned for end of Nov'19
intention is to let Miguel take care of the actual release (with support from @boegel)
significant changes in framework (https://github.com/easybuilders/easybuild-framework/milestone/73)
- significant speedup in easyconfig test suite
  - thanks to https://github.com/easybuilders/easybuild-framework/pull/3074 + https://github.com/easybuilders/easybuild-framework/pull/3073 (latter only relevant for Python 3.x)
  - test suite now runs in 7-8 mins, used to be ~1h with Python 3.x...
- deprecating use of:
  - Lmod 6.x (https://github.com/easybuilders/easybuild-framework/pull/3077)
  - Python 2.6 (https://github.com/easybuilders/easybuild-framework/pull/3076)
  - mostly motivated by wanting to trim test configurations in Travis/GitHub a bit
- fixed minor compatibility issues with using EasyBuild on top of Python 3.8
  - https://github.com/easybuilders/easybuild-framework/pull/3078
- better integration with modules provided via packages in OpenHPC
  - first step: https://github.com/easybuilders/easybuild-framework/pull/3075
    - OpenHPC maintainers are planning to include a metadata files per RPM which EasyBuild can pick up on
  - more minor changes needed in order to use OpenHPC-provided modules as toolchain components
significant changes in easyblocks (https://github.com/easybuilders/easybuild-easyblocks/milestone/62)
- also run easyconfigs test suite in GitHub Actions
  - https://github.com/easybuilders/easybuild-easyblocks/pull/1844
significant changes in easyconfigs (https://github.com/easybuilders/easybuild-easyconfigs/milestone/65)
- fixed definition of fosscuda/2019b for use in hierarchical module naming scheme
  - w.r.t. location of CUDA in module hierarchy (from Core/ to Compiler/*) + correct subtoolchain (not GCCcore)
- also run easyconfigs test suite in GitHub Actions
  - https://github.com/easybuilders/easybuild-easyconfigs/pull/9231 + https://github.com/easybuilders/easybuild-easyconfigs/pull/9255

GitHub Actions vs Travis CI for running test suite

Travis
- pros
  - has worked well for a long time (in combination with @boegelbot)
  - automatic comments by @boegelbot to indicate broken tests in PR
- cons
  - flaky test failures (usually due to connectivity issues)
    - mostly fixed by letting @boegelbot re-trigger these jobs in Travis
  - slow, only 5-6 test runs in parallel (across all easybuilders repositories)
GitHub Actions
- pros
  - 20 test runs in parallel, so results come in a lot faster
  - native in GitHub
  - easily allows contributors to get test suite to pass first in their own fork before opening a PR
- cons
  - (still in beta, but going stable mid Nov'19)
  - minor problem: closing/re-opening triggers a new batch of tests, old batch of (broken) tests in GitHub CI stays in place
    - can be dealt with by implementing support for eb --sync-pr-with-develop?
self-hosted runners for GitHub Actions?
- supported (see https://github.blog/2019-11-05-self-hosted-runners-for-github-actions-is-now-in-beta/), but not recommended for repositories with forks for security reasons...
- workaround could be to implement own webhook which can be triggered by events on GitHub

EUM

Jan 29th-31st 2020 in Barcelona (Spain)
https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild-User-Meeting

EasyBuild @ SC19

"Getting Scientific Software Installed" BoF session hosted by Davide Vanzo (@vanzod) & Robert McLay (Lmod/XALT)
- short talk by Maxime (ComputeCanada) on their CVMFS+Nix+EasyBuild setup (PEARC'19 paper)
- interactive survey on tools related to software installation
get in touch with EasyBuilders attending SC19 via #sc19 channel in EasyBuild Slack
Davide will put together a social event (@ arcade bar)

Q&A

duplicate CUDA in fosscuda vs intelcuda
- CUDAcore with system toolchain (hidden from users in ComputeCanada)
- (empty) CUDA wrapper which loads CUDAcore & extends $MODULEPATH (in HMNS)
- this didn't go happen for fosscuda/2019b
  - Kenneth was reluctant to go through with it, not a big fan of the *core naming scheme (confuses users)
- Bart had another suggestion (which doesn't use CUDAcore)?
Victor & Eirini: aim at CSCS is to fully automate software deployment
- upgrading of software version + toolchain + dependencies
- Python scripts to automate this
  - files_to_copy sometimes disappears when dumping easyconfig files
    - see open issue: https://github.com/easybuilders/easybuild-framework/issues/3049
  - comments move around to weird places (open issue?)
- anyone else working on this?
- any progress on integration easy_update in EasyBuild framework?
  - see https://github.com/fizwit/easy_update
  - no progress there due to lack of time...
- how can we contribute back this functionality?
- Fotis: release monitoring in Fedora could be useful
Victor: how are test reports for PRs triggered?
- currently done manually (via eb --from-pr 12345 -fr --upload-test-report, see https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html)
- this could be automated, but needs some work (keeping security concerns in mind)
  - PRs that pass CI tests & are approved by a maintainer
  - VM could be spin op on demand for this
- PRs that depend on other PRs could be re-visited automatically via scripting in GitHub Actions?