EasyBuild 5.0 working group - easybuilders/easybuild GitHub Wiki

Useful links:

Monthly sync meetings

Every 1st Wed of the month at 14:00 CE(S)T, via Zoom

Next meetings

  • Wed 5 July 2023, 14:00 CEST
  • Wed 2 Aug 2023, 14:00 CEST
  • Wed 6 Sep 2023, 14:00 CEST
  • Wed 4 Oct 2023, 14:00 CEST

Meeting notes

2023-07-05

...

2023-06-07

Attendees: Bart, Sebastian, Kenneth, Jasper, Simon, Mikael, Alexandre

Pre-meeting notes

Discussion

  • GitHub Actions no longer support testing on top of Python 2.7 (cfr. framework PR #4267)

  • proposal on splitting up extensions for Python/Perl/R

    • prepped by Alex + Mikael
    • see https://hackmd.io/K1QhWGfdQWa2ZtlXlWVCCA
    • progress in context of 2023a common toolchains:
      • Perl/5.36.1-GCCcore-12.3.0 merged: (PR #17946)
        • includes only a handful of extensions, those required by Autoconf/Automake (Thread::Queue, Data::Dumper)
        • separate Perl-bundle-CPAN for additional Perl modules from CPAN: (PR #17970)
      • Python/3.11.3-GCCcore-12.3.0 merged (PR #17955)
        • only includes setuptools + pip + wheel (+ flit_core dep) as extensions
        • next step would be Python-bundle-PyPI (+ maybe separate easyconfigs for virtualenv, hatchling, poetry, etc.)
      • R: coming soon...

Progress

docs
  • WIP PR #137 to include overview of changes in EasyBuild 5.0
framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - minor conflicts fixed (commit f378bac)
  • some progress on enabling --trace by default by Simon (PR #4250)
    • Simon: fixing the tests is painful, but we're very close, only 1 test is failing (unexpected output being generated)
    • could consider setting $EASYBUILD_DISABLE_TRACE in test setup?
  • very little progress by Kenneth on new run function (issue #4252)
    • may be useful to pair up with someone on this in the coming weeks?
  • proposals for EasyBuild 5.0:
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit f93e6cd)
  • no merged or WIP PRs for now
  • TODO
    • consistently import LooseVersion from easybuild.tools instead of distutils.version in all easyblocks
    • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage
      • and then remove use_pip = True & co in all easyconfigs
    • update version of config.guess used by ConfigureMake (issue #2826)
      • important to test across a bunch of OSs & CPU architectures
        • Kenneth: aarch64 (Graviton* in AWS + Ampere Altra in Azure + A64FX in Isambard)
        • Kenneth: power9le (emulated)
        • Sebastian: riscv64
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit c0fd7db)
  • no merged or WIP PRs for now
  • TODO
    • archive easyconfigs using a deprecated toolchains + toolchain policy
    • archive ancient easyconfigs using system toolchain
    • archive directories that only have patches in them + cleaned up unused patches
    • remove use_pip = True & co in all easyconfigs

2023-05-03

Attendees: Simon, Kenneth, Adam, Bart, Mikael, Kurt, Alex, Jasper

Done

framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • (2023-04-07) remove (deprecated) EasyBuild bootstrap script (PR #4233)
  • (2023-04-12) remove support for Python 2.7 and 3.5 (PR #4229)
  • (2023-04-12) remove experimental support for YAML-based easyconfig format (.yeb) (PR #4237)
  • clean up deprecated functionality
    • (2023-04-15) remove deprecated --wait-on-lock option (PR #4239)
    • (2023-04-15) remove deprecated dummy toolchain (PR #4240)
    • (2023-04-15) remove deprecated --accept-eula option (PR #4242)
    • (2023-04-15) remove deprecated is_generic_easyblock from easyconfig.py (PR #4243)
    • (2023-04-15) remove deprecated use_git_am option to apply_patch (PR #4244)
    • (2023-04-15) remove deprecated fetch_extension_sources (PR #4245)
    • (2023-04-15) change default for change_into_dir to False for extract_file (PR #4246)
  • (2023-04-15) enforce correct .patch(.*) extension for patch files (PR #4247)
  • (2023-04-15) bump version to 5.0.0dev (PR #4236)
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • (2023-04-07) version bump to 5.0.0dev (PR #2917)
  • (2023-04-13) stop importing from deprecated easybuild.tools.py2vs3 module + stop testing with Python 2.7 and 3.5 (PR #2916)
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • (2023-04-07) version bump to 5.0.0dev (PR #17671)

Work-in-progress

framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Aopen+is%3Apr+label%3AEasyBuild-5.0+

  • enable --trace by default (PR #4250) [WIP Simon]
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Aopen+is%3Apr+label%3AEasyBuild-5.0+

(no open PRs currently)

easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Aopen+is%3Apr+label%3AEasyBuild-5.0+

(no open PRs currently)

Next

framework
  • rename unclear ExtensionEasyblock.run() method (issue #4113)
  • new run function (issue #4252)
    • Kenneth is currently exploring use of run_cmd + run_cmd_qa functions, to design API of run function
    • one run function vs multiple smaller run_* functions
    • keeping desired feature in the back of our head:
      • dropping into an interactive shell for a failing command
      • "wrapper" functions to run make with custom error patterns, etc.
    • also ties to desire for better error reporting
easyblocks
  • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (issues #1865 + #2127)
  • update version of config.guess used by ConfigureMake (issue #2826)
  • import LooseVersion from easybuild.tools
easyconfigs
  • archive easyconfigs for EasyBuild 5.0 (issue #17848) [WIP Simon]
    • we should also check easyconfigs using system toolchain
  • clean up unused patches (issue #11116)

TODO

Notes

  • Unclear whether we'll actually drop support for Lmod 6 + 7
    • Most recent Lmod 6.x was released in 2016
    • Most recent Lmod 7.x was released in 2019
    • Mostly to motivate people to move to Lmod 8.x
    • No real benefit to EasyBuild in terms of code cleanup, etc.
    • Simon will open issue to request input from sites that care (with some deadline)
    • Sites that use Ubuntu include Umeå Univ. + JSC, maybe others
    • We may also be annoying people who can't control the Lmod being used (not sysadmins)
  • What about with Python/R/Perl easyconfigs and their extensions?
    • completely bare (no extensions) + bundles of extensions on top
    • minimal (only key extensions like pip for Python, etc.)
    • for Python, getting rid of cryptography extensions (because it pulls in Rust as a build dependency)
    • only extensions to install Python packages: pip/setuptools/wheel/poetry/hatchling/build (+ virtualenv?)
    • separate bundle of Python build tools like pip & co
    • alias pip="python3 -m pip" in Python easyconfig
    • absolute minimum for Python: pip+wheel+setuptools
    • separate eayconfigs for poetry + hatchling + ...
    • rule to decide which extensions to include in Python could be "which Python packages are needed to install virtualenv with Python"
    • another rule could be "no extra dependencies should be added to Python just to support a particular extension"
    • whatever policy we come up with, it should be clearly motivated, documented, and easy to understand
    • who comes up with a couple of very concrete proposals on how to split Python extensions into bundles, along with clear rules that dictate what goes where?
    • similar discussion/rules needed for R + Perl + ...
    • should also consider user interface, how do we inform users that R doesn't include all of CRAN anymore?
      • may be a non-issue, most people probably just load R-bundle-Bioconductor
    • aspects to come up with rules for extensions
      • where do sources come from (CRAN/PyPI vs GitHub)
      • build vs runtime
      • extra deps needed
      • startup performance (# paths in $PYTHONPATH)