EasyBuild 5.0 sync meetings - easybuilders/easybuild GitHub Wiki

2024-02-07 monthly sync meeting (14:00 CET)

attending: ?

  • status of 5.0.x branches
    • synced with develop on 2024-02-07, only trivial merge conflicts had to be fixed
  • merged PRs
    • framework
      • deprecate support for EnvironmentModulesC (v3.2.x) and EnvironmentModulesTcl (ancient v1.x) module tools (PR #4439)
    • easyblocks
      • various PRs to port software-specific easyblocks to run_shell_cmd
        • for Perl, we are ignoring exit code of shell command in test_step, that seems like a bug (PR #3162)
        • being fixed in easyblocks PR #3170
        • no easyconfig for PALM (recently merged easyblock)
  • open PRs
    • framework
      • drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
        • should also switch to depends_on with Lmod as default?
      • replace parallel easyconfig parameter by maxparallel (PR #4398)
      • rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400)
        • "see http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html for more information" => this really needs to be updated (although link does still work), in a separate framework PR
      • bump minimum required Tmod (4.x) to 4.3.0 (PR #4425)
        • blocked by PR #4415 to fix running of test suite with Tmod >= v4.3
      • deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP) (PR #4433)
        • WIP until run_shell_cmd implementation is complete (incl. Q&A support)
      • create lib/lib64 symlink before running postinstallcmds (PR #4435)
        • requires fixes in easyconfigs that will get broken because of this
      • add support for running shell commands asynchronously with run_shell_cmd (PR #4444)
        • last place where framework itself still relies on run_cmd
        • implementation of EasyBlock.install_extensions_parallel can be improved further, but can be left for future PRs (even post-5.0 release)
        • ready for review/merge! => maybe Alex
        • requires small change in easyblocks (see easyblocks PR #3163)
      • enable RPATH linking by default (PR #4448)
    • easyblocks
      • 33 open PRs to port software-specific easyblocks to run_shell_cmd...
      • 9 open PRs to remove unused easyblocks
      • update run_async methods in RPackage, Rmpi, and Rserve easyblocks to use submit run_shell_cmd call into thread pool (PR #3163)
      • fix extension filter for Perl packages (PR #2699)
    • easyconfigs
      • clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail (PR #19265)
  • TODO
    • directories that don't contain any library files shouldn't be added to $LD_LIBRARY_PATH (issue #3504)
      • makes sense for (bundles of) Python packages
    • Clean up SLURM/Batchsystem environment before doing builds (issue #4434)
      • should be done opt-in, via --clean-up-slurm-env configuration setting
    • change default backend for --job to Slurm (+ deprecate GC3Pie support)
    • document how to start playing with development pre-release version of EasyBuild 5.0
      • create easyconfig to install EasyBuild/dev-5.0.x-<datestamp>

2024-01-29 sprint meeting (14:30 CET)

attending: Simon, Jasper, Kenneth, Adam, Bart

2024-01-22 sprint meeting (16:00 CET)

attending: Jasper, Alex, Simon, Sebastian, Kenneth (sort of)

  • status of 5.0.x branches
    • synced with develop on 20240119
  • merged PRs
    • framework
      • bump minimum required Lmod to 8.0.0 (PR #4424)
      • don't hardcode /bin/bash in eb script, RPATH wrapper script, and run_shell_cmd (PR #4441)
      • fix error reporting when test step fails (PR #4443)
    • easyblocks
      • remove hidden=True from run_shell_cmd in PythonPackage easyblock for consistency (PR #3074)
      • don't change installopts easyconfig parameter value in-place in PythonPackage easyblock (PR #3080)
      • stop running easyblocks test suite with Lmod 7.x (PR #3083)
    • easyconfigs
      • archive EasyBuild 3.x easyconfigs (PR #19656)
  • open PRs
    • framework
      • replace parallel easyconfig parameter by maxparallel (PR #4398)
      • rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400)
      • create lib/lib64 symlink before running postinstallcmds (PR #4435)
      • deprecate support for EnvironmentModulesC and EnvironmentModulesTcl module tools (PR #4439)
        • should just kick out these modules tools from testing configuration
      • add support for running shell commands asynchronously with run_shell_cmd (WIP) (PR #4444)
    • easyblocks
      • fix extension filter for Perl packages (PR #2699)
        • (still) awaiting more tests, need to compose list of relevant easyconfigs
      • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (PR #3022)
        • good progress on dealing with easyconfigs that were not enabling these parameters yet, see checklist in PR comment
        • open issues:
      • update ConfigureMake easyblock to error out on unknown configure args (PR #3025)
        • opt-out mechanism is implemented, some more requested changes => active work-in-progress
      • rename run method to install_extension, and likewise for prerun to pre_install_extension, postrun to post_install_extension, and run_async to install_extension_async (PR #3064)
      • disable use_pip by default for PyTorch, except for recent versions (>= 2.0) (PR #3079)
        • blocker for PR #3022
        • should be ready to merge (despite failed test report)
    • easyconfigs
      • clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail (PR #19265)
        • synced with 5.0.x by @boegel on 18 Jan'24
        • we should also add a check in easyconfigs test suite to prevent that use_pip = True & co are re-introduced in easyconfigs...
      • replace parallel by maxparallel (PR #19375)
    • docs
      • document how to transition from run_cmd to run_shell_cmd (PR #238)
  • TODO
    • (Kenneth) finish run_shell_cmd async PR + review Bart's docs PR
    • (Simon) auto-enable use_pip for PyTorch
    • (Sebastian) auto-enable use_pip & co in PythonPackage
    • (Alex) renaming run PR
    • (Jasper) look into PR for parallel -> maxparallel
    • porting easyblocks to run_shell_cmd
      • separate PRs: GCC, PyTorch, TensorFlow, ...

2024-01-15 sprint meeting (15:00 CET)

attending: Kenneth, Bart, Simon, Sebastian

  • merged PRs (since last meeting)
  • open PRs
    • easyblock PR #2699: fix extension filter for Perl packages
      • waiting until other easyconfigs using PerlModule have been checked/fixed
    • easyblocks PR #3022: enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock
      • Waiting for fixing easyconfigs mentioned it the list
      • once problem with CMakePythonPackage is fixed in PythonPackage easyblock, half of the easyconfigs not enabling use_pip will be working fine
    • easyblock PR #3025: update ConfigureMake easyblock to error out on unknown configure args
      • opt-out support has been added
    • easyblock PR #3074: pythonpackage: Remove hidden=True from run_shell_cmd for consistency
    • framework PR #3285: Deprecate failure to resolve a template value
    • framework PR #4248: Change tar command used in get_source_tarball_from_git to get reproducible tarballs
    • framework PR #4373: drop load storm safe guard for Environment Modules v4.2.4+
    • framework PR #4398: Replace parallel easyconfig parameter by maxparallel
    • framework PR #4400: rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions
      • needs more work, current approach doesn't actually work to make sure that easyblocks that haven't migrated yet to new methods
      • easyblock PR #3064: rename run method to install_extension, and likewise for prerun to pre_install_extension, postrun to post_install_extension, and run_async to install_extension_async
    • framework PR #4424: bump minimum required Lmod to 8.0.0
      • should we required a more recent Lmod 8.x?
      • harder to pick a particular 8.x than just using 8.0
      • except if we require Lmod >= 8.2.8 which supports extensions in module files
      • => ready to merge!
    • framework PR #4425: bump minimum required Tmod (4.x) to 4.3.0
    • framwork PR #4433: deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP)
      • should only be merged once implementation of run_shell_cmd
    • framwork PR #4435: Create lib/lib64 symlink before running postinstallcmds
      • will break some existing easyconfigs, so should only be done for EasyBuild 5.0
  • other
    • we should also deprecate support for Tcl-based modules tools, like ancient pure Tcl + Tmod 3.2.x
    • support for GC3Pie should be deprecated + use Slurm as default job backend
  • TODO
    • documentation on how to convert run_cmd to run_shell_cmd (Bart)
    • support testing easyconfigs + easyblocks with 5.0.x branch (Sebastian)
    • async option or thread pool for run_shell_cmd (Kenneth)
    • fix issue with use_pip = True + CMakePythonPackage (Kenneth)
    • Sebastian: review Bart's easyblock PR

2024-01-03 sync meeting (14:00 CET)

attending: Kenneth, Bart, Simon, Mikael, Sebastian

  • merged PRs (since last meeting)
    • (none)
  • status of 5.0.x branches
    • synced with develop after release of EasyBuild v4.9.0 (on 2023-12-31 + 2024-01-01)
    • for now, let's keep merging changes for EasyBuild v5.0 to 5.0.x branches, and keep develop for a potential v4.9.x release (but hopefully there won't be a need for one)
    • re-evaluate during sync meeting of Feb'24
  • open PRs
    • framework
      • deprecate failure to resolve a template value (PR #3285)
        • Kenneth should take a detailed look at this
        • raising an error when a template could not be resolved is fine, since we don't really want to continue anyway if that's the case
      • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
        • Jasper's testing shows that this won't really work as is
        • doesn't work on macOS
        • only works when .git folder is not retained
        • in the fridge for now, awaiting feedback from contributor + Jasper
      • drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
        • need to think how to align this with Lmod
        • do we change to using depends_on by default? (framework issue #4397)
          • helps with keeping default behaviour with Tmod & Lmod
          • supported since July 2017 (cfr. this commit)
          • => so yes
        • should also bump required Tmod version to 4.3 in PR #4373
          • better in a separate PR
        • along with Lmod to 8.x, see issue #871
      • detect Fortran .mod files in GCCcore installations (PR #4389)
        • only prints warning, unless EasyBuild is configured with --fail-on-mod_files-gcccore, so good to go as is? => yep
        • see also issue #4203 for list of easyconfigs where we need to whitelist (incl. Clang, AOCC, etc.)
      • replace parallel easyconfig parameter by maxparallel (PR #4398)
        • see also easyconfigs PR #19375
          • trivial change parallel -> maxparallel, but does change 721 easyconfigs (457 of which archived easyconfigs)
        • do we need to "soften the blow" a bit w.r.t. easyconfigs still using parallel (deprecate it rather than hard remove it)?
      • rename unclear run methods to install extensions (PR #4400)
        • basically ready to merge, except that log.deprecated calls should use '6.0' instead of '5.0'
        • needs matching easyblocks PR
      • use run_shell_cmd in scripts (PR #4422)
        • ready to review/merge => Simon
      • implement support for 'stream_output' option in run_shell_cmd (PR #4423)
        • ready to review/merge => Simon
    • easyblocks
      • fix extension filter for Perl packages (PR #2699)
        • awaiting test of other easyconfigs that (indirectly) rely on PerlModule easyblock (Kenneth)
      • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (PR #3022)
      • update ConfigureMake easyblock to error out on unknown configure args (PR #3025)
      • convert generic easyblocks to run_shell_cmd (PR #3046)
        • awaiting review/testing (Sebastian)
    • easyconfigs
      • clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail (PR #19265)
  • TODO
    • run_shell_cmd
      • implement async option
      • complete transition to run_shell_cmd in framework
      • complete docs (Bart)

2023-12-12 sync meeting (16:00 CET)

attending: Jasper, Kenneth, Sebastian, Bart, Simon, Alex, Alexandre

  • rename unclear run methods to install extensions (PR #4400)
    • let's deprecate run & co rather than remove them
    • run should print deprecation warning + call install_extension, etc.
  • migrate generic easyblocks to run_shell_command (PR #3046)
  • kim-api (PR #19399 through PR #19405, except #19400)
    • required because of check for Fortran .mod files when using GCCcore
    • there may be others...
  • change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
    • didn't work on macOS, so introduces a regression
    • got different checksums when retaining .git subdirectory
      • maybe because order of hidden files is not controlled?

2023-12-06 sync meeting (14:00 CET)

attending: Simon, Jasper, Sebastian, Kenneth, Adam, Alex, Mikael, Bart

  • merged PRs
    • better structure in docs for changes in EasyBuild v5.0 docs #230
      • note: only in develop branch for now, not in live documentation yet
  • open PRs
    • [Simon] detect Fortran .mod files in GCCcore installations PR #4389
      • we should make some extra effort to filter out blatant false positives?
      • actual Fortran .mod files will have this as output from file command: gzip compressed data
      • one step further would be to unzip and look for something like GFORTRAN module version ...., but that's maybe a bridge too far?
      • for now, let's merge as is, we can try to weed out false positives later if there's a real need for it
      • need to update boegelbot to set $EASYBUILD_FAIL_ON_MOD_FILES_GCCCORE=1 when using EasyBuild v5.0
    • [Jasper] change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
      • Jasper will take a look soon
    • [Kenneth, Sebastian review/merge] enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • [Kenneth] submit test reports
      • [Sebastian] Clean up easyconfigs that set these True => easyconfigs PR #19265
    • Deprecate failure to resolve a template value (framework PR #3285)
      • needs to be ported to 5.0.x
    • update ConfigureMake easyblock to error out on unknown configure args (easyblocks PR #3025)
      • see also easyblocks PR #3026 to start producing warnings for unknown configure options in EasyBuild 4.x
      • we should first fix the easyconfigs that are affected by this...
  • ideas: yay or nay?
    • --read-only-installdir by default? (framework issue #123)
      • impact:
        • annoying to remove install dirs later
        • potentially problematic for sites that have a group of users installing software with EasyBuild
    • change order of extract and prepare steps (framework issue #1376)
      • load (only build?) dependencies before extracting sources
    • don't run sanity check in install dir (framework issue #4044)
      • Simon has some use cases where they're currently "abusing" sanity check commands to make changes in install, which require that environment is correctly set up
    • PythonPackage easyblock: modulename should be a first-class parameter, not under the options parameter (issue #2352)
    • fix inconsistent license variable naming used in easyblocks (easyblocks issue #3043)
      • should always be $EB_SOFTWARENAME_LICENSE_SERVER
    • CMakeMake could default CMAKE_BUILD_TYPE to RelWithDebInfo instead of Release (issue #3032)
    • separate framework issue to open (Bart): enable -g by default (debug toolchain option)
    • including module extensions info in generated module file should be enabled by default
      • implies a sufficiently recent Lmod (>= 8.2.0)
      • will work even with older Lmod versions, because there's a version check in generated module file itself
    • should depends_on by enabled by default? => Mikael will open an issue
  • TODOs
    • [Kenneth,Sebastian] set up copy of boegelbot to test with EasyBuild 5.0.x branches
      • Sebastian has looked into this a bit
      • can add support for BRANCH=5.0.x
    • [Kenneth | Alex | Bart] Document run_shell_cmd changes
      • should be PR to easybuild-v5/run_shell_cmd/ page in docs (develop branch)
    • [Kenneth + Alex] finish implementation of run_shell_cmd
      • improve error output in case of long output produced by command
        • detect long output, point to output file?
      • stream_output
      • asynchronous
      • qa_patterns + qa_wait_patterns
    • [Bart,Alex,Adam,Jasper] start porting easyblocks to run_shell_cmd
    • [Sebastian] get easyblocks PR #3022 merged to auto-enable use_pip & co
    • [Kenneth] update easyconfigs that use an easyblock that derives from PythonPackage but do not enable use_pip & co yet to explicitly use False
      • this should ideally be done before easyblocks PR #3022 is merged...
      • should also check whether easyblocks that derive from PythonPackage will need to get use_pip = False...
      • check in easyconfigs test suite only checks easyconfigs that use PythonPackage or PythonBundle, not stuff that derives from them => should be fixed
    • [Alex] rename Extension.run method that is used for installing extensions (cfr. framework issue #4113)
      • deprecating use of .run(...)
      • better name:
        • Extension.install_as_extension?
      • definitely also affects Extension.postrun and Extension.prerun
    • [WHO?] see if we can limit the damage done by archiving easyconfigs using a deprecated toolchain
  • other
    • Bart's cheat sheet on run_shell_cmd 
      run_cmd -> run_shell_cmd

result:
(out, code) = ... -> res = ...; res.output, res.exit_code

parameter changes:
    log_ok, log_all, simple, regexp -> (remove!)
    inp -> stdin
    log_output=False -> output_file=True (reversed default)
    path -> work_dir
    force_in_dry_run -> in_dry_run
    verbose=True -> verbose_dry_run=False (reversed default)
    shell=None (->True) -> use_bash=True
    trace=True -> hidden=False (reversed meaning)
    stream_output, with_hooks (unchanged)
    asynchronous (NYI, don't touch for now!)
new parameters:
    fail_on_error=True, split_error=False, env=None
    qa_patterns=None (NYI)
    qa_wait_patterns=None (NYI)

Derived from detailed notes:
run_cmd                run_shell_cmd

(out, code)            .output, .exit_code, .stderr (if split_stderr=True)
cmd                    cmd
log_ok=True            (removed: always True)
log_all=False          (removed: always True)
simple=False           (removed: always return named tuple)
inp=None               stdin=None
regexp=True            (removed: no automatic scanning for stuff that looks like errors)
log_output=False       output_file=True
path=None              work_dir=None
force_in_dry_run=False in_dry_run=False
verbose=True           verbose_dry_run=False
shell=None(->True)     use_bash=True
trace=True             hidden=False
stream_output=None     stream_output=False
asynchronous=False     asynchronous=False (NYI)
with_hooks=True        with_hooks=True
                       fail_on_error=True (NEW)
                       split_error=False (NEW)
                       env=None (NEW)
                       qa_patterns=None (NEW,NYI)
                       qa_wait_patterns=None (NEW,NYI)

run_cmd
    :param cmd: command to run
    :param log_ok: only run output/exit code for failing commands (exit code non-zero)
    :param log_all: always log command output and exit code
    :param simple: if True, just return True/False to indicate success, else return a tuple: (output, exit_code)
    :param inp: the input given to the command via stdin
    :param regexp: regex used to check the output for errors;  if True it will use the default (see parse_log_for_error)
    :param log_output: indicate whether all output of command should be logged to a separate temporary logfile
    :param path: path to execute the command in; current working directory is used if unspecified
    :param force_in_dry_run: force running the command during dry run
    :param verbose: include message on running the command in dry run output
    :param shell: allow commands to not run in a shell (especially useful for cmd lists), defaults to True
    :param trace: print command being executed as part of trace output
    :param stream_output: enable streaming command output to stdout
    :param asynchronous: run command asynchronously (old: returns subprocess.Popen instance if set to True)
    :param with_hooks: trigger pre/post run_shell_cmd hooks (if defined)

run_shell_cmd
    :param fail_on_error: fail on non-zero exit code (enabled by default)
    :param split_stderr: split of stderr from stdout output
    :param stdin: input to be sent to stdin (nothing if set to None)
    :param env: environment to use to run command (if None, inherit current process environment)
    :param hidden: do not show command in terminal output (when using --trace, or with --extended-dry-run / -x)
    :param in_dry_run: also run command in dry run mode
    :param verbose_dry_run: show that command is run in dry run mode (overrules 'hidden')
    :param work_dir: working directory to run command in (current working directory if None)
    :param use_bash: execute command through bash shell (enabled by default)
    :param output_file: collect command output in temporary output file
    :param stream_output: stream command output to stdout
    :param asynchronous: run command asynchronously
    :param with_hooks: trigger pre/post run_shell_cmd hooks (if defined)
    :param qa_patterns: list of 2-tuples with patterns for questions + corresponding answers
    :param qa_wait_patterns: list of 2-tuples with patterns for non-questions
                             and number of iterations to allow these patterns to match with end out command output
    :return: Named tuple with:
    - output: command output, stdout+stderr combined if split_stderr is disabled, only stdout otherwise
    - exit_code: exit code of command (integer)
    - stderr: stderr output if split_stderr is enabled, None otherwise

2023-11-27 sprint (10:00 CET)

attending: Mikael, Sebastian, Bart

  • merged PRs
    • start adopting run_shell_cmd in easyblock.py (+ use f-strings) (PR #4383)
  • open PRs
    • [Kenneth,Mikael] better structure in docs for changes in EasyBuild v5.0 docs #230
    • [Kenneth] detect Fortran .mod files in GCCcore installations PR #4389
    • [Jasper] change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
    • [WHO? review/merge] enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • [Kenneth] submit test reports
      • [Sebastian] Clean up easyconfigs that set these true PR #19265
    • Deprecate failure to resolve a template value (framework PR #3285)
      • needs to be ported to 5.0.x
    • update ConfigureMake easyblock to error out on unknown configure args (easyblocks PR #3025)
      • see also easyblocks PR #3026 to start producing warnings for unknown configure options in EasyBuild 4.x
      • we should first fix the easyconfigs that are affected by this...
  • TODOs
    • [Kenneth] set up copy of boegelbot to test with EasyBuild 5.0.x branches
    • [Kenneth | Alex] Document run_shell_cmd changes
    • [Kenneth + Alex] finish implementation of run_shell_cmd
      • stream_output
      • asynchronous
      • qa_patterns + qa_wait_patterns
    • [Bart,Alex,Adam,Jasper] start porting easyblocks to run_shell_cmd
      • start with commonly used ones like ConfigureMake, CMakeMake, PythonPackage, EB_GCC, etc.
      • framework PR #4383 can be helpful for inspiration
    • [Sebastian] get easyblocks PR #3022 merged to auto-enable use_pip & co
    • [Kenneth] update easyconfigs that use an easyblock that derives from PythonPackage but do not enable use_pip & co yet to explicitly use False
      • this should ideally be done before easyblocks PR #3022 is merged...
      • should also check whether easyblocks that derive from PythonPackage will need to get use_pip = False...
      • check in easyconfigs test suite only checks easyconfigs that use PythonPackage or PythonBundle, not stuff that derives from them => should be fixed
    • [WHO?] rename Extension.run method that is used for installing extensions (cfr. framework issue #4113)
      • deprecating use of .run(...)
    • [WHO?] see if we can limit the damage done by archiving easyconfigs using a deprecated toolchain

2023-11-20 sprint (10:00 CET)

attending: Kenneth, Sebastian, Adam, Mikael

  • merged PRs
    • rename shell option in run_shell_cmd to use_bash (PR #4378)
    • print error message in main_with_hooks function when EasyBuildError was raised (PR #4380)
    • start adopting run_shell_cmd in easyblock.py (+ use f-strings) (PR #4383)
  • open PRs
    • [Jasper] change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
    • [WHO? review/merge] enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • [Kenneth] submit test reports
    • Deprecate failure to resolve a template value (framework PR #3285)
      • needs to be ported to 5.0.x
    • update ConfigureMake easyblock to error out on unknown configure args (easyblocks PR #3025)
      • see also easyblocks PR #3026 to start producing warnings for unknown configure options in EasyBuild 4.x
      • we should first fix the easyconfigs that are affected by this...
  • TODOs
    • [Kenneth + Alex] finish implementation of run_shell_cmd
      • stream_output
      • asynchronous
      • qa_patterns + qa_wait_patterns
    • [Bart,Alex,Adam,Jasper] start porting easyblocks to run_shell_cmd
      • start with commonly used ones like ConfigureMake, CMakeMake, PythonPackage, EB_GCC, etc.
      • framework PR #4383 can be helpful for inspiration
    • [Kenneth,Adam] better structure in docs for changes in EasyBuild v5.0
    • [Sebastian] get easyblocks PR #3022 merged to auto-enable use_pip & co
    • [Kenneth] update easyconfigs that use an easyblock that derives from PythonPackage but do not enable use_pip & co yet to explicitly use False
      • this should ideally be done before easyblocks PR #3022 is merged...
      • should also check whether easyblocks that derive from PythonPackage will need to get use_pip = False...
      • check in easyconfigs test suite only checks easyconfigs that use PythonPackage or PythonBundle, not stuff that derives from them => should be fixed
    • [Sebastian] clean up easyconfigs that explicitly set use_pip & co to True
      • only after easyblocks PR #3022 is merged
      • use_pip = True, 'use_pip': True => remove
      • sanity_pip_check = True => remove
      • download_dep_fail = True => remove
    • [WHO?] rename Extension.run method that is used for installing extensions (cfr. framework issue #4113)
      • deprecating use of .run(...)
    • [WHO?] see if we can limit the damage done by archiving easyconfigs using a deprecated toolchain

2023-11-13 sprint (10:00 CET)

attending: Kenneth, Adam, (Mikael)

  • merged PRs
    • change run_shell_cmd to store command output in temporary file(s) by default + pass RunShellCmdResult instance to RunShellCmdError (framework PR #4356)
  • open PRs
    • [Jasper] change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
    • [WHO? review/merge] enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • [Kenneth] submit test reports
    • [Adam to review/merge] rename shell option in run_shell_cmd to use_bash (framework PR #4378)
  • next steps
    • [Alex,Kenneth] fix error reporting in 5.0.x branch
      • failing to resolve dependencies doesn't produce an error message anymore (only non-zero exit code)
    • [Kenneth + Alex] finish implementation of run_shell_cmd so framework can be updated to only use run_shell_cmd (no more run_cmd)
      • implement support for stream_output option (required in easybuild/tools/containers/*.py)
      • implement support for asynchronous option (required for --parallel-extensions-install support)
    • [Kenneth?] implement support for Q&A commands in run_shell_cmd
      • implement support for qa_patterns and qa_wait_patterns options
    • [Bart,Alex,Adam,Jasper] start porting easyblocks to run_shell_cmd
      • start with commonly used ones like ConfigureMake, CMakeMake, PythonPackage, EB_GCC, etc.
    • [WHO?] rename Extension.run method that is used for installing extensions (cfr. framework issue #4113)
    • get easyblocks PR #3022 merged to auto-enable use_pip & co
      • Kenneth can look into submitting a bunch of test reports for affected easyblocks
    • [Kenneth] update easyconfigs that use an easyblock that derives from PythonPackage but do not enable use_pip & co yet to explicitly use False
      • WIP script to identify the easyconfigs to update (which somehow triggers open file handle limit...)
    • [WHO?] clean up easyconfigs that explicitly set use_pip & co to True
    • [Alex(structure),Bart(run_shell_cmd transition table),Adam?] update documentation to have a more complete overview of changes in EasyBuild 5.0
      • cfr. https://docs.easybuild.io/easybuild-v5/overview-of-changes
      • should include an overview of how to transition from run_cmd to run_shell_cmd, inspired by how easyblocks are updated to adopt run_shell_cmd
      • organise based on users vs contributors vs developers
        • Changes relevant to end users => docs/roles/end-users/easybuild-v5
          • trace output by default => docs/easybuild-v5/trace_output
          • error reporting => docs/easybuild-v5/error_reporting
          • Lmod version
          • toolchain support policy
          • Python 2 no longer supported
          • RIP experimental .yeb
          • RIP bootstrap script
        • contributors
          • easyconfigs
            • toolchain support policy
            • enforce correct .patch(.*) extension for patch files
          • easyblocks/framework
            • run_shell_cmd
            • py2vs3
            • RIP Python 2
            • removal of deprecated functionality
            • changed defaults
              • change_dir in extract_file
        • developers/maintainers
          • toolchain support policy
          • (see easyblocks/framework contributors)
      • separate pages for:
        • run_shell_cmd
    • [WHO?] see if we can limit the damage done by archiving easyconfigs using a deprecated toolchain

2023-11-08 sync meeting (14:00 CET)

attending: Kenneth, Bart, Simon, Alexandre, Sebastian, Adam, Alex, Jasper

  • merged PRs
    • use 5.0.x branch for easyblocks + easyconfigs in CI workflows (for now) (framework PR #4358)
    • take into account that VERBOSE_VERSION imported from easybuild.easyblocks is now a string value (framework PR #4357)
  • open PRs
    • [Kenneth,Alex] change run_shell_cmd to store command output in temporary file(s) by default + pass RunShellCmdResult instance to RunShellCmdError (framework PR #4356)
      • status: reviewed + updated based on review, ready to merge?
    • [Jasper] change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
      • PR branch needs to be updated with current 5.0.x to re-trigger CI
      • someone should play with this across different systems/OSs to verify whether this is indeed sufficient to obtain reproducible tarballs for easyconfigs using git_config
    • deprecate failure to resolve a template value (framework PR #3285)
      • needs thorough review/testing
      • needs to be updated with current 5.0.x branch to re-trigger CI
      • Kenneth will ask Alexander to move to 5.0.x
    • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • needs review + testing with a handful of existing easyconfigs to ensure no bugs have crept in the refactoring, at least one test for each easyblock that was touched in the PR
    • [Alexander?] update ConfigureMake easyblock to error out on unknown configure args (easyblocks PR #3025)
  • next steps
    • [Alex] get framework PR #4356 merged to fix storing of command output to file(s)
    • [Alex] fix error reporting in 5.0.x branch
      • failing to resolve dependencies doesn't produce an error message anymore (only non-zero exit code)
    • [Kenneth + Alex] finish implementation of run_shell_cmd so framework can be updated to only use run_shell_cmd (no more run_cmd)
      • rename shell option to use_bash
      • implement support for stream_output option (required in easybuild/tools/containers/*.py)
      • implement support for asynchronous option (required for --parallel-extensions-install support)
    • [Kenneth?] implement support for Q&A commands in run_shell_cmd
      • implement support for qa_patterns and qa_wait_patterns options
    • [Bart,Alex,Adam,Jasper] start porting easyblocks to run_shell_cmd
      • start with commonly used ones like ConfigureMake, CMakeMake, PythonPackage, EB_GCC, etc.
    • [WHO?] rename Extension.run method that is used for installing extensions (cfr. framework issue #4113)
    • get easyblocks PR #3022 merged to auto-enable use_pip & co
      • Kenneth can look into submitting a bunch of test reports for affected easyblocks
    • [Kenneth] update easyconfigs that use an easyblock that derives from PythonPackage but do not enable use_pip & co yet to explicitly use False
      • WIP script to identify the easyconfigs to update (which somehow triggers open file handle limit...)
    • [WHO?] clean up easyconfigs that explicitly set use_pip & co to True
    • [Alex(structure),Bart(run_shell_cmd transition table)] update documentation to have a more complete overview of changes in EasyBuild 5.0
    • [WHO?] see if we can limit the damage done by archiving easyconfigs using a deprecated toolchain
      • supported software (excl. versions) in EasyBuild v4.8.2 vs current 5.0.x branch: 3,292 vs 2,365 => ~29% decrease :-/

2023-10-23 sprint (10:00 CEST)

attending: Kenneth, Mikael

  • merged PRs
    • document policy on supported toolchain generations (docs PRs #200 + #216)
    • improve error reporting for failing shell commands (and EasyBuild crashes) (PR #4351)
    • use LooseVersion from easybuild.tools in all easyblocks (easyblocks PR #3018)
      • VERSION and VERBOSE_VERSION constants in easybuild.easyblocks are now string values (used to be LooseVersion instance)
      • requires framework #4357: take into account that VERBOSE_VERSION imported from easybuild.easyblocks is now a string value
    • don't auto-enable use of -DSCOTCH_PTHREAD when using MPI library other than Intel MPI in SCOTCH easyblock (PR #3019)
    • improve error reporting for failing shell commands (and EasyBuild crashes) #4351
  • open PRs
    • change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
    • take into account that VERBOSE_VERSION imported from easybuild.easyblocks is now a string value (framwork PR #4357)
    • change run_shell_cmd to always store command output in temporary file(s) + pass RunShellCmdResult instance to RunShellCmdError (WIP (framework PR #4356)
      • tests are failing, needs more work
    • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • worth doing a couple of test build for this...
  • other
  • next 5 goals
  • timeline
    • EasyBuild 4.8.2 release soon
      • may be last EasyBuild 4.x release
      • 2023b common toolchain update could be reason for EasyBuild v4.9.0 release (~end of Nov'23)
    • for now, stick to 5.0.x branches, until we're sure that EasyBuild 5.0 can be released soon enough

2023-10-16 sprint (10:00 CEST)

attending: Kenneth, Alex, Simon, Mikael, Adam

  • merged PRs
    • stop importing from easybuild.tools.py2vs3 (+ minor cleanup in init easyblocks test) (easyblocks PR #3015)
    • scripts to archive easyconfigs (easyconfigs PR #18934)
      • script used to archive easyconfigs using SYSTEM toolchain needs a whitelist to avoid archiving things like GCC-system.eb, etc.
    • archiving of old easyconfigs
      • all done for all toolchain versions older than 2020a (across 14 PRs)
      • remove old archived easyconfigs (EasyBuild 4.x archive) (PR #18982)
      • use more recent easyconfigs in tests for --search (PR #18995)
      • archive SYSTEM level easyconfigs from before 2018-10-11 (PR #19013)
      • fallout
        • support for 937 (out of ~3,300) software packages will be lost in EasyBuild 5.0
          • ~28% of all software EasyBuild currently supports...
          • with EasyBuild 4.0, is was way less dramatic (1840 -> 1669, ~9%)
          • we can/should add easyconfigs back with more recent toolchains to partially mitigate this...
          • a bunch of these are:
            • toolchains we no longer use
            • stuff like pandas that's still supported as extension
            • complex ones like RELION, ...
      • we should also do a cleanup sweep of patches
      • overview of software in EasyBuild docs should also include extensions?
  • open PRs
    • improve error reporting for failing shell commands (and EasyBuild crashes) (framework PR #4351)
      • reviewed by Alex, suggested changes made by Kenneth, ready for re-review + merge...
      • in some way, it would be easier to always have stdout and stderr command output split
        • but then we lose "context" of errors
        • can we collect both stdout+stderr mixed and stderr output only when running a command?
    • Use LooseVersion from easybuild.tools and sort imports (easyblocks PR #3018)
      • use of LooseVersion in easybuild/easyblocks/__init__.py is causing trouble
    • don't auto-enable use of -DSCOTCH_PTHREAD when using MPI library other than Intel MPI in SCOTCH easyblock (easyblocks PR #3019)
  • next 5 goals
    • [Kenneth,Alex] get (framework PR #4351) merged (better error reporting)
    • [Kenneth] implement support for stream_output in run_cmd_shell function + switch to run_cmd_shell in easybuild/tools/containers/*.py
    • [Mikael,Kenneth] get (easyblocks PR #3018) merged (LooseVersion import)
      • need to stop using LooseVersion in easybuild/easyblocks/__init__.py
    • [Sebastian?] enable download_dep_fail + use_pip + sanity_pip_check by default in PythonPackage + PythonBundle easyblocks
      • easyconfigs cleanup should be less painful now
      • Kenneth is working on script to identify eaysconfigs that do not already enable these parameters
    • [Alex] start switching to run_shell_cmd in easyblocks...

2023-10-09 sprint (10:00 CEST)

attending: Kenneth, Mikael, Sebastian, Adam, Alexandre

  • merged PRs
  • open PRs
    • improve error reporting for failing shell commands (and EasyBuild crashes) (framework PR #4351)
      • => will be reviewed/tested/merged this week by ...
    • stop importing from easybuild.tools.py2vs3 (+ minor cleanup in init easyblocks test) (easyblocks PR #3015)
      • => will be reviewed/merged this week by ...
    • scripts to archive easyconfigs (easyconfigs PR #18934)
      • => will be reviewed/tested/merged this week by ...
  • next 5 goals
    • [Alex + Kenneth] get framework PR #4351 that implements better error reporting merged
    • [Kenneth] implement support for stream_output in run_cmd_shell function + switch to run_cmd_shell in easybuild/tools/containers/*.py
    • [Chia-jung Hsu] switch easyblocks to LooseVersion from easybuild.tools.* (only change import statements)
    • [Sebastian] enable download_dep_fail + use_pip + sanity_pip_check by default in PythonPackage + PythonBundle easyblocks
      • explicitely set these to False in easyconfigs not setting them to True
        • easyconfigs using any easyblock that derives from PythonPackage or PythonBundle
        • Kenneth can look into generating a list of easyconfigs that will need fixing (setting to False)
      • clean up easyconfigs explicitly setting these to True
      • easyconfigs test suite should report explicit setting these to True
    • [Mikael] archive easyconfigs
      • using a 2016a/2016b/2016.x toolchain
      • using GCC 5.4.0 or older as toolchain
      • using scripts implemented by Mikael (easyconfigs PR #18934)

2023-10-04

Attendees: Kenneth, Sebastian, Mikael, Alexandre, Bart, Simon, Jasper, Adam

framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-10-02 (commit d450a80)
  • merged PRs:
    • ...
  • open PRs:
    • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
  • TODO
    • for run_shell_cmd
      • improve error reporting
        • raise EasyBuildError instead of subprocess.CalledProcessError
        • implement better error reporting, using a new report_error function that uses return value produced by run_shell_cmd (which cmd used, includes output, exit code, working dir, ...)
        • WIP: 
          == installing...
  >> running command:
        [started at: 2023-10-04 13:59:58]
        [working dir: /private/tmp/kehoste/bzip2/1.0.6/system-system/bzip2-1.0.6]
        [output logged in /tmp/eb-leexxvrd/easybuild-run-rrpla98w.log]
        xxmake -f Makefile-libbz2_so CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'

- full command: xxmake -f Makefile-libbz2_so CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'
- exit code: 127
- working directory: /private/tmp/kehoste/bzip2/1.0.6/system-system/bzip2-1.0.6
- output: /tmp/eb-leexxvrd/shell-cmd-errorq2fsapve/xxmake.out
- stderr: /tmp/eb-leexxvrd/shell-cmd-errorq2fsapve/xxmake.err

ERROR: shell command 'xxmake' failed!
        • run_shell_cmd could create script to source to start interactive debug shell, something like: 
          export TEST=foo
cd /tmp/test
history -s "test 1 2 3"
echo "Failing command was 'test 1 2 3'"
      • implement missing options in run_shell_cmd
        • stream_output
        • asynchronous
        • qa_patterns + qa_wait_patterns
      • complete transition to run_shell_cmd in easybuild-framework
        • TODO:
          • easybuild/framework/easyblock.py
          • easybuild/framework/extension.py
          • easybuild/toolchains/linalg/flexiblas.py
          • easybuild/tools/containers/*.py
          • easybuild/tools/job/slurm.py
          • easybuild/tools/options.py
          • easybuild/tools/package/utilities.py
      • gradually switch easyblocks to run_shell_cmd (in 5.0.x branch)
      • add support for dropping into a interactive shell if command failed (cfr. issue #3950)
        • global configuration option to opt-in to letting EasyBuild drop into an interactive shell if a shell command failed
        • only when command is not expected to fail
        • only when there's a tty
        • opt-out option in run_shell_cmd for specific commands
    • tackle other ideas, see https://github.com/easybuilders/easybuild-framework/labels/EasyBuild-5.0
  • how can we activate more people?
    • half-day "sprint" to outline next things to do?
    • better overview of easy things to do to make progress => open issues
  • archiving of easyconfigs
    • for oldest toolchains (older than 2020a)
      • generation by generation
      • step 1: grep '^toolchain.*2019b' easybuild/easyconfigs
      • step 2: grep '^toolchain.*GCC.*8.3.0' easybuild/easyconfigs
      • step 3: close easyocnfig PRs using these toolchains
      • don't worry about conflicts that get introduced by PRs currently open to develop
    • by letter subdir
      • doesn't really work due to dependencies
    • oldest easyconfigs using system toolchain => archive software versions older than 5 years
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-10-04 (commit b2d86f5)
    • trivial conflicts fixed
  • next steps
    • switching from run_cmd to run_shell_cmd
      • tricky for ones using run_cmd_qa
    • switching to LooseVersion from easybuild.tools.* (only change import statements)
    • enable use_pip + sanity_pip_check in PythonPackage
      • clean up easyconfigs after archiving
    • remove dead easyblocks
    • remove dead code in easyblocks (ancient software versions)
    • update config.guess in ConfigureMake (issue #2826)
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-10-04 (commit db2d79d)
    • trivial conflicts fixed
task assignments
  • [Kenneth] run_shell_cmd error reporting
  • [Mikael] archiving easyconfigs
    • {foss,intel}/2016a + GCC(core) 4.9.3-2.25
    • script in scripts/ subdirectory in easybuild-easyconfigs repo
  • [Mikael's colleague?] flip defaults for download_dep_fail + use_pip + sanity_check_pip in PythonPackage easyblock
  • [Mikael's colleague?] fix LooseVersion imports in easyblocks
  • [Sebastian] update config.guess in ConfigureMake (issue #2826)
  • [Mikael] clean up SCOTCH easyblock (issue #2972)
sprint meeting (max 1h)
  • Mon 9 Oct 10:00 CEST

2023-09-06

Attendees: Kenneth Hoste, Simon Branford, Alex Domingo, Jasper Grimm, Mikael Öhman, Bart Oldeman, Alexandre Strube

framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-09-06 (commit 02ed943)
    • conflicts fixed, mostly due to support for run_shell_cmd hook that was added in develop via PR #4323
  • merged PRs:
    • Simon's PR to remove deprecated functionality:
      • PR #4273 remove deprecated descr option for simple_option
      • PR #4274 remove deprecated Toolchain.add_dependencies
      • PR #4275 remove deprecated functionality from filetools
      • PR #4276 remove deprecated log_error option from which function
      • PR #4277 remove deprecated skip_lower option from template_constant_dict
      • PR #4278 remove deprecated options from easyconfig.py
      • PR #4279 removed deprecated mod_exists_regex_template in ModulesTool.exist
    • progress on run_shell_cmd:
      • initial implementation of run function to replace run_cmd + run_cmd_qa (PR #4284)
      • copy load_source function from easybuild.tools.py2vs3.py3 to easybuild.tools.filetools so it can be used in easybuild.tools.hooks (PR #4306)
      • disable trace output for commands run in get_source_tarball_from_git (PR #4310)
      • fix broken github tests in options test.framework.options by disabling trace output in download_repo and fetch_files_from_pr functions (PR #4311)
      • fix for test_get_source_tarball_from_git broken by trace output (PR #4310)
      • fix broken github tests in options test.framework.options by disabling trace output in download_repo and fetch_files_from_pr functions (PR #4311)
      • implement fail_on_error/in_dry_run/output_file options + enable caching for run function, and switch from run_cmd to run function in systemtools (PR #4314)
      • implement support in run function for splitting stdout and stderr output (split_stderr) + passing down environment to use, and switch to run function for running module commands (PR #4321)
      • also include command used + working directory in return value of run function (PR #4322)
      • implement support in 'run' function for running command in different working directory + switch to run function in filetools (PR #4327)
      • trigger run_shell_cmd hook in run function (PR #4334)
        • run_shell_cmd hook will also be supported in EasyBuild v4.8.1
      • rename run function to run_shell_cmd (PR #4335 + PR #4336)
  • open PRs:
    • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
  • TODO
    • for run_shell_cmd
      • improve error reporting
        • raise EasyBuildError instead of subprocess.CalledProcessError
        • implement better error reporting, using a new report_error function that uses return value produced by run_shell_cmd (which cmd used, includes output, exit code, working dir, ...)
      • implement missing options in run_shell_cmd
        • stream_output
        • asynchronous
        • qa_patterns + qa_wait_patterns
      • complete transition to run_shell_cmd in easybuild-framework
        • TODO:
          • easybuild/framework/easyblock.py
          • easybuild/framework/extension.py
          • easybuild/toolchains/linalg/flexiblas.py
          • easybuild/tools/containers/*.py
          • easybuild/tools/job/slurm.py
          • easybuild/tools/options.py
          • easybuild/tools/package/utilities.py
      • gradually switch easyblocks to run_shell_cmd (in 5.0.x branch)
      • add support for dropping into a interactive shell if command failed (cfr. issue #3950)
        • global configuration option to opt-in to letting EasyBuild drop into an interactive shell if a shell command failed
        • only when command is not expected to fail
        • only when there's a tty
        • opt-out option in run_shell_cmd for specific commands
    • tackle other ideas, see https://github.com/easybuilders/easybuild-framework/labels/EasyBuild-5.0
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-09-06 (commit d7b8b45)
    • trivial conflicts fixed
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-09-06 (commit a82280d)
    • trivial conflicts fixed

2023-08-02

Attendees: Kenneth, Kurt, Alexandre, Sebastian, ...

Discussion

docs
Policy changes
  • toolchain support policy (issue #872)
  • supported Lmod versions (issue #871)
    • definitely drop support for Lmod 6
    • dropping support for Lmod 7 probably has low impact
    • should we require a minimal Lmod 8.x version (8.2 or 8.3)
      • for extensions, depends_on, etc.
      • bump to REQ_VERSION to 8.0 (or 8.3), DEPR_VERSION to 8.5
framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - minor conflicts fixed (commit f378bac)
  • merged PRs:
    • PR #4272 remove support for 32-bit targets
  • open PRs:
    • PR #4284: initial progress by Kenneth, following discussion with Alex, on new run function (issue #4252)
  • several PRs for code cleanup in 5.0.x branch, by removing deprecated stuff
    • PR #4273 remove deprecated descr option for simple_option
    • PR #4274 remove deprecated Toolchain.add_dependencies
    • PR #4275 remove deprecated functionality from filetools
    • PR #4276 remove deprecated log_error option from which function
    • PR #4277 remove deprecated skip_lower option from template_constant_dict
    • PR #4278 remove deprecated options from easyconfig.py
    • PR #4279 removed deprecated mod_exists_regex_template in ModulesTool.exist
  • proposals for EasyBuild 5.0:
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit f93e6cd)
  • no merged or WIP PRs for now
  • TODO
    • stop using deprecated py2vs3 module in easyblocks
    • start switching to new run function
    • consistently import LooseVersion from easybuild.tools instead of distutils.version in all easyblocks
    • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage
      • and then remove use_pip = True & co in all easyconfigs
    • update version of config.guess used by ConfigureMake (issue #2826)
      • important to test across a bunch of OSs & CPU architectures
        • Kenneth: aarch64 (Graviton* in AWS + Ampere Altra in Azure + A64FX in Isambard)
        • Kenneth: power9le (emulated)
        • Sebastian: riscv64
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit c0fd7db)
  • no merged or WIP PRs for now
  • TODO
    • archive easyconfigs using a deprecated toolchains + toolchain policy
    • archive ancient easyconfigs using system toolchain
    • archive directories that only have patches in them + cleaned up unused patches
    • remove use_pip = True & co in all easyconfigs

2023-07-05

Attendees: Kenneth, Simon, Jasper, Kurt, Mikael, Alex, Adam, Sebastian, Bart

Discussion

docs
Policy changes
  • toolchain support policy (issue #872)
  • supported Lmod versions (issue #871)
    • definitely drop support for Lmod 6
    • dropping support for Lmod 7 probably has low impact
    • should we require a minimal Lmod 8.x version (8.2 or 8.3)
      • for extensions, depends_on, etc.
      • bump to REQ_VERSION to 8.0 (or 8.3), DEPR_VERSION to 8.5
framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - minor conflicts fixed (commit f378bac)
  • merged PRs:
    • PR #4250: enabling --trace by default (by Simon)
    • PR #4270 update check_python_version to reflect that only Python >= 3.6 is supported
    • stop relying on imp (will no longer be available in Python 3.12) (issue #4264)
  • open PRs:
    • PR #4284: initial progress by Kenneth, following discussion with Alex, on new run function (issue #4252)
      • feedback from Simon in PR
      • next steps:
        • expand tests run with new run function (in same PR?)
        • start using new run function in framework where possible (no unsupported options used) - follow-up PR?
        • start using new run in easyblocks
          • probably split across multiple PRs
          • with reasonabe effort regarding testing, not testing all easyconfigs affected...
        • implement missing features by expanding basic run function
        • support for setting environment variables in the scope of the command being run
        • implement support for dropping into a shell when a command fails (issue #3950)
          • should be configurable (globally + per run invocation)
            • shell_on_error=True?
          • only do this when exit on error is enabled (fail_on_error=True)
          • only do this when eb is run in an interactive shell
          • Mikael has looked into this a bit
          • use pty.spawn + something like cmd || bash -l ?
  • several PRs for code cleanup in 5.0.x branch, by removing deprecated stuff
    • PR #4272 remove support for 32-bit targets
    • PR #4273 remove deprecated desc option for simple_option
    • PR #4274 remove deprecated Toolchain.add_dependencies
    • PR #4275 remove deprecated functionality from filetools
    • PR #4276 remove deprecated log_error option from which function
    • PR #4277 remove deprecated skip_lower option from template_constant_dict
    • PR #4278 remove deprecated options from easyconfig.py
    • PR #4279 removed deprecated mod_exists_regex_template in ModulesTool.exist
  • proposals for EasyBuild 5.0:
    • rename unclear ExtensionEasyblock.run() method (issue #4113)
    • include entire flag in optarch (instead of excluding the initial dash) and maybe make it only per-compiler (issue #4269)
      • --optarch='march=x86-64 -mtune=generic' vs --optarch='-march=x86-64 -mtune=generic'
    • stop relying on distutils (https://github.com/easybuilders/easybuild-framework/issues/3963)
    • require Lmod >= 7.0 ?
    • eb options simplification
      • John: "Using eb to build software is the last step. eb has a large number of arguments that continue to grow."
        • separate ec command doesn't seem like a good idea
      • Simon: It might be possible to make a split using subparser, but I have no firm idea or suggestion here and I'm unlikely to spend time on this.
        • eb build, eb github, ...
        • unclear where all options would end up
        • significant effort
        • some options would apply to multiple subcommands, so would be "top-level" (like --debug)
        • opportunity to review current options and see if they're still needed and make sense
        • re-design of CLI may result in significant delay for EasyBuild 5.0
          • so maybe something for a "quick" EasyBuild 6.0 release?
        • to some extent this "complaint" can be tackled by defining eb* aliases
        • main issue is that output of eb --help is massive
        • maybe this is more about shortening of help output
          • eb --help=github, ...
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit f93e6cd)
  • no merged or WIP PRs for now
  • TODO
    • stop using deprecated py2vs3 module in easyblocks
    • start switching to new run function
    • consistently import LooseVersion from easybuild.tools instead of distutils.version in all easyblocks
    • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage
      • and then remove use_pip = True & co in all easyconfigs
    • update version of config.guess used by ConfigureMake (issue #2826)
      • important to test across a bunch of OSs & CPU architectures
        • Kenneth: aarch64 (Graviton* in AWS + Ampere Altra in Azure + A64FX in Isambard)
        • Kenneth: power9le (emulated)
        • Sebastian: riscv64
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit c0fd7db)
  • no merged or WIP PRs for now
  • TODO
    • archive easyconfigs using a deprecated toolchains + toolchain policy
    • archive ancient easyconfigs using system toolchain
    • archive directories that only have patches in them + cleaned up unused patches
    • remove use_pip = True & co in all easyconfigs

2023-06-07

Attendees: Bart, Sebastian, Kenneth, Jasper, Simon, Mikael, Alexandre

Pre-meeting notes

Discussion

  • GitHub Actions no longer support testing on top of Python 2.7 (cfr. framework PR #4267)

  • proposal on splitting up extensions for Python/Perl/R

    • prepped by Alex + Mikael
    • see https://hackmd.io/K1QhWGfdQWa2ZtlXlWVCCA
    • progress in context of 2023a common toolchains:
      • Perl/5.36.1-GCCcore-12.3.0 merged: (PR #17946)
        • includes only a handful of extensions, those required by Autoconf/Automake (Thread::Queue, Data::Dumper)
        • separate Perl-bundle-CPAN for additional Perl modules from CPAN: (PR #17970)
      • Python/3.11.3-GCCcore-12.3.0 merged (PR #17955)
        • only includes setuptools + pip + wheel (+ flit_core dep) as extensions
        • next step would be Python-bundle-PyPI (+ maybe separate easyconfigs for virtualenv, hatchling, poetry, etc.)
      • R: coming soon...

Done

docs
  • WIP PR #137 to include overview of changes in EasyBuild 5.0
framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - minor conflicts fixed (commit f378bac)
  • some progress on enabling --trace by default by Simon (PR #4250)
    • Simon: fixing the tests is painful, but we're very close, only 1 test is failing (unexpected output being generated)
    • could consider setting $EASYBUILD_DISABLE_TRACE in test setup?
  • very little progress by Kenneth on new run function (issue #4252)
    • may be useful to pair up with someone on this in the coming weeks?
  • proposals for EasyBuild 5.0:
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit f93e6cd)
  • no merged or WIP PRs for now
  • TODO
    • consistently import LooseVersion from easybuild.tools instead of distutils.version in all easyblocks
    • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage
      • and then remove use_pip = True & co in all easyconfigs
    • update version of config.guess used by ConfigureMake (issue #2826)
      • important to test across a bunch of OSs & CPU architectures
        • Kenneth: aarch64 (Graviton* in AWS + Ampere Altra in Azure + A64FX in Isambard)
        • Kenneth: power9le (emulated)
        • Sebastian: riscv64
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit c0fd7db)
  • no merged or WIP PRs for now
  • TODO
    • archive easyconfigs using a deprecated toolchains + toolchain policy
    • archive ancient easyconfigs using system toolchain
    • archive directories that only have patches in them + cleaned up unused patches
    • remove use_pip = True & co in all easyconfigs

TODO

⚠️ **GitHub.com Fallback** ⚠️