Conference call notes 20171025 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 88th EasyBuild conference call, Wednesday October 25th 2017 (5pm - 6pm CEST)

Attendees

Alphabetical list of attendees (8):

• Damian Alvarez (JSC, Germany)
• Pablo Escobar (UniBas, Switzerland)
• Markus Geimer (JSC, Germany)
• Fotis Georgatos (Illumina, UK)
• Kenneth Hoste (HPC-UGent, Belgium)
• Adam Huffman (Big Data Institute, University of Oxford)
• Shahzeb Siddiqui (Pfizer)
• Davide Vanzo (Vanderbilt University)

Agenda

• Åke's PR to support use of an environment variable together with --subdir-user-modules
• Miguel's PR to add support for --preview-pr
• EasyBuild/Singularity integration
  • first step would be to let EasyBuild generate a Singularity .def file that can be used to create a Singularity image with
  • cfr. https://github.com/easybuilders/easybuild-framework/pull/2332
• Q&A

Notes

Support use of environment variable(s) in --subdir-user-modules

• https://github.com/easybuilders/easybuild-framework/pull/2330
• original idea was to add additional --subdir-arch-env option (which is kind of a misnomer too)
  • but may be too limiting: only one env var, no flexibility w.r.t. ordering of subdir & env var
• other idea was to parse value passed to --subdir-user-modules
  • e.g. Public/easybuild/modules/::ENV::SYSTEM_ARCH::
  • ::ENV::SYSTEM_ARCH:: gets replaced by Tcl or Lua code in module to grab $SYSTEM_ARCH
• better format may be --subdir-user-modules='Public/easybuild/modules/${SYSTEM_ARCH}'
  • more natural syntax, easier to understand
  • only downside is that escaping is required to avoid ${SYSTEM_ARCH} from being resolved too soon...
  • (what about $SYSTEM_ARCH? disallow use of $ without ${...}?)
• in environment: export EASYBUILD_SUBDIR_USER_MODULES='Public/easybuild/modules/${SYSTEM_ARCH}'
• in configuration file: subdir-user-modules = Public/easybuild/modules/${SYSTEM_ARCH}
  • we may also support use of environment variables that are resolved in EasyBuild configuration files
  • would require escaping of ${SYSTEM_ARCH} to maybe $${SYSTEM_ARCH}?

Support for --preview-pr

• great idea by Miguel, cfr. https://github.com/easybuilders/easybuild-framework/pull/2331
• equivalent to --review-pr, but for PRs that haven't been opened yet
  • cfr. docs @ http://easybuild.readthedocs.io/en/latest/Contributing.html#contributing-review-process-review-pr
  • demo for --review-pr @ http://easybuild.readthedocs.io/en/latest/demos/review_pr.html#demo-review-pr

EasyBuild/Singularity integration

• ground work to support generating of Docker/Singularity recipes/images via --package @ https://github.com/easybuilders/easybuild-framework/pull/2263
  • --package makes sense if container images would be spit out, less so for recipes...
• see also Shahzeb's recent PR
  • https://github.com/easybuilders/easybuild-framework/pull/2332
• https://github.com/easybuilders/eb-singularity
  • could be linked to Singularity Hub
• see Shahzeb's EasyBuild container image at http://www.singularity-hub.org/collections/133
• overlaying containers on top of each other could be done via OverlayFS, cfr. work by Jordi Blasco (HPCNow!)
• new APP support in Singularity 2.4 is also interesting
  • cfr. http://singularity.lbl.gov/docs-scif-apps
  • example: https://github.com/singularityware/singularity/blob/development/examples/apps/Singularity.cowsay
• first approach should be to let eb generate clean definition files
  • preferably stand-alone?
• later:
  • trim down size of images by cleaning up build deps, build path outside of container, ...
  • speed up building of images
    • e.g. by layering images on top of each other; e.g. base container per toolchain
    • or by pulling in dependencies via some other way (e.g. RPMs created via --package)
  • some type of check whether container is compatible with host architecture (AVX2, etc.)
    • could be part of Singularity itself?
    • Intel compilers include a check when using -xHost
    • or built generic binary with different code paths per architecture with -axHost
      • optimised for both host architecture with fallback for generic architecture
    • see also FMV: https://github.com/easybuilders/easybuild-framework/issues/2309
    • sanity check could be done based on detected host architecture vs location of where software was installed
      • could signal incompatibility between system on which image was built and host, to warn about potential crash or loss of performance
      • cfr. https://bitbucket.org/hpcnow/snow-tools/src/886787f00679bcfd76de4a5d61ca4ac2bcd6cc67/etc/cpu-id-map.conf?at=master&fileviewer=file-view-default, https://bitbucket.org/hpcnow/snow-tools/src/886787f00679bcfd76de4a5d61ca4ac2bcd6cc67/bin/easybuild-source.sh?at=master&fileviewer=file-view-default

Other

• EasyBuild User Meeting?
  • week after FOSDEM'18 (Feb 6-8 2018?)
  • location to be determined, currently considering Amsterdam...
    • potential interested parties: SurfSARA, Bright Computing, NLUUG (Dutch Linux User Group), X/OS