Conference call notes 20180425 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 100th EasyBuild conference call, Wednesday April 25th 2018 (5pm - 6pm CET)

Attendees

Alphabetical list of attendees (10):

• Damian Alvarez (JSC, Germany)
• Pablo Escobar (UniBas/SciCORE, Switzerland)
• Fotis Georgatos (Illumina, UK)
• Kenneth Hoste (HPC-UGent, Belgium)
• Adam Huffman (Big Data Institute, University of Oxford)
• Bart Oldeman (ComputeCanada)
• Alan O'Cais (JSC, Germany)
• Åke Sandgren (Umeå University, Sweden)
• Shahzeb Siddiqui (Pfizer)
• Davide Vanzo (Vanderbilt University)

Agenda

• outlook to upcoming EasyBuild v3.6.0
• discussion on enabling automatic fallback to lib64 in sanity check by default (or not)
• feedback on including support in PythonPackage easyblock to detect auto-downloading of dependencies
• proposal by Jack to handle Java security updates using wrapper module
• Q&A

EasyBuild v3.6.0

• ETA: Fri Apr 27th 2018
• 263 merged PRs (39+16+208)
• release notes:
  • framework: https://github.com/easybuilders/easybuild-framework/pull/2485
  • easyblocks: https://github.com/easybuilders/easybuild-easyblocks/pull/1406
  • easyconfigs: https://github.com/easybuilders/easybuild-easyconfigs/pull/6232
• regression test in under way...
• will include initial integration with Singularity: creating container recipes & images
  • see http://easybuild.readthedocs.io/en/develop/Containers.html
  • currently several aspects are hardcoded
    • e.g. yum install only works on RHEL-based systems
  • no support yet for creating a base container image from scratch with EasyBuild
    • will be added in the future
    • for now, can use the examples provided by Shahzeb at https://singularity-hub.org/collections/143
  • base image has several requirements: easybuild user that can write to /scratch and /app in container, Lmod installed, ...
    • cfr. http://easybuild.readthedocs.io/en/develop/Containers.html#requirements-for-base-container-image

Enabling automatic fallback to lib64 in sanity check?

• cfr. https://github.com/easybuilders/easybuild-framework/pull/2477
• let sanity check fall back to checking for lib64/libfoo.a when hardcoded lib/libfoo.a was not found
• could/should be enhanced to work both ways (lib->lib64 and lib64->lib)
• no objections from EasyBuilders attending conf call
• should be OK to enable this by default
• Åke: should def be an opt-out option for it if enabled by default
  • there will be, e.g. --disable-lib64-fallback-sanity-check
• Markus: could only be an actual problem on a real multi-arch system (32/64-bit)
  • but unlikely to be very relevant nowadays?

Detecting auto-downloaded dependencies in PythonPackage easyblock

• cfr. https://github.com/easybuilders/easybuild-easyblocks/pull/1377
• implements detection of auto-downloaded dependencies of Python packages by pip or setup.py install
• opt-in: only when download_dep_fail is set to True in easyconfigs
  • default will remain False in EasyBuild 3.x, may change to True in 4.x
• we could make it a policy to have download_dep_fail enabled in new easyconfig files that installed Python packages
• Åke: should only error out at the end of installing all extensions, if possible...
  • Kenneth: not so trivial, since each extension is handled by PythonPackage in isolation of others
  • would require further changes to Python, Bundle easyblocks, maybe even to framework somehow...

Migrating to Java wrapper module to deal with security updates

• cfr. https://github.com/easybuilders/easybuild-easyconfigs/pull/5203
• change Java to a bundle that wraps around a specific JDK version
• allows to update JDK to most recent bugfix/security release by just changing a single module
• no objections from EasyBuilders attending the conf call, seems like a good approach
• Fotis: Java 1.7 vs 1.8 compatibility?
  • this approach still locks you in to either Java 1.7 or 1.8, just not to a particular bugfix release of it
• Kenneth will work with Jack on transitioning to this approach for Java in the central repository
• Åke: could also be useful in other contexts
  • Markus: e.g. OpenSSL (if you want to install it through EB rather than using the system OpenSSL)
  • Åke: maybe also CUDA/ cuDNN (or maybe not...)

Other

• rant from Fotis on guy jumping from a cliff, just to show off the nice environment he calling in from

  • if you're interested in the long story, see https://en.wikipedia.org/wiki/Theseus ;-)

• Markus: libdrm easyconfigs are missing a pkg-config build dep

  • only checked recent libdrm versions, should this be fixed in all easyconfigs?
    • unclear whether pkg-config is actually required in older versions...
  • Kenneth can check whether old libdrm versions need pkg-config too
  • we hope to catch these type of problems earlier on in the future, by testing easyconfig contributions in a (Singularity) container rather than directly on a production system
    • could also help with making EasyBuild more stable across different OSs

• Åke: SCons easyblock: to build SCons itself, or to build with SCons?

  • the generic SCons easyblock is for building with SCons
  • SCons itself is installed using PythonPackage easyblock
  • will need to use it for Scipion, a Cryo-EM framework