Conference call notes 20190220 - easybuilders/easybuild GitHub Wiki

(back to Conference calls)

Notes on the 119th EasyBuild conference call, Wednesday Feb 20th 2019 (17:00 - 18:00 CET)

Attendees

Alphabetical list of attendees (5):

• Fotis Georgatos (SDSC, Switzerland)
• Kenneth Hoste (HPC-UGent, Belgium)
• Mikael Öhman (Chalmers University of Technology, Sweden)
• Bart Oldeman (ComputeCanada)
• Davide Vanzo (Vanderbilt University, US)

Agenda

• update on next EasyBuild release (3.8.2 or 3.9.0)
• updates on porting of EasyBuild framework to support Python 3
• follow-up on moving Python to GCCcore
• Q&A

Update on next EasyBuild release

• ETA: early March
• framework
  • cfr. https://github.com/easybuilders/easybuild-framework/milestone/65?closed=1
  • bunch of (minor) bug fixes
  • TODO:
    • iterative build deps: https://github.com/easybuilders/easybuild-framework/pull/2741
    • Alan's PR: https://github.com/easybuilders/easybuild-framework/pull/2599
• easyblocks
  • https://github.com/easybuilders/easybuild-easyblocks/milestone/56
  • minor updates/fixes to existing easyblocks
  • new easyblock for TensorRT (+ enhancement for TensorFlow)
• easyconfigs
  • https://github.com/easybuilders/easybuild-easyconfigs/milestone/59
  • mostly large bunch of software updates + some new software
  • (mostly) Miguel's cleanup effort: ~150 open PRs down!
  • large surge of incoming PRs by Åke

Follow-up on moving Python to GCCcore + multi-Python installations of Python packages

• cfr. https://github.com/easybuilders/easybuild-easyconfigs/issues/7463
• Python 2.7.15 w/ GCCcore + SciPy with foss/2019a
• Mikael: deap may require numpy?
• SciPy with intel/2019a doesn't work straight away
  • numpy doesn't find compiler?
  • Davide: same issue as colleague reported?
    • cfr. https://github.com/easybuilders/easybuild-easyconfigs/issues/7552
• Mikael: alias of Python/2.7.15-foss-2019a to SciPy/* doesn't work
  • alias is tied to software name for which it's created
  • only way out is to rename Python@GCCcore to PythonCore :(
    • would require fix in framework like for GCC(core)
  • check with Lmod maintainer whether cross-name aliases can work?
  • is this a blocker w.r.t. user experience?
• multi-Python support is still in the works (cfr. Bart's PR in framework)

Python 3 porting

• current status 97% of EasyBuild framework of tests pass (22 out of 590 tests)
  • cfr. https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Apr+label%3Apython3+is%3Aclosed
  • done in small sprints to make progress with limited efforts + make PRs easy to review/merge
  • remaining failing tests are expected to require a bit more effort to fix
  • work on easyblocks is expected to be (very) limited, nothing much to do for easyconfigs?

Other

• Mikael: moving down software to GCCcore or GCC/iccifort level

  • based on performance benchmarking "home work" (is GCCcore a sensible option?)
  • should things be moved down from intel to iccifort (similar for foss to GCC)? What's the benefit?
  • Bart: moving down to compiler-level done religiously at ComputeCanada if it doesn't need MPI/BLAS/LAPACK
    • MKL is also pulled down to GCCcore level (so more stuff can be pulled down to compiler-level because of that)
    • Bart: examples of software that relies on parallel FFT?
    • Davide: downside is breaking symmetry between foss/intel wrt where FFTW lives
  • Bart: main benefit is that these packages do not need to be recompiled for different MPI libraries/versions
  • Davide: stuff is being pulled down to GCC/iccifort in own repo
  • Davide: floating-point accuracy could also be affected for math libraries by using different compilers
  • Kenneth: should we start tracking what toolchain capabilities are required per software?
    • that would open possibility to let Travis fail if packages are added with top-level toolchain...
    • Davide: toolchain being picked is mostly done based on existing easyconfigs
    • maybe useful for being more religious on this starting with 2019a?
  • Davide: compiler+MPI level is often overlooked right now (gompi, iimpi)
    • Mikael: little packages in this cat?
    • Bart: Valgrind, ParMETIS, Boost, Ray, CGNS, RaxML, ...

• Davide: use of --force-download when testing PRs

  • should have a bypass in case there are no source URLs listed?
  • makes sense if it makes your life easier...