Meeting 2024 03 07 - openpmix/openpmix GitHub Wiki

March 7, 2024 OpenPMIx-devel call notes

Attendees

  • Ralph Castain
  • Howard Pritchard (LANL)
  • Tim Wickberg (SchedMD)
  • Michael Karo (Altair)
  • Thomas Naughton (ORNL)
  • Sam Gutierrez (LANL)
  • Aurelien Bouteiller (UTK)

Notes

  • Agenda: We will discuss the last month’s development efforts, and plans for the next few months as we prep for release of PMIx v6 and PRRTE v4.

  • Released openpmix 4.2.9 (early feb)

    • this is the end of the v4.2 series, the maintenance will be too difficult to continue down this branch
    • plan for all new fixes go into v5.0 series

  • Release openpmix 5.0.0 and 5.0.1

    • There is a bug in them that limits cross-version betwen 5.0.0 & 5.0.1 (just these two) other versions work.
    • v5.0.2 will not have this problem for cross-version
    • this v5.x will be used in future Open MPI release
    • dec-2024 for 5.0.1, so a 5.0.2 is at reasonable time
    • tentatively thinking openpmix 5.0.2 sometime next week, and a new prrte-3.0.5

  • prrte some bugfixes coming in slowly as getting more use, but very slow issue count

  • trying to keep the scheduler revisions out of the prrte/openpmix releases to decrease the probabily of new bugs coming in

  • Good to have testing on openpmi 5.0 and prrte 3.0 branches for testing

  • Group extension

    • Extended group capability to bootstrap groups more easily
    • Originally PMIx_Group_construct all members had to know all members
    • You can now add a "bootstrap" attribute, which state how many are going to call this group construct, e.g., 2, and then you can have the ability to "add members" that can be added to the group. So you would know exactly 2 in construct and then have others added afterward. So shifted much of the work down into the library for bootstrapping, and caller can just pass NULL to reduce complexity.
    • See documentation for more details.

  • Documenation

    • Migrating some text out of standard into the OpenPMIx ReadTheDocs information to allow for easy use of the docs w/o having to switch back and forth.

  • Dynamic Scheduling work

    • Will be bring this work to close in spring, adding documentation, etc
    • Intent is to lean on the preemption capabilities
    • Some early users of the dynamic work are identifying problems with "gaming" the system by getting a single node and then dynamically (at higher priorty) getting more nodes.
    • The preemption ends up having similar challenges
    • So trying to get PRRTE into state where the research community can work on these issues and investigate these areas
    • Some papers to come out from European research teams
    • Another related work "SUNK" (kubernetes+slurm) project may be of interest, and related to elasticity. Unsure what the availability/release of this work is currently.
    • SUNK: https://www.coreweave.com/blog/sunk-slurm-on-kubernetes-implementations
    • Note, mention of Slurm possibly starting to building some elastic capabilities.
    • Isais (european project) is also planning on bringing some of their code back into PMIx, and where best to put things This would include tools that go directly to the scheduler. Look for this in the coming months.

  • Error handling w/ Shared memory component

    • Note: Loop on MPI_Comm_spawn(), was able to hit this issue and was repeatable. Need to look more closely to figure out what is triggering the problem.
    • So some condition that is copying things and triggering.
    • Bug being at 50-75 times, eventually the pmix shm component raises an error. This is not a common behaviour (tight loop calling CommSpawn), but still worth investigation.