Stage 2: Spawning the local processes - openpmix/openpmix GitHub Wiki
Prior to spawning any local processes, each RM daemon registers the application and local processes with its internal PMIx server, providing it with all relevant information pertaining to the application and local environment. Host environments are required to provide a wide range of session-, job-, application-, node-, and process-realm information when registering the job itself via the PMIx_server_register_nspace API, and may choose to provide a similarly wide range of optional information. A complete list of the required and optional information is provided in the PMIx Standard - see Section 16.2.3. A synopsis can be found in the following pages:
There are a number of important values that are to be passed during the nspace registration call. However, one that is sometimes overlooked and yet is of significant importance is the PMIX_MODEL_LIBRARY_NAME. Many launchers include an option by which the user can declare their programming model/library - e.g., "--mpi=mpich" or "--oshmem=sos". In addition to its fabric and GPU vendor plugins, many library teams have included plugins to support their model and/or implementation. Thus, including the PMIX_MODEL_LIBRARY_NAME attribute during nspace registration allows the PMIx server library to utilize the corresponding plugin to customize the environment for the associated application processes, providing each with the library/implementation-specific values it needs.
In addition to registering the job, the host must register each local application process via the PMIx_server_register_client API prior to spawning the processes. This allows the PMIx server to create proc-specific internal tracking logic and prepare for accepting a connection request from the client once spawned. Note that the registration of clients can occur prior to nspace registration - no particular ordering between these two operations is required.
At some point after registration, the RM daemon is ready to spawn its local processes. Prior to spawning each child process, the daemon should call the PMIx_server_setup_fork API. This provides the PMIx server library with an opportunity to prepare the environment for this specific process based on the information provided to it during registration and application setup. This includes the passing of any network security keys and endpoint assignments that were made, any programming model/library environmental parameters, and environmental parameters specifying the method and protocol to be used when connecting back to the host daemon.
This completes the launch sequence.