Registration: Bare Minimum - openpmix/openpmix GitHub Wiki

Some programming model implementations (e.g., OpenMPI) are able to execute with a subset of the information required by the PMIx Standard. It should be noted that execution does not equate to optimized performance - it only means that the application will have enough information to allow it to operate. This list, therefore, is provided to assist RM implementers seeking to initially stand up a limited integration.

The information listed below cannot be accurate across all programming model implementations - such a survey would be difficult to conduct. It should, therefore, be considered as guidance that hopefully will provide a reasonably close starting point.

Session-realm information

  • PMIX_UNIV_SIZE: Maximum number of processes that can be simultaneously executing in a session. Typically, this is a constraint imposed by a scheduler or by user settings in a hostfile or other resource description.

Job-realm information

  • PMIX_NSPACE: Namespace of the job - may be a numerical value expressed as a string, but is often an alphanumeric string carrying information solely of use to the system. Required to be unique within the scope of the host environment. Identifies the namespace of the job being registered.

  • PMIX_JOBID: Job identifier assigned by the scheduler to the specified job - may be identical to the namespace, but is often a numerical value expressed as a string (e.g., "12345.3").

  • PMIX_JOB_SIZE: Total number of processes in the specified job across all contained applications. Note that this value can be different from PMIX_MAX_PROCS. For example, users may choose to subdivide an allocation (running several jobs in parallel within it), and dynamic programming models may support adding and removing processes from a running job on-the-fly. In the latter case, PMIx events may be used to notify processes within the job that the job size has changed.

  • PMIX_JOB_NUM_APPS: Number of applications in the job. This is a required attribute if more than one application is included in the job - it may otherwise be omitted.

  • PMIX_WDIR: Working directory for spawned processes. This attribute is required for all registrations, but may be provided as an individual pmix_info_t entry if only one application is included in the namespace.

  • PMIX_CREDENTIAL: Blob containing security key for communications across fabrics requiring secure messaging.

Node-realm information

Node-realm information may be passed as individual pmix_info_t entries if only one node will host processes from the job being registered, or as part of a pmix_data_array_t using the PMIX_NODE_INFO_ARRAY attribute when multiple nodes are involved in the job. The list of data referenced in this way shall include:

  • PMIX_LOCAL_SIZE: Number of processes in the specified job on the node.

  • PMIX_TMPDIR: Full path to the top-level temporary directory assigned to the session

  • PMIX_NSDIR: Full path to the temporary directory assigned to the specified job, under PMIX_TMPDIR.

  • PMIX_LOCAL_PEERS: Comma-delimited list of ranks that are executing on the node within the specified namespace

  • PMIX_NODE_OVERSUBSCRIBED: True if the number of processes from this job on the node exceeds the number of slots allocated to it

  • PMIX_NODEID: Node identifier expressed as the node’s index (beginning at zero) in an array of nodes within the active session. The value must be unique and directly correlate to the PMIX_HOSTNAME of the node

  • PMIX_HOSTNAME: Name of the host, as returned by the gethostname utility or its equivalent. As this information is not related to the namespace, it can be passed using the PMIx_server_register_resources API.

  • PMIX_HOSTNAME_ALIASES: Comma-delimited list of names by which the target node is known. As this information is not related to the namespace, it is best passed using the PMIx_server_register_resources API.

  • PMIX_TOPOLOGY_SIGNATURE: Topology signature string for the local node

Application-realm information

  • PMIX_MODEL_LIBRARY_NAME: Programming model implementation ID (e.g., “ompi” or “mpich”). Note that this information may be required for proper execution by various programming model implementations.

Proc-realm information

Process-realm information shall include the following data for each process in the job, passed as a pmix_data_array_t using the PMIX_PROC_INFO_ARRAY attribute, with the process rank being the first element of that array:

  • PMIX_RANK: Process rank within the job, starting from zero

  • PMIX_NODEID: Node identifier indicating where the proc is executing, expressed as the node’s index (beginning at zero) in an array of nodes within the active session.

  • PMIX_HOSTNAME: Name of the host where this proc is executing, as returned by the gethostname utility or its equivalent.

  • PMIX_LOCAL_RANK: Rank of the specified process on its node - refers to the numerical location (starting from zero) of the process on its node when counting only those processes from the same job that share the node, ordered by their overall rank within that job.

  • PMIX_NODE_RANK: Rank of the specified process on its node spanning all jobs- refers to the numerical location (starting from zero) of the process on its node when counting all processes (regardless of job) that share the node, ordered by their overall rank within the job. The value represents a snapshot in time when the specified process was started on its node and is not dynamically adjusted as processes from other jobs are started or terminated on the node.

  • PMIX_APPNUM: The application number within the job in which the specified process is a member. This attribute may be omitted if only one application is present in the namespace.

  • PMIX_PROCDIR: Full path to the subdirectory under PMIX_NSDIR assigned to the specified process.

  • PMIX_PACKAGE_RANK: Rank of the specified process on the package where this process resides - refers to the numerical location (starting from zero) of the process on its package when counting only those processes from the same job that share the package, ordered by their overall rank within that job. Note that processes that are not bound to PUs within a single specific package cannot have a package rank.

  • PMIX_CPUSET: A string representation of the PU binding bitmap applied to the process upon launch. The string shall begin with the name of the library that generated it (e.g., "hwloc") followed by a colon and the bitmap string itself, as returned by the PMIx_server_generate_cpuset_string API.

  • PMIX_LOCALITY_STRING: String describing the process’ bound location. The string is prefixed by the implementation that created it (e.g., "hwloc") followed by a colon. The remainder of the string represents the corresponding locality as expressed by the underlying implementation. The PMIx_server_generate_locality_string API can be used to create the string in the correct format.

Optional information