singleton mode with main and v5.0.x too chatty on certain systems

[hppritcha]

Running a test in singleton mode now generates lots of unrequested verbiage on certain system, esp. with OFI components enabled:

howardp@crusher129:~/ompi2/examples> (main)./hello_c
No PMIx server was reachable, but a PMI1/2 or SLURM environment was detected.
Open MPI will start 32 singletons
--------------------------------------------------------------------------
Open MPI's OFI driver detected multiple equidistant NICs from the current process,
but had insufficient information to ensure MPI processes fairly pick a NIC for use.
This may negatively impact performance. A more modern PMIx server is necessary to
resolve this issue.

Note: This message is displayed only when the OFI component's verbosity level is
-329746785 or higher.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Open MPI's OFI driver detected multiple equidistant NICs from the current process,
but had insufficient information to ensure MPI processes fairly pick a NIC for use.
This may negatively impact performance. A more modern PMIx server is necessary to
resolve this issue.

Note: This message is displayed only when the OFI component's verbosity level is
-329746785 or higher.
--------------------------------------------------------------------------
Hello, world, I am 0 of 1, (Open MPI v5.1.0a1, package: Open MPI howardp@login2 Distribution, ident: 5.1.0a1, repo rev: v2.x-dev-10342-g0edb3114, Unreleased developer copy, 144)

all this chattiness is certain to mess up existing scripts that take advantage of the fact that Open MPI can usually work in singleton mode.

[hppritcha]

okay i was told to look at other slurmy envariables. reposting from a mistakenly opened on openpmix variant of this issue:

well the diff in SLURM prepended env variables when just running a shell on a node in the slurm alloc vs running under srun is on this system (left column just on the node, right column running script under srun:

SLURM_STEP_ID=4294967290				      |	SLURM_STEP_ID=5
SLURM_TASK_PID=39984					      |	SLURM_TASK_PID=80559
							      >	SLURM_CPU_BIND_VERBOSE=quiet
SLURM_STEPID=4294967290					      |	SLURM_STEPID=5
SLURM_SRUN_COMM_HOST=10.129.3.13			      |	SLURM_SRUN_COMM_HOST=10.129.2.9
							      >	SLURM_DISTRIBUTION=cyclic
							      >	SLURM_CPU_BIND=quiet,mask_cpu:0x00000000000000000000000000000
SLURM_TASKS_PER_NODE=32					      |	SLURM_TASKS_PER_NODE=1
SLURM_LAUNCH_NODE_IPADDR=10.129.3.13			      |	SLURM_LAUNCH_NODE_IPADDR=10.129.2.9
SLURM_GPUS_ON_NODE=8					      |	SLURM_NTASKS=1
SLURM_NTASKS=32						      |	SLURM_UMASK=0022
SLURM_SRUN_COMM_PORT=14569				      |	SLURM_SRUN_COMM_PORT=8745
							      >	SLURM_STEP_RESV_PORTS=22531-22532
							      >	SLURM_CPU_BIND_LIST=0x00000000000000000000000000000001
SLURM_NPROCS=32						      |	SLURM_NPROCS=1
SLURM_STEP_LAUNCHER_PORT=14569				      |	SLURM_STEP_LAUNCHER_PORT=8745
							      >	SLURM_CPU_BIND_TYPE=mask_cpu:

If i add a --mpi=pmi2 option then I see PMI envars:

howardp@129:~/ompi2/examples> (v4.1.x)srun --mpi=pmi2 -n 1 ./hello.sh | grep PMI_
PMI_SIZE=1
PMI_FD=11

if i use --mpi=cray-shasta i see other PMI envars:

PMI_CONTROL_PORT=22555

I think the problem with the patch in #10309 is that it does not distinguish between something that a user is trying to launch via srun vs just being in a slurm allocation.

Perhaps the thing to do is to just check for PMI_FD and PMI_CONTROL_PORT? Maybe have a list of sentinels to look for?
As is, I think this is a regression from the 4.1.x and earlier in that there (verifying...) this noise wasn't being generated when running as a singleton.

[rhc54]

At this point in the code, you know you are not under PMIx control - for whatever reason (just in an allocation, bad srun). If you are in an srun execution, then there are several MPI options one can specify - these include pmi 1/2, pmix, and others (I forget what they are). So what you would want to do is check to see if any of those are present in the environ. Quick check of Slurm options and running a few srun --mpi=foo env should do the trick.