Error when writing file on more than one node

[romainhild]

I use Open MPI v4.0.2 from the tar.gz on a cluster running Ubuntu 16.04 with 6 compute nodes (24 cores each). The jobs are managed by slurm.

I have the following error when I try to write some files on disk when running on multiple nodes:

[atlas6:242817] mca_sharedfp_individual_file_open: Error during datafile file open

This error is displayed for each proc of one of the two nodes I use, not the other one.

The problem is that it does not happen all the times I launch the same program.
I could not find a way to reproduce the issue consistently.

Is there anyone that have an idea about this ?

[edgargabriel]

what filesystem are you using, and would it be possible to provide a reproducer for this problem?

[romainhild]

the compute nodes are in ext4 and where I'm trying to write is in nfs.
And I'm sorry but I'm still trying to find a way to reproduce in a consistent way the problem, as I said, it does not happen every time...

[edgargabriel]

is your code actually using shared file pointer operations? If not, it might be easiest to tell Open MPI to not try to initialize a sharedfp component at all

[romainhild]

We use HDF5 to export some fields and we don't know exactly what is done by it.
But otherwise, I don't think we use them.

How can I disable the sharedfp component ?

[prudhomm]

@edgargabriel we don't use shared file pointers anymore only explicit offsets version for MPI IO. is there a way to disable the sharedfp component (individual)?

[edgargabriel]

@prudhomm it turns out its not that simple. The individual sharedfp component is only loaded, if the other sharedfp components disqualify themselves, i.e. its the last resort. Now, in the 3.x series ompi used to be able to operate without a sharedfp component, but that is not the case in the 4.x series. So if we would disable the individual component, your application would segfault.

I am currently looking into this, from both aspects:

1. why does the sharedfp_individual_file_open raise an error ( which I can not yet reproduce), and
2. whether its possible to fix the sequence of running without a sharedfp component.

Will try to keep you posted, I hope to have a fix for at least one of the two items very soon.

[edgargabriel]

actually, it might be worth a trial: can you see whether adding the following parameter to your command line fixes the issue:

mpirun --mca sharedfp ^individual

[prudhomm]

Thanks a lot. We will test asap with @romainhild

[edgargabriel]

@prudhomm @romainhild I looked in detail into the sharedfp_individual_file_open routine, and under which conditions it can raise the error that was listed. I am 99% sure that the only reason would be if it really cannot create a file on the file system, which under NFS is maybe not entirely impossible (each process creates two file underneath the hood in this component). I was able to reproduce it on NFS in approx. 1 out of 100 runs on my system.

I have a fix for that, namely that the individual component will not raise an error during execution if this happens, but will raise an error if the user subsequently tries to use an write_shared or similar operations. I am about to file a pr for that fix, I can not tell however whether it will make it into the next release of the 4.0 series, since that is due potenitally

[prudhomm]

There is something strange in our case, it is easier to get a case for some applications while for others we don't think we ever got a case.
All the apps follow similar execution patterns and share the same code base.