mpi_file_open fails on symlink with Lustre filesystem

[anton-seaice]

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

v4.1.4

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

From source tarball

Please describe the system on which you are running

• Operating system/version: Rocky Linux 8.8 (Green Obsidian)
• Computer hardware: Intel Xeon Scalable ‘Cascade Lake’
• Network type: ? (see https://nci.org.au/our-systems/hpc-systems)

---

Details of the problem

When I attempt to open a file through its symlink, it fails with the error :

get_stripe failed: 61 (No data available)
MPI_ERR_OTHER: known error not in list

The expected result is that the symlink to the file opens the same way the file does.

This is my minimal test example:

program mpi_file_open_example
  implicit none

  include 'mpif.h'
  
  integer, parameter :: nprocs = 4
  integer :: rank, ierr, file_handle, status, len 
  character(len=MPI_MAX_ERROR_STRING) :: error_string
  character(len=80) :: filename = "parallel_file.txt", link = "link_to_parallel_file.txt"

  ! Confirm the link can be opened using open
  open(newunit=file_handle, file=link, status="old", action="read", &
  iostat=ierr, iomsg=error_string)
  if (ierr /= 0) then
    print *, trim(error_string)
    call abort()
  end if
    close(file_handle)

  
  ! Initialize MPI
  call MPI_Init(ierr)
  call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)

  ! Open the file in parallel
  write(*, *) "Open the file in parallel: "
  call MPI_File_open(MPI_COMM_WORLD, filename, &
                     MPI_MODE_RDONLY, MPI_INFO_NULL, file_handle, ierr)
    
  if (ierr /= MPI_SUCCESS) then
    call MPI_Error_string(ierr, error_string, len, ierr)
    write(*, *) "Error opening ", trim(filename), " : ", trim(error_string)
  else
    write(*, *) "Success opening ", trim(filename)
  end if
  
  ! Close the file
  call MPI_File_close(file_handle, ierr)

  write(*, *) "==================="

  ! Open the symlink in parallel
  write(*, *) "Open the symlink in parallel: "
  call MPI_File_open(MPI_COMM_WORLD, link, &
                     MPI_MODE_RDONLY, MPI_INFO_NULL, file_handle, ierr)
    
  if (ierr /= MPI_SUCCESS) then
    call MPI_Error_string(ierr, error_string, len, ierr)
    write(*, *) "Error opening ", trim(link), " : ", trim(error_string)
  else
    write(*, *) "Success opening ", trim(link)
  end if

  ! Close the file
  call MPI_File_close(file_handle, ierr)

  ! Finalize MPI
  call MPI_Finalize(ierr)
  
end program mpi_file_open_example

to reproduce:

module purge
module load intel-compiler/2021.6.0 openmpi/4.1.4
mpifort mpi_file_open.f90 -o mpi_file_open.exe

touch parallel_file.txt
ln -s parallel_file.txt link_to_parallel_file.txt

./mpi_file_open.exe 

[edgargabriel]

@anton-seaice thank you for submitting the issue, I will try to look into this in the next few days

[anton-seaice]

ompi/ompi/mca/fs/lustre/fs_lustre_file_open.c

Line 157 in 8ddc997

rc = llapi_file_get_stripe(filename, lump);

is the troublesome line. I don't know if there needs to be some checking for a symlink before that line, or something else is going on.

[anton-seaice]

Hi @edgargabriel How did you go with this?

[edgargabriel]

@anton-seaice really sorry, I didn't get to it last week. It is on my todo list for this week however.

[benmenadue]

I did some testing: llapi_file_get_stripe doesn't work through symlinks. Or more precisely it tries to get the striping of the symlink itself, which naturally fails as they're metadata-only entities with no objects.

[edgargabriel]

Does it work btw. if you force using the romio component, e.g. adding --mca io romio321 to your mpirun command line?

[anton-seaice]

Does it work btw. if you force using the romio component, e.g. adding --mca io romio321 to your mpirun command line?

Thanks for the suggestion. That does work. In our model code though, the read/write timers report it being a lot slower (takes about twice as long). I will investigate some more :)

[benmenadue]

That works because romio opens the file itself and uses the LL_IOC_LOV_GETSTRIPE ioctl to get the striping info. The open follows the symlink and hence this works find with symlinks (and even with symlinks that aren't themselves on a Lustre filesystem but do target objects that are).

On the other hand, llapi_file_get_stripe opens the parent directory and uses the IOC_MDC_GETFILESTRIPE ioctl. Since the name of the object to query is passed as part of the ioctl, it bypasses the kernel's symlink handling and hence operates on the symlink itself instead of the target.

[edgargabriel]

@anton-seaice @benmenadue could you please test the fix in #12161 and let me know whether it resolves the issue?

I tested bits and pieces of it, but I do not have a working Lustre file system at the moment for testing the full patch.
If it works, I will backport the fix to the 4.1 and the 5.0 branches