The `AbstractPersistentAcceptOnceFileListFilter` does not deal with full file name

[karim789]

In what version(s) of Spring Integration are you seeing this issue?

6.5.0

Describe the bug

SftpClient.DirEntry do not hold the full path of the file.
ScanDirEntry does.

To Reproduce

N/A

Expected behavior

N/A

Sample

N/A

[artembilan]

The ScanDirEntry is a part of SftpClientDirectoryScanner. I don't think we can use that API here.
At the same time I see this in the SftpClient.DirEntry:

public DirEntry(String filename, String longFilename, Attributes attributes) {

Where it does not look like we utilize it properly.
We might come up with our own DirEntry extension where we would populate that longFilename from the SftpSession.list().
Looks like Apache MINA does that only for SFTP v3:

                    String name = getReferencedName(cmd, buffer, nameIndex.getAndIncrement());
                    String longName = null;
                    if (version == SftpConstants.SFTP_V3) {
                        longName = getReferencedName(cmd, buffer, nameIndex.getAndIncrement());
                    }

The SmbPersistentAcceptOnceFileListFilter could simply be fixed like this:

	protected String fileName(SmbFile file) {
		return file.getUncPath();
	}

The FtpPersistentAcceptOnceFileListFilter one is a bit harder because doesn't look like there is anything we can hook for a "long file name".
We might come up with a FTPFile extension with a delegation to original file and extra property we populated in the FtpSession.list().
Then we check for instanceof in the FtpPersistentAcceptOnceFileListFilter to take long name or short.

While it might feel as a bug I don't think we can back-port such a fix into previous versions.
Imaging you use persistent stores with some entries populated.
Now you upgrade to a new point version it has this new logic for full file name.
So, those existing entries in the store won't be valid any more.

Let me know if this makes sense!

[karim789]

The ScanDirEntry is a part of SftpClientDirectoryScanner. I don't think we can use that API here.

Why ?

ScanDirEntry is already an extension of DirEntry, and it contains all the needed infos.
I don't really see why DirEntry doesn't have a referene to a parent, like File or Path would have.

I have overriden SftpSession.doList( ) and used a SftpClientDirectoryScanner

        final SftpClientDirectoryScanner sftpClientDirectoryScanner = new SftpClientDirectoryScanner(remoteDir, "*");
        
        return sftpClientDirectoryScanner.scan(this.sftpClient)....

But also instead of using a specific implementation, you could make your own dir entry object, or even use a java.nio.Path interface and implementation ?

[artembilan]

I have overriden SftpSession.doList( ) and used a SftpClientDirectoryScanner

That is a big change in the behavior and it is really not what we would aim for.
You see this SftpClientDirectoryScanner does a recursive walking.
Is it really what we would like to have here unconditionally?
And since it is a bit awkward to bring an API not related to our scope, there I don't see a reasoning to rely on that internal ScanDirEntry type.

[karim789]

At first I used a wrapper object that would hold DirEntry and the parent path.

I didn't really realised that scandir was recursive, thanks for pointing that out, although I think recursivity is a missing feature, and would be a great option.

[artembilan]

I think recursivity is a missing feature, and would be a great option.

Correct: #3407

[artembilan]

To summarize the issue.
The AbstractPersistentAcceptOnceFileListFilter may cause a problem with entries in the MetadataStore when same file name is used from different directories.
Let's imaging a scenario when remote directories are create every day with respective timestamp, e.g. today 20250828, tomorrow 20250829. The files in those directories are based on customer names: customer1.xml, customer2.xml etc.
Since most of the AbstractPersistentAcceptOnceFileListFilter implementations do just something this:

return file.getName();

That would lead to the same key for the MetadataStore operations.

The idea of this issue is to expose as much as possible information about the whole path of the remote file to be used as a key in AbstractPersistentAcceptOnceFileListFilter.