Skip to content

HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus #2113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 13, 2020
Merged

HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus #2113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3977,6 +3977,8 @@ public boolean isMagicCommitPath(Path path) {

/**
* Increments the statistic {@link Statistic#INVOCATION_GLOB_STATUS}.
* Override superclass so as to disable symlink resolution as symlinks
* are not supported by S3A.
* {@inheritDoc}
*/
@Override
Expand All @@ -3985,9 +3987,9 @@ public FileStatus[] globStatus(Path pathPattern) throws IOException {
}

/**
* Override superclass so as to disable symlink resolution and so avoid
* some calls to the FS which may have problems when the store is being
* inconsistent.
* Increments the statistic {@link Statistic#INVOCATION_GLOB_STATUS}.
* Override superclass so as to disable symlink resolution as symlinks
* are not supported by S3A.
* {@inheritDoc}
*/
@Override
Expand All @@ -3999,7 +4001,7 @@ public FileStatus[] globStatus(
return Globber.createGlobber(this)
.withPathPattern(pathPattern)
.withPathFiltern(filter)
.withResolveSymlinks(true)
.withResolveSymlinks(false)
.build()
.glob();
}
Expand Down
44 changes: 44 additions & 0 deletions ...op-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -574,4 +574,48 @@ public void testCreateCost() throws Throwable {
}

}

@Test
public void testCostOfGlobStatus() throws Throwable {
describe("Test globStatus has expected cost");
S3AFileSystem fs = getFileSystem();
assume("Unguarded FS only", !fs.hasMetadataStore());

Path basePath = path("testCostOfGlobStatus/nextFolder/");

// create a bunch of files
int filesToCreate = 10;
for (int i = 0; i < filesToCreate; i++) {
try (FSDataOutputStream out = fs.create(basePath.suffix("/" + i))) {
verifyOperationCount(1, 1);
}
}

fs.globStatus(basePath.suffix("/*"));
// 2 head + 1 list from getFileStatus on path,
// plus 1 list to match the glob pattern
verifyOperationCount(2, 2);
}

@Test
public void testCostOfGlobStatusNoSymlinkResolution() throws Throwable {
Copy link
Contributor

@mukund-thakur mukund-thakur Jul 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both these tests look almost the same in terms operation counts and also the symlink resolution is always disabled.
Can you please tell me why do we need two tests here? Thanks.

Copy link
Contributor Author

@jimmy-zuber-amzn jimmy-zuber-amzn Jul 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So these tests two different things, a directory with multiple objects and a directory with one object. The directory with a single object is the special case that triggers attempted symlink resolution, so I wanted to carve out that special case in its own test.

The multiple-objects-in-a-directory test is a general test that it felt like globStatus should have, whereas the second one was specifically made to catch the regression.

If you don't think that the multiple objects test is justified, I can remove it. Thoughts?

Copy link
Contributor

@mukund-thakur mukund-thakur Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got it it.

hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java

Line 292 in f77bbc2

if (children.length == 1) {

Thanks.

describe("Test globStatus does not attempt to resolve symlinks");
S3AFileSystem fs = getFileSystem();
assume("Unguarded FS only", !fs.hasMetadataStore());

Path basePath = path("testCostOfGlobStatusNoSymlinkResolution/f/");

// create a single file, globStatus returning a single file on a pattern
// triggers attempts at symlinks resolution if configured
String fileName = "/notASymlinkDOntResolveMeLikeOne";
try (FSDataOutputStream out = fs.create(basePath.suffix(fileName))) {
verifyOperationCount(1, 1);
}

fs.globStatus(basePath.suffix("/*"));
// unguarded: 2 head + 1 list from getFileStatus on path,
// plus 1 list to match the glob pattern
// no additional operations from symlink resolution
verifyOperationCount(2, 2);
}
}