Skip to content

HADOOP-16085 S3Guard versioning (stevel patch 002) #807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

HADOOP-16085 S3Guard versioning (stevel patch 002) #807

wants to merge 5 commits into from

Conversation

@steveloughran
Copy link
Contributor

@steveloughran steveloughran commented May 9, 2019 •
edited
Loading

-move CopyOutcome into fs.s3a.impl package
-add new (failing) test for directory renaming where one of the files underneath is eventually consistent.
-pull out commonanity from test methods (spyOnFilesystem(), expectReadFailure())
-intercept() calls provide details on failure, primarily just by returning the result of the operation expected to fail.
-ChangeDetectionPolicy-raised exceptions on copy failures don't include position any more
-cache/restore metastore so even on test failures mockito is involved on teardown

Ben Roling and others added 5 commits May 2, 2019 12:30
Squashed commit of the following:
f726901 
commit ae876ab2df46c68ddd923edf8dd1d314191fcc94
Merge: 2e0254e 6a42745
Author: Ben Roling <[email protected]>
Date:   Thu May 2 10:14:10 2019 -0500

    Merge branch 'trunk' into HADOOP-16085-squashed-2

commit 2e0254e
Author: Ben Roling <[email protected]>
Date:   Thu Apr 18 12:13:40 2019 -0500

    Remove unused import

commit d1275e4
Merge: 450ba66 df76cdc
Author: Ben Roling <[email protected]>
Date:   Thu Apr 18 12:10:01 2019 -0500

    Merge branch 'trunk' into HADOOP-16085-squashed

commit 450ba66
Author: Ben Roling <[email protected]>
Date:   Thu Apr 18 11:45:41 2019 -0500

    Improvements to TestObjectChangeDetectionAttributes, AbstractS3AMockTest

commit 408af6c
Author: Ben Roling <[email protected]>
Date:   Thu Apr 18 10:29:05 2019 -0500

    Use HttpStatus code constant instead of magic number

commit 5f0532b
Author: Ben Roling <[email protected]>
Date:   Thu Apr 18 10:02:50 2019 -0500

    Update core-default.xml

commit 3488b20
Author: Ben Roling <[email protected]>
Date:   Wed Apr 17 16:14:43 2019 -0500

    Fix runaround of creating FileStatus and then calling fromFileStatus()

commit 90d5c9c
Author: Ben Roling <[email protected]>
Date:   Wed Apr 17 15:45:51 2019 -0500

    Fix minor nits

commit 3ff59e4
Author: Ben Roling <[email protected]>
Date:   Wed Apr 17 15:07:02 2019 -0500

    Mutate S3AFileStatus instead of creating new instance

commit 13fab97
Author: Ben Roling <[email protected]>
Date:   Wed Apr 17 14:30:55 2019 -0500

    Rename S3LocatedFileStatus to S3ALocatedFileStatus

commit bee4e52
Author: Ben Roling <[email protected]>
Date:   Wed Apr 17 14:25:17 2019 -0500

    Stop pretending to support group and permission attributes on S3AFileStatus

commit 807e13b
Author: Ben Roling <[email protected]>
Date:   Wed Apr 17 14:20:14 2019 -0500

    Add serialVersionUID to S3LocatedFileStatus

commit 9974cec
Author: Ben Roling <[email protected]>
Date:   Mon Apr 8 13:34:38 2019 -0500

    Fix missed group or owner tweak

commit 708c001
Author: Ben Roling <[email protected]>
Date:   Mon Apr 8 12:58:14 2019 -0500

    Fix S3AFileStatus group handling

    ITestS3AConfiguration.testUsernameFromUGI was failing, expecting the
    user to be copied into the group.

    Strict copying of user into group causes
    TestLocalMetadataStore.testPutNew() to fail since it expects the group
    to be preserved from the original FileStatus.

    This change copies user into group when group is null/empty. With this
    change, all existing tests pass.

commit 5239a9f
Author: Ben Roling <[email protected]>
Date:   Thu Apr 4 16:38:31 2019 -0500

    Skip tests that require versionId when bucket doesn't have versioning enabled

commit 4c6331e
Author: Ben Roling <[email protected]>
Date:   Mon Apr 1 13:58:24 2019 -0500

    Fix broken TestObjectChangeDetectionAttributes

commit 8a19c42
Author: Ben Roling <[email protected]>
Date:   Mon Apr 1 10:08:33 2019 -0500

    Squashed commit of the following:

    commit 9f4ad88
    Author: Ben Roling <[email protected]>
    Date:   Mon Apr 1 09:29:35 2019 -0500

        Add test for 412 response

    commit dc0a3fb
    Author: Ben Roling <[email protected]>
    Date:   Thu Mar 28 16:53:46 2019 -0500

        Update tests that started failing due to HADOOP-15999

    commit 5e1f3e3
    Author: Ben Roling <[email protected]>
    Date:   Thu Mar 28 15:49:26 2019 -0500

        Speed up ITestS3ARemoteFileChanged

    commit 1b6be40
    Author: Ben Roling <[email protected]>
    Date:   Thu Mar 28 14:23:53 2019 -0500

        Skip invalid test when object versioning enabled

    commit 8597d2e
    Merge: 2d235f8 b5db238
    Author: Ben Roling <[email protected]>
    Date:   Thu Mar 28 11:54:50 2019 -0500

        Merge remote-tracking branch 'apache/trunk' into HADOOP-16085

    commit 2d235f8
    Author: Ben Roling <[email protected]>
    Date:   Thu Mar 28 11:51:26 2019 -0500

        Fix typo

    commit dc83cef
    Author: Ben Roling <[email protected]>
    Date:   Thu Mar 28 10:28:09 2019 -0500

        Generalize TestObjectETag to cover versionId and test overwrite

    commit 0d71f32
    Author: Ben Roling <[email protected]>
    Date:   Thu Mar 28 08:45:42 2019 -0500

        Fix trailing whitespace

    commit 324be6d
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 22:00:57 2019 -0500

        S3GuardTool updates to correct ETag or versionId metadata

    commit 2a2bba7
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 21:27:27 2019 -0500

        Clarify log message

    commit 6e62a3a
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 21:17:48 2019 -0500

        Documentation updates per PR feedback

    commit 1ff8bef
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 16:05:59 2019 -0500

        check version.required on CopyResult

    commit e296275
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 16:04:50 2019 -0500

        Minor javadoc improvements from PR review

    commit 3e9ea19
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 13:15:58 2019 -0500

        Skip tests that aren't applicable with change.detection.source=versionId

    commit ddbf68b
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 11:56:38 2019 -0500

        Add tests of case where no version metadata is present

    commit 21d37dd
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 09:25:46 2019 -0500

        Fix compiler deprecation warning

    commit b8e1569
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 09:19:46 2019 -0500

        Fix license issue

    commit 33bb5f9
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 09:19:32 2019 -0500

        Fix findbugs issue

    commit 5b7fadb
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 09:00:39 2019 -0500

        Fix checkstyle issues

    commit 6110a11
    Author: Ben Roling <[email protected]>
    Date:   Wed Mar 27 08:28:37 2019 -0500

        Remove trailing whitespace

    commit d82069b
    Author: Ben Roling <[email protected]>
    Date:   Tue Mar 26 16:05:01 2019 -0500

        Improve S3Guard doc

    commit ca2f0e9
    Author: Ben Roling <[email protected]>
    Date:   Tue Mar 26 14:29:03 2019 -0500

        Fix ITestS3ARemoteFileChanged

    commit 1e4fa85
    Author: Ben Roling <[email protected]>
    Date:   Tue Mar 26 11:37:48 2019 -0500

        Increase local metastore cache timeout

    commit 34b0c80
    Author: Ben Roling <[email protected]>
    Date:   Tue Mar 26 11:35:34 2019 -0500

        Fix isEmptyDir inconsistency

    commit bbf8365
    Author: Ben Roling <[email protected]>
    Date:   Mon Mar 25 16:55:24 2019 -0500

        TestPathMetadataDynamoDBTranslation tests null etag, versonId

    commit 2ae7d16
    Author: Ben Roling <[email protected]>
    Date:   Mon Mar 25 16:54:49 2019 -0500

        Add constants in TestDirListingMetadata

    commit 068a55d
    Author: Ben Roling <[email protected]>
    Date:   Mon Mar 25 15:43:45 2019 -0500

        Add copy exception handling

    commit 0eca6f3
    Author: Ben Roling <[email protected]>
    Date:   Mon Mar 25 12:43:51 2019 -0500

        Don't process response from copy

    commit ad9e152
    Author: Ben Roling <[email protected]>
    Date:   Mon Feb 25 16:41:54 2019 -0600

        HADOOP-16085-003.patch

        Rebase of previous work after merge of HADOOP-15625.
Retries for eventually consistent read-after-overwrite
9a880ed 
Includes retries for regular reads, select(), and rename()
@steveloughran
HADOOP-16085 S3Guard to support version tracking
f45bf3e 
+add stevel review (primarily of tests)

Change-Id: I75a3b70917eefc0a0ec3190ca1de527e2081551e
Fix test failures when bucket versioning disabled
3824980
@steveloughran
HADOOP-16085-steve-patch-002
50f52ea 
-move CopyOutcome
-add new (failing) test for directory renaming where one of the files underneath is eventually consistent.
-pull out commonanity from test methods (spyOnFilesystem(), expectReadFailure())
-intercept() calls provide details on failure, primarily just by returning the result of the operation expected to fail.
-ChangeDetectionPolicy-raised exceptions on copy failures don't include position any more
-cache/restore metastore so even on test failures mockito is involved on teardown

Change-Id: I1fbdc38e9083881e4d72f13f29845e34539ac4ed
@steveloughran steveloughran mentioned this pull request May 9, 2019
Closed
ben-roling
ben-roling reviewed May 9, 2019
View reviewed changes
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CopyOutcome.java
/**
* Calls {@code Copy.waitForCopyResult()} to await the result, converts
* it to a copy outcome.
* Exceptions caught and
Copy link
Contributor

@ben-roling ben-roling May 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exceptions caught and

incomplete statement

@hadoop-yetus
Copy link

hadoop-yetus commented May 9, 2019

🎊 +1 overall

Vote Subsystem Runtime Comment
0 reexec 50 Docker mode activated.
_ Prechecks _
+1 dupname 2 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 25 new or modified test files.
_ trunk Compile Tests _
0 mvndep 66 Maven dependency ordering for branch
+1 mvninstall 1039 trunk passed
+1 compile 1059 trunk passed
+1 checkstyle 142 trunk passed
+1 mvnsite 118 trunk passed
+1 shadedclient 942 branch has no errors when building and testing our client artifacts.
+1 javadoc 91 trunk passed
0 spotbugs 60 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 171 trunk passed
_ Patch Compile Tests _
0 mvndep 22 Maven dependency ordering for patch
+1 mvninstall 72 the patch passed
+1 compile 1092 the patch passed
+1 javac 1092 the patch passed
-0 checkstyle 138 root: The patch generated 31 new + 70 unchanged - 4 fixed = 101 total (was 74)
+1 mvnsite 126 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 3 The patch has no ill-formed XML file.
+1 shadedclient 664 patch has no errors when building and testing our client artifacts.
+1 javadoc 93 the patch passed
+1 findbugs 196 the patch passed
_ Other Tests _
+1 unit 540 hadoop-common in the patch passed.
+1 unit 286 hadoop-aws in the patch passed.
+1 asflicense 43 The patch does not generate ASF License warnings.
6964
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-807/1/artifact/out/Dockerfile
GITHUB PR #807
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname Linux 2cc8e2263bc3 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 90add05
Default Java 1.8.0_191
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-807/1/artifact/out/diff-checkstyle-root.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-807/1/testReport/
Max. process+thread count 1389 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-807/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@steveloughran steveloughran deleted the s3/HADOOP-16085-s3guard-versioning branch May 16, 2019 11:57
shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
@weisong44
SAMZA-1998: Table API refactoring
5f7a22c 
Table API refactoring
     - Removed TableSpec
     - Consolidated configuration generation for tables to table descriptors
     - Refactored constructor so that only local table would require serde's
     - Removed table provider for RocksDB- and in-memory tables, and added LocalTableProvider
     - Updates to unit tests
     - Various refactoring

Author: Wei Song <[email protected]>

Reviewers: Prateek Maheshwari <[email protected]>

Closes apache#807 from weisong44/SAMZA-1998
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@ben-roling ben-roling ben-roling left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@steveloughran @hadoop-yetus @ben-roling