HADOOP-13551. AWS metrics wire-up (#2778) #2814
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tion validation. (apache#2427). Contributed by Steve Loughran.
…he#2433) Reviewed-by: Ayush Saxena <[email protected]> (cherry picked from commit 34e2abd)
…created pools. Contributed by Peter Bacsko.
…rollback. Contributed by Deegue. (cherry picked from commit 82de07c)
(cherry picked from commit 61f8c57)
…. Contributed by Stephen Jung (Stripe). (cherry picked from commit 45434c9)
…pache#2411) Contributed by Chao Sun.
…ing with skipShade (apache#2458)
…nient (apache#2447) Contributed by Attila Doroszlai. Change-Id: I21c29256b52449b7fea335704b3afa02e39c6a39
) Co-authored-by: ahussein <[email protected]> (cherry picked from commit 71071e5)
…reams in hdfs-default.xml. Contributed by Xieming Li. (cherry picked from commit f56cd88)
…ntributed by Daryn Sharp and Ahmed Hussein (cherry picked from commit ebe1d1f)
…pache#2462) Contributed by Attila Doroszlai. Change-Id: Ie15624ec07b1c5e34ca7fde0a72a54431d79e746
). Contributed by Ahmed Hussein (cherry picked from commit 0b2510e)
…he#2441) Co-authored-by: ahussein <[email protected]> (cherry picked from commit 1ea3f74)
…omainNameUsingDNS fails. (apache#2465) (cherry picked from commit b57f04c)
…pache#2466) Unless you explicitly set it, the issue date of a delegation token identifier is 0, which confuses spark renewal (SPARK-33440). This patch makes sure that all S3A DT identifiers have the current time as issue date, fixing the problem as far as S3A tokens are concerned. Contributed by Jungtaek Lim. Change-Id: Ic80ac7895612a1aa669459c73a78a9c17ecf0c0d
(cherry picked from commit 2045a9d)
…pache#2303) Contributed by Steve Loughran. Change-Id: I8e89a402a24bd9fb958e0fa93d1a28191093851d
apache#2310) This fixes the S3Guard/Directory Marker Retention integration so that when fs.s3a.directory.marker.retention=keep, failures during multipart delete are handled correctly, as are incremental deletes during directory tree operations. In both cases, when a directory marker with children is deleted from S3, the directory entry in S3Guard is not deleted, because it is still critical to representing the structure of the store. Contributed by Steve Loughran. Change-Id: I4ca133a23ea582cd42ec35dbf2dc85b286297d2f
…. Contributed by Daryn Sharp and Ahmed Hussein
…e#2477) Followup to HADOOP-17379. Contributed by Jungtaek Lim. Change-Id: I7b2fce36028d297c1e095499691a08caba92d9fd
… notify the given URL via a custom class. Contributed by Zoltan Erdmann
Contributed by Steve Loughran.
…pache#2469) Contributed by Zhihua Deng. Change-Id: Iee62539d02bd8f8a928171d8258e640487050a05
Contributed by Ayush Saxena and Steve Loughran Change-Id: I4bb76815bc1d11d1804dc67bafde68b6a995b974
…ls. Contributed by Ahmed Hussein (ahussein).
This patch * fixes the inversion * adds a precondition check * if the commands are supplied inverted, swaps them with a warning. This is to stop breaking any tests written to cope with the existing behavior. Contributed by Steve Loughran Change-Id: I15c40863f0db0675c7d60db477cb3bf1693cae49
…l should support it. Contributed by zhuqi. (cherry picked from commit c4ba0ab)
Contributed by Mukund Thakur. Change-Id: I1709ad72b829999b6dd324f0755b51bc38918d30
…te (apache#2788) (cherry picked from commit 343ce8a)
…ll retry forever. Contributed by AMC-team. (cherry picked from commit e3d1966)
… value can cause strange failure. Contributed by AMC-team. (cherry picked from commit c2a1765)
…ner. Contributed by Ahmed Hussein (cherry picked from commit ede490d)
… Contributed by Wei-Chiu Chuang. (cherry picked from commit ae13a5c)
…fied. Contributed by Ayush Saxena. (cherry picked from commit e0ae232)
…er than capacity. Contributed by Bilwa S T. (cherry picked from commit 174f3a9)
…teger. Contributed by Stephen O'Donnell.
…pache#2796) Contributed by Viraj Jasani. Signed-off-by: Mingliang Liu <[email protected]> Signed-off-by: Ayush Saxena <[email protected]>
(cherry picked from commit c12ddbd)
(cherry picked from commit 714427c)
….server.Journal (apache#2790). Contributed by Narges Shadab. Signed-off-by: Ayush Saxena <[email protected]> (cherry picked from commit d77c7ab)
…#2736). Contributed by Narges Shadab. Reviewed-by: He Xiaoqiao <[email protected]> (cherry picked from commit f5f3fc6)
…SClients. Contributed by Jinglun. (cherry picked from commit f8769e0)
Contributed by imbajin. Reviewed-by: leosunli <[email protected]> Signed-off-by: He Xiaoqiao <[email protected]> (cherry picked from commit 89428f1)
…closed. Contributed by Jinglun. (cherry picked from commit ff84a57)
…e set. Contributed by Jinglun. (cherry picked from commit 7025f39)
…pache#2569) (cherry picked from commit 9aefe72)
…by Bhavik Patel. (cherry picked from commit b4d97a8)
(cherry picked from commit 17ffcab)
…ds. Contributed by tomscut. (cherry picked from commit 5ab1db9)
…Lisheng Sun. (cherry picked from commit 6e04b00)
…t and data are true. Contributed by hemanthboyina. (cherry picked from commit 1fdfaeb)
…nts (apache#2395) Contributed by Mukund Thakur
Moves to the builder API for AWS S3 client creation, and offers a similar style of API to the S3A FileSystem and tests, hiding the details of which options are client, which are in AWS Conf, and doing the wiring up of S3A statistics interfaces to the AWS SDK internals. S3A Statistics, including IOStatistics, should now count throttling events handled in the AWS SDK itself. This patch restores endpoint determination by probes to US-East-1 if the client isn't configured with fs.s3a.endpoint. Explicitly setting the endpoint will save the cost of these probe HTTP requests. Contributed by Steve Loughran.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
64 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of trunk #2778 to 3.3; had to include another patch to avoid merge conflict.
Not yet done any s3 tests