diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000000000..2b5014b7b234d --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,6 @@ +## NOTICE + +Please create an issue in ASF JIRA before opening a pull request, +and you need to set the title of the pull request which starts with +the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.) +For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute diff --git a/BUILDING.txt b/BUILDING.txt index cc9ac177ca298..d3c9a1a7f51ee 100644 --- a/BUILDING.txt +++ b/BUILDING.txt @@ -6,7 +6,7 @@ Requirements: * Unix System * JDK 1.8 * Maven 3.3 or later -* ProtocolBuffer 2.5.0 +* Protocol Buffers 3.7.1 (if compiling native code) * CMake 3.1 or newer (if compiling native code) * Zlib devel (if compiling native code) * Cyrus SASL devel (if compiling native code) @@ -62,8 +62,16 @@ Installing required packages for clean install of Ubuntu 14.04 LTS Desktop: $ sudo apt-get -y install maven * Native libraries $ sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev libsasl2-dev -* ProtocolBuffer 2.5.0 (required) - $ sudo apt-get -y install protobuf-compiler +* Protocol Buffers 3.7.1 (required to build native code) + $ mkdir -p /opt/protobuf-3.7-src \ + && curl -L -s -S \ + https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ + -o /opt/protobuf-3.7.1.tar.gz \ + && tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C /opt/protobuf-3.7-src \ + && cd /opt/protobuf-3.7-src \ + && ./configure\ + && make install \ + && rm -rf /opt/protobuf-3.7-src Optional packages: @@ -78,6 +86,8 @@ Optional packages: $ sudo apt-get install fuse libfuse-dev * ZStandard compression $ sudo apt-get install zstd +* PMDK library for storage class memory(SCM) as HDFS cache backend + Please refer to http://pmem.io/ and https://github.com/pmem/pmdk ---------------------------------------------------------------------------------- Maven main modules: @@ -262,6 +272,32 @@ Maven build goals: invoke, run 'mvn dependency-check:aggregate'. Note that this plugin requires maven 3.1.1 or greater. + PMDK library build options: + + The Persistent Memory Development Kit (PMDK), formerly known as NVML, is a growing + collection of libraries which have been developed for various use cases, tuned, + validated to production quality, and thoroughly documented. These libraries are built + on the Direct Access (DAX) feature available in both Linux and Windows, which allows + applications directly load/store access to persistent memory by memory-mapping files + on a persistent memory aware file system. + + It is currently an optional component, meaning that Hadoop can be built without + this dependency. Please Note the library is used via dynamic module. For getting + more details please refer to the official sites: + http://pmem.io/ and https://github.com/pmem/pmdk. + + * -Drequire.pmdk is used to build the project with PMDK libraries forcibly. With this + option provided, the build will fail if libpmem library is not found. If this option + is not given, the build will generate a version of Hadoop with libhadoop.so. + And storage class memory(SCM) backed HDFS cache is still supported without PMDK involved. + Because PMDK can bring better caching write/read performance, it is recommended to build + the project with this option if user plans to use SCM backed HDFS cache. + * -Dpmdk.lib is used to specify a nonstandard location for PMDK libraries if they are not + under /usr/lib or /usr/lib64. + * -Dbundle.pmdk is used to copy the specified libpmem libraries into the distribution tar + package. This option requires that -Dpmdk.lib is specified. With -Dbundle.pmdk provided, + the build will fail if -Dpmdk.lib is not specified. + ---------------------------------------------------------------------------------- Building components separately @@ -274,16 +310,6 @@ level once; and then work from the submodule. Keep in mind that SNAPSHOTs time out after a while, using the Maven '-nsu' will stop Maven from trying to update SNAPSHOTs from external repos. ----------------------------------------------------------------------------------- -Protocol Buffer compiler - -The version of Protocol Buffer compiler, protoc, must match the version of the -protobuf JAR. - -If you have multiple versions of protoc in your system, you can set in your -build shell the HADOOP_PROTOC_PATH environment variable to point to the one you -want to use for the Hadoop build. If you don't define this environment variable, -protoc is looked up in the PATH. ---------------------------------------------------------------------------------- Importing projects to eclipse @@ -352,6 +378,49 @@ export MAVEN_OPTS="-Xms256m -Xmx1536m" ---------------------------------------------------------------------------------- +Building on macOS (without Docker) + +---------------------------------------------------------------------------------- +Installing required dependencies for clean install of macOS 10.14: + +* Install Xcode Command Line Tools + $ xcode-select --install +* Install Homebrew + $ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" +* Install OpenJDK 8 + $ brew tap AdoptOpenJDK/openjdk + $ brew cask install adoptopenjdk8 +* Install maven and tools + $ brew install maven autoconf automake cmake wget +* Install native libraries, only openssl is required to compile native code, +you may optionally install zlib, lz4, etc. + $ brew install openssl +* Protocol Buffers 3.7.1 (required to compile native code) + $ wget https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz + $ mkdir -p protobuf-3.7 && tar zxvf protobuf-java-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7 + $ cd protobuf-3.7 + $ ./configure + $ make + $ make check + $ make install + $ protoc --version + +Note that building Hadoop 3.1.1/3.1.2/3.2.0 native code from source is broken +on macOS. For 3.1.1/3.1.2, you need to manually backport YARN-8622. For 3.2.0, +you need to backport both YARN-8622 and YARN-9487 in order to build native code. + +---------------------------------------------------------------------------------- +Building command example: + +* Create binary distribution with native code but without documentation: + $ mvn package -Pdist,native -DskipTests -Dmaven.javadoc.skip \ + -Dopenssl.prefix=/usr/local/opt/openssl + +Note that the command above manually specified the openssl library and include +path. This is necessary at least for Homebrewed OpenSSL. + +---------------------------------------------------------------------------------- + Building on Windows ---------------------------------------------------------------------------------- @@ -360,7 +429,7 @@ Requirements: * Windows System * JDK 1.8 * Maven 3.0 or later -* ProtocolBuffer 2.5.0 +* Protocol Buffers 3.7.1 * CMake 3.1 or newer * Visual Studio 2010 Professional or Higher * Windows SDK 8.1 (if building CPU rate control for the container executor) diff --git a/LICENSE-binary b/LICENSE-binary new file mode 100644 index 0000000000000..588b9dc865ebc --- /dev/null +++ b/LICENSE-binary @@ -0,0 +1,522 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +-------------------------------------------------------------------------------- +This project bundles some components that are also licensed under the Apache +License Version 2.0: + + +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/nvd3-1.8.5.* (css and js files) +hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java +hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/TimeoutFuture.java + +com.aliyun:aliyun-java-sdk-core:3.4.0 +com.aliyun:aliyun-java-sdk-ecs:4.2.0 +com.aliyun:aliyun-java-sdk-ram:3.0.0 +com.aliyun:aliyun-java-sdk-sts:3.0.0 +com.aliyun.oss:aliyun-sdk-oss:3.4.1 +com.amazonaws:aws-java-sdk-bundle:1.11.563 +com.cedarsoftware:java-util:1.9.0 +com.cedarsoftware:json-io:2.5.1 +com.fasterxml.jackson.core:jackson-annotations:2.9.9 +com.fasterxml.jackson.core:jackson-core:2.9.9 +com.fasterxml.jackson.core:jackson-databind:2.9.9.2 +com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.9.9 +com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.9.9 +com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.9.9 +com.fasterxml.uuid:java-uuid-generator:3.1.4 +com.fasterxml.woodstox:woodstox-core:5.0.3 +com.github.davidmoten:rxjava-extras:0.8.0.17 +com.github.stephenc.jcip:jcip-annotations:1.0-1 +com.google:guice:4.0 +com.google:guice-servlet:4.0 +com.google.api.grpc:proto-google-common-protos:1.0.0 +com.google.code.gson:2.2.4 +com.google.errorprone:error_prone_annotations:2.2.0 +com.google.j2objc:j2objc-annotations:1.1 +com.google.json-simple:json-simple:1.1.1 +com.google.guava:failureaccess:1.0 +com.google.guava:guava:20.0 +com.google.guava:guava:27.0-jre +com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava +com.microsoft.azure:azure-storage:7.0.0 +com.nimbusds:nimbus-jose-jwt:4.41.1 +com.squareup.okhttp:okhttp:2.7.5 +com.squareup.okio:okio:1.6.0 +com.zaxxer:HikariCP-java7:2.4.12 +commons-beanutils:commons-beanutils:1.9.3 +commons-cli:commons-cli:1.2 +commons-codec:commons-codec:1.11 +commons-collections:commons-collections:3.2.2 +commons-daemon:commons-daemon:1.0.13 +commons-io:commons-io:2.5 +commons-lang:commons-lang:2.6 +commons-logging:commons-logging:1.1.3 +commons-net:commons-net:3.6 +de.ruedigermoeller:fst:2.50 +io.grpc:grpc-context:1.15.1 +io.grpc:grpc-core:1.15.1 +io.grpc:grpc-netty:1.15.1 +io.grpc:grpc-protobuf:1.15.1 +io.grpc:grpc-protobuf-lite:1.15.1 +io.grpc:grpc-stub:1.15.1 +io.netty:netty:3.10.6.Final +io.netty:netty-all:4.0.52.Final +io.netty:netty-all:4.1.27.Final +io.netty:netty-buffer:4.1.27.Final +io.netty:netty-codec:4.1.27.Final +io.netty:netty-codec-http:4.1.27.Final +io.netty:netty-codec-http2:4.1.27.Final +io.netty:netty-codec-socks:4.1.27.Final +io.netty:netty-common:4.1.27.Final +io.netty:netty-handler:4.1.27.Final +io.netty:netty-handler-proxy:4.1.27.Final +io.netty:netty-resolver:4.1.27.Final +io.netty:netty-transport:4.1.27.Final +io.opencensus:opencensus-api:0.12.3 +io.opencensus:opencensus-contrib-grpc-metrics:0.12.3 +io.reactivex:rxjava:1.3.8 +io.reactivex:rxjava-string:1.1.1 +io.reactivex:rxnetty:0.4.20 +io.swagger:swagger-annotations:1.5.4 +javax.inject:javax.inject:1 +log4j:log4j:1.2.17 +net.java.dev.jna:jna:5.2.0 +net.jpountz.lz4:lz4:1.2.0 +net.minidev:accessors-smart:1.2 +net.minidev:json-smart:2.3 +org.apache.avro:avro:1.7.7 +org.apache.commons:commons-collections4:4.2 +org.apache.commons:commons-compress:1.19 +org.apache.commons:commons-configuration2:2.1.1 +org.apache.commons:commons-csv:1.0 +org.apache.commons:commons-digester:1.8.1 +org.apache.commons:commons-lang3:3.7 +org.apache.commons:commons-math3:3.1.1 +org.apache.commons:commons-text:1.4 +org.apache.commons:commons-validator:1.6 +org.apache.curator:curator-client:2.13.0 +org.apache.curator:curator-framework:2.13.0 +org.apache.curator:curator-recipes:2.13.0 +org.apache.geronimo.specs:geronimo-jcache_1.0_spec:1.0-alpha-1 +org.apache.hbase:hbase-annotations:1.4.8 +org.apache.hbase:hbase-client:1.4.8 +org.apache.hbase:hbase-common:1.4.8 +org.apache.hbase:hbase-protocol:1.4.8 +org.apache.htrace:htrace-core:3.1.0-incubating +org.apache.htrace:htrace-core4:4.1.0-incubating +org.apache.httpcomponents:httpclient:4.5.6 +org.apache.httpcomponents:httpcore:4.4.10 +org.apache.kafka:kafka-clients:0.8.2.1 +org.apache.kerby:kerb-admin:1.0.1 +org.apache.kerby:kerb-client:1.0.1 +org.apache.kerby:kerb-common:1.0.1 +org.apache.kerby:kerb-core:1.0.1 +org.apache.kerby:kerb-crypto:1.0.1 +org.apache.kerby:kerb-identity:1.0.1 +org.apache.kerby:kerb-server:1.0.1 +org.apache.kerby:kerb-simplekdc:1.0.1 +org.apache.kerby:kerb-util:1.0.1 +org.apache.kerby:kerby-asn1:1.0.1 +org.apache.kerby:kerby-config:1.0.1 +org.apache.kerby:kerby-pkix:1.0.1 +org.apache.kerby:kerby-util:1.0.1 +org.apache.kerby:kerby-xdr:1.0.1 +org.apache.kerby:token-provider:1.0.1 +org.apache.yetus:audience-annotations:0.5.0 +org.apache.zookeeper:zookeeper:3.4.13 +org.codehaus.jackson:jackson-core-asl:1.9.13 +org.codehaus.jackson:jackson-jaxrs:1.9.13 +org.codehaus.jackson:jackson-mapper-asl:1.9.13 +org.codehaus.jackson:jackson-xc:1.9.13 +org.codehaus.jettison:jettison:1.1 +org.eclipse.jetty:jetty-annotations:9.3.27.v20190418 +org.eclipse.jetty:jetty-http:9.3.27.v20190418 +org.eclipse.jetty:jetty-io:9.3.27.v20190418 +org.eclipse.jetty:jetty-jndi:9.3.27.v20190418 +org.eclipse.jetty:jetty-plus:9.3.27.v20190418 +org.eclipse.jetty:jetty-security:9.3.27.v20190418 +org.eclipse.jetty:jetty-server:9.3.27.v20190418 +org.eclipse.jetty:jetty-servlet:9.3.27.v20190418 +org.eclipse.jetty:jetty-util:9.3.27.v20190418 +org.eclipse.jetty:jetty-util-ajax:9.3.27.v20190418 +org.eclipse.jetty:jetty-webapp:9.3.27.v20190418 +org.eclipse.jetty:jetty-xml:9.3.27.v20190418 +org.eclipse.jetty.websocket:javax-websocket-client-impl:9.3.27.v20190418 +org.eclipse.jetty.websocket:javax-websocket-server-impl:9.3.27.v20190418 +org.ehcache:ehcache:3.3.1 +org.objenesis:objenesis:1.0:compile +org.xerial.snappy:snappy-java:1.0.5 +org.yaml:snakeyaml:1.16: +org.wildfly.openssl:wildfly-openssl:1.0.7.Final + + +-------------------------------------------------------------------------------- +This product bundles various third-party components under other open source +licenses. This section summarizes those components and their licenses. +See licenses-binary/ for text of these licenses. + + +BSD 2-Clause +------------ + +hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/{lz4.h,lz4.c,lz4hc.h,lz4hc.c} +hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/util/tree.h +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/compat/{fstatat|openat|unlinkat}.h + +dnsjava:dnsjava:2.1.7 +org.codehaus.woodstox:stax2-api:3.1.4 + + +BSD 3-Clause +------------ + +hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/* +hadoop-common-project/hadoop-common/src/main/native/gtest/gtest-all.cc +hadoop-common-project/hadoop-common/src/main/native/gtest/include/gtest/gtest.h +hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c +hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/d3.v3.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/d3-3.5.17.min.js +leveldb v1.13 + +com.google.protobuf:protobuf-java:2.5.0 +com.google.protobuf:protobuf-java:3.6.1 +com.google.re2j:re2j:1.1 +com.jcraft:jsch:0.1.54 +com.thoughtworks.paranamer:paranamer:2.3 +javax.activation:javax.activation-api:1.2.0 +org.fusesource.leveldbjni:leveldbjni-all:1.8 +org.jline:jline:3.9.0 +org.hamcrest:hamcrest-core:1.3 +org.ow2.asm:asm:5.0.4 +org.ow2.asm:asm-commons:6.0 +org.ow2.asm:asm-tree:6.0 + + +MIT License +----------- + +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-1.6.4.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-nvd3-1.0.9.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-route-1.6.4.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/bootstrap-3.4.1 +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dataTables.bootstrap.css +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dataTables.bootstrap.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dust-full-2.0.0.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dust-helpers-1.1.1.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/jquery-3.4.1.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/jquery.dataTables.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/moment.min.js +hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/bootstrap.min.js +hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/jquery.js +hadoop-tools/hadoop-sls/src/main/html/css/bootstrap.min.css +hadoop-tools/hadoop-sls/src/main/html/css/bootstrap-responsive.min.css +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.18/* +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jt/jquery.jstree.js +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL + +bootstrap v3.3.6 +broccoli-asset-rev v2.4.2 +broccoli-funnel v1.0.1 +datatables v1.10.8 +em-helpers v0.5.13 +em-table v0.1.6 +ember v2.2.0 +ember-array-contains-helper v1.0.2 +ember-bootstrap v0.5.1 +ember-cli v1.13.13 +ember-cli-app-version v1.0.0 +ember-cli-babel v5.1.6 +ember-cli-content-security-policy v0.4.0 +ember-cli-dependency-checker v1.2.0 +ember-cli-htmlbars v1.0.2 +ember-cli-htmlbars-inline-precompile v0.3.1 +ember-cli-ic-ajax v0.2.1 +ember-cli-inject-live-reload v1.4.0 +ember-cli-jquery-ui v0.0.20 +ember-cli-qunit v1.2.1 +ember-cli-release v0.2.8 +ember-cli-shims v0.0.6 +ember-cli-sri v1.2.1 +ember-cli-test-loader v0.2.1 +ember-cli-uglify v1.2.0 +ember-d3 v0.1.0 +ember-data v2.1.0 +ember-disable-proxy-controllers v1.0.1 +ember-export-application-global v1.0.5 +ember-load-initializers v0.1.7 +ember-qunit v0.4.16 +ember-qunit-notifications v0.1.0 +ember-resolver v2.0.3 +ember-spin-spinner v0.2.3 +ember-truth-helpers v1.2.0 +jquery v2.1.4 +jquery-ui v1.11.4 +loader.js v3.3.0 +momentjs v2.10.6 +qunit v1.19.0 +select2 v4.0.0 +snippet-ss v1.11.0 +spin.js v2.3.2 + +com.microsoft.azure:azure-cosmosdb:2.4.5 +com.microsoft.azure:azure-cosmosdb-commons:2.4.5 +com.microsoft.azure:azure-cosmosdb-direct:2.4.5 +com.microsoft.azure:azure-cosmosdb-gateway:2.4.5 +com.microsoft.azure:azure-data-lake-store-sdk:2.3.3 +com.microsoft.azure:azure-keyvault-core:1.0.0 +com.microsoft.sqlserver:mssql-jdbc:6.2.1.jre7 +org.bouncycastle:bcpkix-jdk15on:1.60 +org.bouncycastle:bcprov-jdk15on:1.60 +org.checkerframework:checker-qual:2.5.2 +org.codehaus.mojo:animal-sniffer-annotations:1.17 +org.jruby.jcodings:jcodings:1.0.13 +org.jruby.joni:joni:2.1.2 +org.slf4j:jul-to-slf4j:jar:1.7.25 +org.ojalgo:ojalgo:43.0:compile +org.slf4j:jul-to-slf4j:1.7.25 +org.slf4j:slf4j-api:1.7.25 +org.slf4j:slf4j-log4j12:1.7.25 + + +CDDL 1.1 + GPLv2 with classpath exception +----------------------------------------- + +com.sun.jersey:jersey-client:1.19 +com.sun.jersey:jersey-core:1.19 +com.sun.jersey:jersey-guice:1.19 +com.sun.jersey:jersey-json:1.19 +com.sun.jersey:jersey-server:1.19 +com.sun.jersey:jersey-servlet:1.19 +com.sun.xml.bind:jaxb-impl:2.2.3-1 +javax.annotation:javax.annotation-api:1.3.2 +javax.servlet:javax.servlet-api:3.1.0 +javax.servlet.jsp:jsp-api:2.1 +javax.websocket:javax.websocket-api:1.0 +javax.ws.rs:jsr311-api:1.1.1 +javax.xml.bind:jaxb-api:2.2.11 + + +Eclipse Public License 1.0 +-------------------------- + +junit:junit:4.12 + + +HSQL License +------------ + +org.hsqldb:hsqldb:2.3.4 + + +JDOM License +------------ + +org.jdom:jdom:1.1 + + +Public Domain +------------- + +aopalliance:aopalliance:1.0 diff --git a/LICENSE.txt b/LICENSE.txt index e100d26aeb1b0..d0d57461e76df 100644 --- a/LICENSE.txt +++ b/LICENSE.txt @@ -201,2687 +201,58 @@ See the License for the specific language governing permissions and limitations under the License. +-------------------------------------------------------------------------------- +This product bundles various third-party components under other open source +licenses. This section summarizes those components and their licenses. +See licenses/ for text of these licenses. -APACHE HADOOP SUBCOMPONENTS: - -The Apache Hadoop project contains subcomponents with separate copyright -notices and license terms. Your use of the source code for the these -subcomponents is subject to the terms and conditions of the following -licenses. - -For the org.apache.hadoop.util.bloom.* classes: - -/** - * - * Copyright (c) 2005, European Commission project OneLab under contract - * 034819 (http://www.one-lab.org) - * All rights reserved. - * Redistribution and use in source and binary forms, with or - * without modification, are permitted provided that the following - * conditions are met: - * - Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * - Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the distribution. - * - Neither the name of the University Catholique de Louvain - UCL - * nor the names of its contributors may be used to endorse or - * promote products derived from this software without specific prior - * written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS - * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER - * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN - * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - */ - -For portions of the native implementation of slicing-by-8 CRC calculation -in src/main/native/src/org/apache/hadoop/util: - -Copyright (c) 2008,2009,2010 Massachusetts Institute of Technology. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: - -* Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. -* Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. -* Neither the name of the Massachusetts Institute of Technology nor - the names of its contributors may be used to endorse or promote - products derived from this software without specific prior written - permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - -Other portions are under the same license from Intel: -http://sourceforge.net/projects/slicing-by-8/ -/*++ - * - * Copyright (c) 2004-2006 Intel Corporation - All Rights Reserved - * - * This software program is licensed subject to the BSD License, - * available at http://www.opensource.org/licenses/bsd-license.html - * - * Abstract: The main routine - * - --*/ - -For src/main/native/src/org/apache/hadoop/io/compress/lz4/{lz4.h,lz4.c,lz4hc.h,lz4hc.c}, - -/* - LZ4 - Fast LZ compression algorithm - Header File - Copyright (C) 2011-2014, Yann Collet. - BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are - met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above - copyright notice, this list of conditions and the following disclaimer - in the documentation and/or other materials provided with the - distribution. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - You can contact the author at : - - LZ4 source repository : http://code.google.com/p/lz4/ - - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c -*/ - - -For hadoop-common-project/hadoop-common/src/main/native/gtest ---------------------------------------------------------------------- -Copyright 2008, Google Inc. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: - - * Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above -copyright notice, this list of conditions and the following disclaimer -in the documentation and/or other materials provided with the -distribution. - * Neither the name of Google Inc. nor the names of its -contributors may be used to endorse or promote products derived from -this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -The binary distribution of this product bundles these dependencies under the -following license: -re2j 1.1 ---------------------------------------------------------------------- -(GO license) -This is a work derived from Russ Cox's RE2 in Go, whose license -http://golang.org/LICENSE is as follows: - -Copyright (c) 2009 The Go Authors. All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - - * Neither the name of Google Inc. nor the names of its contributors - may be used to endorse or promote products derived from this - software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -For hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/util/tree.h ---------------------------------------------------------------------- -Copyright 2002 Niels Provos -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions -are met: -1. Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. -2. Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - -THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR -IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES -OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. -IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, -INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT -NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF -THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -The binary distribution of this product bundles binaries of leveldbjni -(https://github.com/fusesource/leveldbjni), which is available under the -following license: - -Copyright (c) 2011 FuseSource Corp. All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: - - * Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above -copyright notice, this list of conditions and the following disclaimer -in the documentation and/or other materials provided with the -distribution. - * Neither the name of FuseSource Corp. nor the names of its -contributors may be used to endorse or promote products derived from -this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -For hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/compat/{fstatat|openat|unlinkat}.h: - -Copyright (c) 2012 The FreeBSD Foundation -All rights reserved. - -This software was developed by Pawel Jakub Dawidek under sponsorship from -the FreeBSD Foundation. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions -are met: - -1. Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. -2. Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - -THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE -FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL -DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS -OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) -HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT -LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY -OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF -SUCH DAMAGE. - -============= -The binary distribution of this product bundles binaries of leveldb -(http://code.google.com/p/leveldb/), which is available under the following -license: +Apache Software Foundation License 2.0 +-------------------------------------- -Copyright (c) 2011 The LevelDB Authors. All rights reserved. +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/nvd3-1.8.5.* (css and js files) +hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java +hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/TimeoutFuture.java -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: - * Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above -copyright notice, this list of conditions and the following disclaimer -in the documentation and/or other materials provided with the -distribution. - * Neither the name of Google Inc. nor the names of its -contributors may be used to endorse or promote products derived from -this software without specific prior written permission. +BSD 2-Clause +------------ -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/{lz4.h,lz4.c,lz4hc.h,lz4hc.c} +hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/util/tree.h +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/compat/{fstatat|openat|unlinkat}.h -The binary distribution of this product bundles binaries of snappy -(http://code.google.com/p/snappy/), which is available under the following -license: -Copyright 2011, Google Inc. -All rights reserved. +BSD 3-Clause +------------ -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: +hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/* +hadoop-common-project/hadoop-common/src/main/native/gtest/gtest-all.cc +hadoop-common-project/hadoop-common/src/main/native/gtest/include/gtest/gtest.h +hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c +hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/d3.v3.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/d3-3.5.17.min.js - * Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above -copyright notice, this list of conditions and the following disclaimer -in the documentation and/or other materials provided with the -distribution. - * Neither the name of Google Inc. nor the names of its -contributors may be used to endorse or promote products derived from -this software without specific prior written permission. -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +MIT License +----------- -For: -hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dataTables.bootstrap.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-1.6.4.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-nvd3-1.0.9.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-route-1.6.4.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/bootstrap-3.4.1 hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dataTables.bootstrap.css -hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/jquery.dataTables.min.js -hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.18/ --------------------------------------------------------------------------------- -Copyright (C) 2008-2016, SpryMedia Ltd. - -Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - -For: +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dataTables.bootstrap.js hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dust-full-2.0.0.min.js hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dust-helpers-1.1.1.min.js --------------------------------------------------------------------------------- - -Copyright (c) 2010 Aleksander Williams - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -THE SOFTWARE. - -For: +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/jquery-3.4.1.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/jquery.dataTables.min.js hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/moment.min.js --------------------------------------------------------------------------------- - -Copyright (c) 2011-2016 Tim Wood, Iskren Chernev, Moment.js contributors - -Permission is hereby granted, free of charge, to any person -obtaining a copy of this software and associated documentation -files (the "Software"), to deal in the Software without -restriction, including without limitation the rights to use, -copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the -Software is furnished to do so, subject to the following -conditions: - -The above copyright notice and this permission notice shall be -included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -OTHER DEALINGS IN THE SOFTWARE. - -The binary distribution of this product bundles these dependencies under the -following license: -hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/bootstrap-3.3.7 hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/bootstrap.min.js +hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/jquery.js hadoop-tools/hadoop-sls/src/main/html/css/bootstrap.min.css hadoop-tools/hadoop-sls/src/main/html/css/bootstrap-responsive.min.css -bootstrap v3.3.6 -broccoli-asset-rev v2.4.2 -broccoli-funnel v1.0.1 -datatables v1.10.8 -em-helpers v0.5.13 -em-table v0.1.6 -ember v2.2.0 -ember-array-contains-helper v1.0.2 -ember-bootstrap v0.5.1 -ember-cli v1.13.13 -ember-cli-app-version v1.0.0 -ember-cli-babel v5.1.6 -ember-cli-content-security-policy v0.4.0 -ember-cli-dependency-checker v1.2.0 -ember-cli-htmlbars v1.0.2 -ember-cli-htmlbars-inline-precompile v0.3.1 -ember-cli-ic-ajax v0.2.1 -ember-cli-inject-live-reload v1.4.0 -ember-cli-jquery-ui v0.0.20 -ember-cli-qunit v1.2.1 -ember-cli-release v0.2.8 -ember-cli-shims v0.0.6 -ember-cli-sri v1.2.1 -ember-cli-test-loader v0.2.1 -ember-cli-uglify v1.2.0 -ember-d3 v0.1.0 -ember-data v2.1.0 -ember-disable-proxy-controllers v1.0.1 -ember-export-application-global v1.0.5 -ember-load-initializers v0.1.7 -ember-qunit v0.4.16 -ember-qunit-notifications v0.1.0 -ember-resolver v2.0.3 -ember-spin-spinner v0.2.3 -ember-truth-helpers v1.2.0 -jquery v2.1.4 -jquery-ui v1.11.4 -loader.js v3.3.0 -momentjs v2.10.6 -qunit v1.19.0 -select2 v4.0.0 -snippet-ss v1.11.0 -spin.js v2.3.2 -Azure Data Lake Store - Java client SDK 2.0.11 -JCodings 1.0.8 -Joni 2.1.2 -Mockito 2.23.4 -JUL to SLF4J bridge 1.7.25 -SLF4J API Module 1.7.25 -SLF4J LOG4J-12 Binding 1.7.25 --------------------------------------------------------------------------------- - -The MIT License (MIT) - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -THE SOFTWARE. - - -For: -./hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/nvd3-1.8.5.* (css and js files) --------------------------------------------------------------------------------- -Copyright (c) 2011-2014 Novus Partners, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); you may not use this -file except in compliance with the License. You may obtain a copy of the License at - -http://www.apache.org/licenses/LICENSE-2.0 -Unless required by applicable law or agreed to in writing, software distributed under the - License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either - express or implied. See the License for the specific language governing permissions and - limitations under the License. - - - -For: -hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-nvd3-1.0.9.min.js --------------------------------------------------------------------------------- -The MIT License (MIT) -Copyright (c) 2014 Konstantin Skipor - -Permission is hereby granted, free of charge, to any person obtaining a copy of this software -and associated documentation files (the "Software"), to deal in the Software without restriction, -including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, -and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, -subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT -LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. -IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE -OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - - -For: -hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-1.6.4.min.js -hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-route-1.6.4.min.js --------------------------------------------------------------------------------- -The MIT License - -Copyright (c) 2010-2017 Google, Inc. http://angularjs.org - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -THE SOFTWARE. - - - - -For: -hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/jquery-3.3.1.min.js -hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/jquery.js +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.18/* hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery -Apache HBase - Server which contains JQuery minified javascript library version 1.8.3 -Microsoft JDBC Driver for SQLServer - version 6.2.1.jre7 --------------------------------------------------------------------------------- - -MIT License - -Copyright (c) 2003-2017 Optimatika - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. - -For: -oj! Algorithms - version 43.0 --------------------------------------------------------------------------------- - -Copyright 2005, 2012, 2013 jQuery Foundation and other contributors, https://jquery.org/ - -This software consists of voluntary contributions made by many -individuals. For exact contribution history, see the revision history -available at https://github.com/jquery/jquery - -The following license applies to all parts of this software except as -documented below: - -==== - -Permission is hereby granted, free of charge, to any person obtaining -a copy of this software and associated documentation files (the -"Software"), to deal in the Software without restriction, including -without limitation the rights to use, copy, modify, merge, publish, -distribute, sublicense, and/or sell copies of the Software, and to -permit persons to whom the Software is furnished to do so, subject to -the following conditions: - -The above copyright notice and this permission notice shall be -included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - -==== - -All files located in the node_modules and external directories are -externally maintained libraries used by this software which have their -own licenses; we recommend you read them, as their terms may differ from -the terms above. - -For: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jt/jquery.jstree.js --------------------------------------------------------------------------------- - -Copyright (c) 2014 Ivan Bozhanov - -Permission is hereby granted, free of charge, to any person -obtaining a copy of this software and associated documentation -files (the "Software"), to deal in the Software without -restriction, including without limitation the rights to use, -copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the -Software is furnished to do so, subject to the following -conditions: - -The above copyright notice and this permission notice shall be -included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -OTHER DEALINGS IN THE SOFTWARE. - -For: -hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/d3.v3.js -hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/d3-3.5.17.min.js --------------------------------------------------------------------------------- - -D3 is available under a 3-clause BSD license. For details, see: -hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/d3-LICENSE - -The binary distribution of this product bundles these dependencies under the -following license: -HSQLDB Database 2.3.4 --------------------------------------------------------------------------------- -(HSQL License) -"COPYRIGHTS AND LICENSES (based on BSD License) - -For work developed by the HSQL Development Group: - -Copyright (c) 2001-2016, The HSQL Development Group -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: - -Redistributions of source code must retain the above copyright notice, this -list of conditions and the following disclaimer. - -Redistributions in binary form must reproduce the above copyright notice, -this list of conditions and the following disclaimer in the documentation -and/or other materials provided with the distribution. - -Neither the name of the HSQL Development Group nor the names of its -contributors may be used to endorse or promote products derived from this -software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ""AS IS"" -AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -ARE DISCLAIMED. IN NO EVENT SHALL HSQL DEVELOPMENT GROUP, HSQLDB.ORG, -OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, -EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, -PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - -For work originally developed by the Hypersonic SQL Group: - -Copyright (c) 1995-2000 by the Hypersonic SQL Group. -All rights reserved. -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: - -Redistributions of source code must retain the above copyright notice, this -list of conditions and the following disclaimer. - -Redistributions in binary form must reproduce the above copyright notice, -this list of conditions and the following disclaimer in the documentation -and/or other materials provided with the distribution. - -Neither the name of the Hypersonic SQL Group nor the names of its -contributors may be used to endorse or promote products derived from this -software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ""AS IS"" -AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -ARE DISCLAIMED. IN NO EVENT SHALL THE HYPERSONIC SQL GROUP, -OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, -EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, -PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -This software consists of voluntary contributions made by many individuals on behalf of the -Hypersonic SQL Group." - -The binary distribution of this product bundles these dependencies under the -following license: -Java Servlet API 3.1.0 -servlet-api 2.5 -jsp-api 2.1 -jsr311-api 1.1.1 -Glassfish Jasper 6.1.14 -Servlet Specification 2.5 API 6.1.14 --------------------------------------------------------------------------------- -(CDDL 1.0) -COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0 -1. Definitions.  - -1.1. Contributor means each individual or entity -that creates or contributes to the creation of -Modifications.  - -1.2. Contributor Version means the combination of the -Original Software, prior Modifications used by a Contributor (if any), and the -Modifications made by that particular Contributor.  - -1.3. Covered -Software means (a) the Original Software, or (b) Modifications, or (c) the -combination of files containing Original Software with files containing -Modifications, in each case including portions -thereof.  - -1.4. Executable means the Covered Software in any form other -than Source Code.  - -1.5. Initial Developer means the individual or entity -that first makes Original Software available under this -License.  - -1.6. Larger Work means a work which combines Covered Software or -portions thereof with code not governed by the terms of this -License.  - -1.7. License means this document.  - -1.8. Licensable means -having the right to grant, to the maximum extent possible, whether at the time -of the initial grant or subsequently acquired, any and all of the rights -conveyed herein.  - -1.9. Modifications means the Source Code and Executable -form of any of the following: -A. Any file that results from an addition to, -deletion from or modification of the contents of a file containing Original -Software or previous Modifications; -B. Any new file that contains any part of the Original Software -or previous Modification; or -C. Any new file that is contributed or otherwise made available -under the terms of this License.  - -1.10. Original Software means the Source Code and Executable form of -computer software code that is originally released under this License.  - -1.11. Patent Claims means any patent claim(s), now owned or -hereafter acquired, including without limitation, method, process, and apparatus -claims, in any patent Licensable by grantor.  - -1.12. Source Code means (a) the common form of computer software code in which -modifications are made and (b) associated documentation included in or -with such code.  - -1.13. You (or Your) means an individual or a legal entity exercising rights -under, and complying with all of the terms of, this License. For legal entities, -You includes any entity which controls, is controlled by, or is under common control -with You. For purposes of this definition, control means (a) the power, direct -or indirect, to cause the direction or management of such entity, whether by -contract or otherwise, or (b) ownership of more than fifty percent (50%) of the -outstanding shares or beneficial ownership of such entity.  - -2. License Grants. - -2.1. The Initial Developer Grant. Conditioned upon Your compliance -with Section 3.1 below and subject to third party intellectual property claims, -the Initial Developer hereby grants You a world-wide, royalty-free, -non-exclusive license:  - -(a) under intellectual property rights (other than -patent or trademark) Licensable by Initial Developer, to use, reproduce, modify, -display, perform, sublicense and distribute the Original Software (or portions -thereof), with or without Modifications, and/or as part of a Larger Work; -and  - -(b) under Patent Claims infringed by the making, using or selling of -Original Software, to make, have made, use, practice, sell, and offer for sale, -and/or otherwise dispose of the Original Software (or portions -thereof); - -(c) The licenses granted in Sections 2.1(a) and (b) are -effective on the date Initial Developer first distributes or otherwise makes the -Original Software available to a third party under the terms of this -License; - -(d) Notwithstanding Section 2.1(b) above, no patent license is -granted: (1) for code that You delete from the Original Software, or (2) for -infringements caused by: (i) the modification of the Original Software, or -(ii) the combination of the Original Software with other software or -devices.  - -2.2. Contributor Grant. Conditioned upon Your compliance with -Section 3.1 below and subject to third party intellectual property claims, each -Contributor hereby grants You a world-wide, royalty-free, non-exclusive -license:  - -(a) under intellectual property rights (other than patent or -trademark) Licensable by Contributor to use, reproduce, modify, display, -perform, sublicense and distribute the Modifications created by such Contributor -(or portions thereof), either on an unmodified basis, with other Modifications, -as Covered Software and/or as part of a Larger Work; and  - -(b) under Patent -Claims infringed by the making, using, or selling of Modifications made by that -Contributor either alone and/or in combination with its Contributor Version (or -portions of such combination), to make, use, sell, offer for sale, have made, -and/or otherwise dispose of: (1) Modifications made by that Contributor (or -portions thereof); and (2) the combination of Modifications made by that -Contributor with its Contributor Version (or portions of such -combination).  - -(c) The licenses granted in Sections 2.2(a) and 2.2(b) are -effective on the date Contributor first distributes or otherwise makes the -Modifications available to a third party. - -(d) Notwithstanding Section 2.2(b) -above, no patent license is granted: (1) for any code that Contributor has -deleted from the Contributor Version; (2) for infringements caused by: -(i) third party modifications of Contributor Version, or (ii) the combination -of Modifications made by that Contributor with other software (except as part of -the Contributor Version) or other devices; or (3) under Patent Claims infringed -by Covered Software in the absence of Modifications made by that -Contributor.  - -3. Distribution Obligations.  - -3.1. Availability of Source -Code. Any Covered Software that You distribute or otherwise make available in -Executable form must also be made available in Source Code form and that Source -Code form must be distributed only under the terms of this License. You must -include a copy of this License with every copy of the Source Code form of the -Covered Software You distribute or otherwise make available. You must inform -recipients of any such Covered Software in Executable form as to how they can -obtain such Covered Software in Source Code form in a reasonable manner on or -through a medium customarily used for software exchange.  - -3.2. -Modifications. The Modifications that You create or to which You contribute are -governed by the terms of this License. You represent that You believe Your -Modifications are Your original creation(s) and/or You have sufficient rights to -grant the rights conveyed by this License.  - -3.3. Required Notices. You must -include a notice in each of Your Modifications that identifies You as the -Contributor of the Modification. You may not remove or alter any copyright, -patent or trademark notices contained within the Covered Software, or any -notices of licensing or any descriptive text giving attribution to any -Contributor or the Initial Developer.  - -3.4. Application of Additional Terms. -You may not offer or impose any terms on any Covered Software in Source Code -form that alters or restricts the applicable version of this License or the -recipients rights hereunder. You may choose to offer, and to charge a fee for, -warranty, support, indemnity or liability obligations to one or more recipients -of Covered Software. However, you may do so only on Your own behalf, and not on -behalf of the Initial Developer or any Contributor. You must make it absolutely -clear that any such warranty, support, indemnity or liability obligation is -offered by You alone, and You hereby agree to indemnify the Initial Developer -and every Contributor for any liability incurred by the Initial Developer or -such Contributor as a result of warranty, support, indemnity or liability terms -You offer. - -3.5. Distribution of Executable Versions. You may distribute the -Executable form of the Covered Software under the terms of this License or under -the terms of a license of Your choice, which may contain terms different from -this License, provided that You are in compliance with the terms of this License -and that the license for the Executable form does not attempt to limit or alter -the recipients rights in the Source Code form from the rights set forth in this -License. If You distribute the Covered Software in Executable form under a -different license, You must make it absolutely clear that any terms which differ -from this License are offered by You alone, not by the Initial Developer or -Contributor. You hereby agree to indemnify the Initial Developer and every -Contributor for any liability incurred by the Initial Developer or such -Contributor as a result of any such terms You offer.  - -3.6. Larger Works. You -may create a Larger Work by combining Covered Software with other code not -governed by the terms of this License and distribute the Larger Work as a single -product. In such a case, You must make sure the requirements of this License are -fulfilled for the Covered Software.  - -4. Versions of the License.  - -4.1. -New Versions. Sun Microsystems, Inc. is the initial license steward and may -publish revised and/or new versions of this License from time to time. Each -version will be given a distinguishing version number. Except as provided in -Section 4.3, no one other than the license steward has the right to modify this -License.  - -4.2. Effect of New Versions. You may always continue to use, -distribute or otherwise make the Covered Software available under the terms of -the version of the License under which You originally received the Covered -Software. If the Initial Developer includes a notice in the Original Software -prohibiting it from being distributed or otherwise made available under any -subsequent version of the License, You must distribute and make the Covered -Software available under the terms of the version of the License under which You -originally received the Covered Software. Otherwise, You may also choose to use, -distribute or otherwise make the Covered Software available under the terms of -any subsequent version of the License published by the license -steward.  - -4.3. Modified Versions. When You are an Initial Developer and You -want to create a new license for Your Original Software, You may create and use -a modified version of this License if You: (a) rename the license and remove -any references to the name of the license steward (except to note that the -license differs from this License); and (b) otherwise make it clear that the -license contains terms which differ from this License.  - -5. DISCLAIMER OF WARRANTY. - -COVERED SOFTWARE IS PROVIDED UNDER THIS LICENSE ON AN AS IS BASIS, -WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, WITHOUT -LIMITATION, WARRANTIES THAT THE COVERED SOFTWARE IS FREE OF DEFECTS, -MERCHANTABLE, FIT FOR A PARTICULAR PURPOSE OR NON-INFRINGING. THE ENTIRE RISK AS -TO THE QUALITY AND PERFORMANCE OF THE COVERED SOFTWARE IS WITH YOU. SHOULD ANY -COVERED SOFTWARE PROVE DEFECTIVE IN ANY RESPECT, YOU (NOT THE INITIAL DEVELOPER -OR ANY OTHER CONTRIBUTOR) ASSUME THE COST OF ANY NECESSARY SERVICING, REPAIR OR -CORRECTION. THIS DISCLAIMER OF WARRANTY CONSTITUTES AN ESSENTIAL PART OF THIS -LICENSE. NO USE OF ANY COVERED SOFTWARE IS AUTHORIZED HEREUNDER EXCEPT UNDER -THIS DISCLAIMER.  - -6. TERMINATION.  - -6.1. This License and the rights -granted hereunder will terminate automatically if You fail to comply with terms -herein and fail to cure such breach within 30 days of becoming aware of the -breach. Provisions which, by their nature, must remain in effect beyond the -termination of this License shall survive.  - -6.2. If You assert a patent -infringement claim (excluding declaratory judgment actions) against Initial -Developer or a Contributor (the Initial Developer or Contributor against whom -You assert such claim is referred to as Participant) alleging that the -Participant Software (meaning the Contributor Version where the Participant is a -Contributor or the Original Software where the Participant is the Initial -Developer) directly or indirectly infringes any patent, then any and all rights -granted directly or indirectly to You by such Participant, the Initial Developer -(if the Initial Developer is not the Participant) and all Contributors under -Sections 2.1 and/or 2.2 of this License shall, upon 60 days notice from -Participant terminate prospectively and automatically at the expiration of such -60 day notice period, unless if within such 60 day period You withdraw Your -claim with respect to the Participant Software against such Participant either -unilaterally or pursuant to a written agreement with Participant.  - -6.3. In -the event of termination under Sections 6.1 or 6.2 above, all end user licenses -that have been validly granted by You or any distributor hereunder prior to -termination (excluding licenses granted to You by any distributor) shall survive -termination.  - -7. LIMITATION OF LIABILITY. -UNDER NO CIRCUMSTANCES AND UNDER -NO LEGAL THEORY, WHETHER TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE, -SHALL YOU, THE INITIAL DEVELOPER, ANY OTHER CONTRIBUTOR, OR ANY DISTRIBUTOR OF -COVERED SOFTWARE, OR ANY SUPPLIER OF ANY OF SUCH PARTIES, BE LIABLE TO ANY -PERSON FOR ANY INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF ANY -CHARACTER INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOST PROFITS, LOSS OF -GOODWILL, WORK STOPPAGE, COMPUTER FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER -COMMERCIAL DAMAGES OR LOSSES, EVEN IF SUCH PARTY SHALL HAVE BEEN INFORMED OF THE -POSSIBILITY OF SUCH DAMAGES. THIS LIMITATION OF LIABILITY SHALL NOT APPLY TO -LIABILITY FOR DEATH OR PERSONAL INJURY RESULTING FROM SUCH PARTYS NEGLIGENCE TO -THE EXTENT APPLICABLE LAW PROHIBITS SUCH LIMITATION. SOME JURISDICTIONS DO NOT -ALLOW THE EXCLUSION OR LIMITATION OF INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO -THIS EXCLUSION AND LIMITATION MAY NOT APPLY TO YOU.  - -8. U.S. GOVERNMENT END USERS. - -The Covered Software is a commercial item, as that term is defined in -48 C.F.R. 2.101 (Oct. 1995), consisting of commercial computer software (as -that term is defined at 48 C.F.R.  252.227-7014(a)(1)) and commercial computer -software documentation as such terms are used in 48 C.F.R. 12.212 (Sept. -1995). Consistent with 48 C.F.R. 12.212 and 48 C.F.R. 227.7202-1 through -227.7202-4 (June 1995), all U.S. Government End Users acquire Covered Software -with only those rights set forth herein. This U.S. Government Rights clause is -in lieu of, and supersedes, any other FAR, DFAR, or other clause or provision -that addresses Government rights in computer software under this -License.  - -9. MISCELLANEOUS. -This License represents the complete agreement -concerning subject matter hereof. If any provision of this License is held to be -unenforceable, such provision shall be reformed only to the extent necessary to -make it enforceable. This License shall be governed by the law of the -jurisdiction specified in a notice contained within the Original Software -(except to the extent applicable law, if any, provides otherwise), excluding -such jurisdictions conflict-of-law provisions. Any litigation relating to this -License shall be subject to the jurisdiction of the courts located in the -jurisdiction and venue specified in a notice contained within the Original -Software, with the losing party responsible for costs, including, without -limitation, court costs and reasonable attorneys fees and expenses. The -application of the United Nations Convention on Contracts for the International -Sale of Goods is expressly excluded. Any law or regulation which provides that -the language of a contract shall be construed against the drafter shall not -apply to this License. You agree that You alone are responsible for compliance -with the United States export administration regulations (and the export control -laws and regulation of any other countries) when You use, distribute or -otherwise make available any Covered Software.  - -10. RESPONSIBILITY FOR CLAIMS. -As between Initial Developer and the Contributors, each party is -responsible for claims and damages arising, directly or indirectly, out of its -utilization of rights under this License and You agree to work with Initial -Developer and Contributors to distribute such responsibility on an equitable -basis. Nothing herein is intended or shall be deemed to constitute any admission -of liability.  - -The binary distribution of this product bundles these dependencies under the -following license: -jersey-client 1.19 -jersey-core 1.19 -jersey-grizzly2 1.19 -jersey-grizzly2-servlet 1.19 -jersey-json 1.19 -jersey-server 1.19 -jersey-servlet 1.19 -jersey-guice 1.19 -Jersey Test Framework - Grizzly 2 Module 1.19 -JAXB RI 2.2.3 -Java Architecture for XML Binding 2.2.11 -grizzly-framework 2.2.21 -grizzly-http 2.2.21 -grizzly-http-server 2.2.21 -grizzly-http-servlet 2.2.21 -grizzly-rcm 2.2.21 -JavaBeans Activation Framework 1.2.0 --------------------------------------------------------------------------------- -(CDDL 1.1) -COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL)Version 1.1 - -1. Definitions. - -1.1. “Contributor” means each individual or entity that creates or -contributes to the creation of Modifications. -1.2. “Contributor Version” means the combination of the Original Software, -prior Modifications used by a Contributor (if any), and the Modifications made -by that particular Contributor. -1.3. “Covered Software” means (a) the Original Software, or (b) -Modifications, or (c) the combination of files containing Original Software with -files containing Modifications, in each case including portions thereof. -1.4. “Executable” means the Covered Software in any form other than Source -Code. -1.5. “Initial Developer” means the individual or entity that first makes -Original Software available under this License. -1.6. “Larger Work” means a work which combines Covered Software or portions -thereof with code not governed by the terms of this License. -1.7. “License” means this document. -1.8. “Licensable” means having the right to grant, to the maximum extent -possible, whether at the time of the initial grant or subsequently acquired, any -and all of the rights conveyed herein. -1.9. “Modifications” means the Source Code and Executable form of any of the -following: -A. Any file that results from an addition to, deletion from or modification of -the contents of a file containing Original Software or previous Modifications; -B. Any new file that contains any part of the Original Software or previous -Modification; or -C. Any new file that is contributed or otherwise made available under the terms -of this License. -1.10. “Original Software” means the Source Code and Executable form of -computer software code that is originally released under this License. -1.11. “Patent Claims” means any patent claim(s), now owned or hereafter -acquired, including without limitation, method, process, and apparatus claims, -in any patent Licensable by grantor. -1.12. “Source Code” means (a) the common form of computer software code in -which modifications are made and (b) associated documentation included in or -with such code. -1.13. “You” (or “Your”) means an individual or a legal entity exercising -rights under, and complying with all of the terms of, this License. For legal -entities, “You” includes any entity which controls, is controlled by, or is -under common control with You. For purposes of this definition, “control” -means (a) the power, direct or indirect, to cause the direction or management of -such entity, whether by contract or otherwise, or (b) ownership of more than -fifty percent (50%) of the outstanding shares or beneficial ownership of such -entity. - -2. License Grants. - -2.1. The Initial Developer Grant. - -Conditioned upon Your compliance with Section 3.1 below and subject to -third party intellectual property claims, the Initial Developer hereby grants -You a world-wide, royalty-free, non-exclusive license: -(a) under intellectual -property rights (other than patent or trademark) Licensable by Initial -Developer, to use, reproduce, modify, display, perform, sublicense and -distribute the Original Software (or portions thereof), with or without -Modifications, and/or as part of a Larger Work; and -(b) under Patent Claims -infringed by the making, using or selling of Original Software, to make, have -made, use, practice, sell, and offer for sale, and/or otherwise dispose of the -Original Software (or portions thereof). -(c) The licenses granted in Sections -2.1(a) and (b) are effective on the date Initial Developer first distributes or -otherwise makes the Original Software available to a third party under the terms -of this License. -(d) Notwithstanding Section 2.1(b) above, no patent license is -granted: (1) for code that You delete from the Original Software, or (2) for -infringements caused by: (i) the modification of the Original Software, or (ii) -the combination of the Original Software with other software or devices. - -2.2. Contributor Grant. - -Conditioned upon Your compliance with Section 3.1 below and -subject to third party intellectual property claims, each Contributor hereby -grants You a world-wide, royalty-free, non-exclusive license: -(a) under -intellectual property rights (other than patent or trademark) Licensable by -Contributor to use, reproduce, modify, display, perform, sublicense and -distribute the Modifications created by such Contributor (or portions thereof), -either on an unmodified basis, with other Modifications, as Covered Software -and/or as part of a Larger Work; and -(b) under Patent Claims infringed by the -making, using, or selling of Modifications made by that Contributor either alone -and/or in combination with its Contributor Version (or portions of such -combination), to make, use, sell, offer for sale, have made, and/or otherwise -dispose of: (1) Modifications made by that Contributor (or portions thereof); -and (2) the combination of Modifications made by that Contributor with its -Contributor Version (or portions of such combination). -(c) The licenses granted -in Sections 2.2(a) and 2.2(b) are effective on the date Contributor first -distributes or otherwise makes the Modifications available to a third -party. -(d) Notwithstanding Section 2.2(b) above, no patent license is granted: -(1) for any code that Contributor has deleted from the Contributor Version; (2) -for infringements caused by: (i) third party modifications of Contributor -Version, or (ii) the combination of Modifications made by that Contributor with -other software (except as part of the Contributor Version) or other devices; or -(3) under Patent Claims infringed by Covered Software in the absence of -Modifications made by that Contributor. - -3. Distribution Obligations. - -3.1. Availability of Source Code. -Any Covered Software that You distribute or -otherwise make available in Executable form must also be made available in -Source Code form and that Source Code form must be distributed only under the -terms of this License. You must include a copy of this License with every copy -of the Source Code form of the Covered Software You distribute or otherwise make -available. You must inform recipients of any such Covered Software in Executable -form as to how they can obtain such Covered Software in Source Code form in a -reasonable manner on or through a medium customarily used for software -exchange. -3.2. Modifications. -The Modifications that You create or to which -You contribute are governed by the terms of this License. You represent that You -believe Your Modifications are Your original creation(s) and/or You have -sufficient rights to grant the rights conveyed by this License. -3.3. Required Notices. -You must include a notice in each of Your Modifications that -identifies You as the Contributor of the Modification. You may not remove or -alter any copyright, patent or trademark notices contained within the Covered -Software, or any notices of licensing or any descriptive text giving attribution -to any Contributor or the Initial Developer. -3.4. Application of Additional Terms. -You may not offer or impose any terms on any Covered Software in Source -Code form that alters or restricts the applicable version of this License or the -recipients' rights hereunder. You may choose to offer, and to charge a fee for, -warranty, support, indemnity or liability obligations to one or more recipients -of Covered Software. However, you may do so only on Your own behalf, and not on -behalf of the Initial Developer or any Contributor. You must make it absolutely -clear that any such warranty, support, indemnity or liability obligation is -offered by You alone, and You hereby agree to indemnify the Initial Developer -and every Contributor for any liability incurred by the Initial Developer or -such Contributor as a result of warranty, support, indemnity or liability terms -You offer. -3.5. Distribution of Executable Versions. -You may distribute the -Executable form of the Covered Software under the terms of this License or under -the terms of a license of Your choice, which may contain terms different from -this License, provided that You are in compliance with the terms of this License -and that the license for the Executable form does not attempt to limit or alter -the recipient's rights in the Source Code form from the rights set forth in -this License. If You distribute the Covered Software in Executable form under a -different license, You must make it absolutely clear that any terms which differ -from this License are offered by You alone, not by the Initial Developer or -Contributor. You hereby agree to indemnify the Initial Developer and every -Contributor for any liability incurred by the Initial Developer or such -Contributor as a result of any such terms You offer. -3.6. Larger Works. -You -may create a Larger Work by combining Covered Software with other code not -governed by the terms of this License and distribute the Larger Work as a single -product. In such a case, You must make sure the requirements of this License are -fulfilled for the Covered Software. - -4. Versions of the License. - -4.1. New Versions. -Oracle is the initial license steward and may publish revised and/or -new versions of this License from time to time. Each version will be given a -distinguishing version number. Except as provided in Section 4.3, no one other -than the license steward has the right to modify this License. -4.2. Effect of New Versions. -You may always continue to use, distribute or otherwise make the -Covered Software available under the terms of the version of the License under -which You originally received the Covered Software. If the Initial Developer -includes a notice in the Original Software prohibiting it from being distributed -or otherwise made available under any subsequent version of the License, You -must distribute and make the Covered Software available under the terms of the -version of the License under which You originally received the Covered Software. -Otherwise, You may also choose to use, distribute or otherwise make the Covered -Software available under the terms of any subsequent version of the License -published by the license steward. -4.3. Modified Versions. -When You are an -Initial Developer and You want to create a new license for Your Original -Software, You may create and use a modified version of this License if You: (a) -rename the license and remove any references to the name of the license steward -(except to note that the license differs from this License); and (b) otherwise -make it clear that the license contains terms which differ from this -License. - -5. DISCLAIMER OF WARRANTY. - -COVERED SOFTWARE IS PROVIDED UNDER THIS -LICENSE ON AN “AS IS” BASIS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED -OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES THAT THE COVERED SOFTWARE -IS FREE OF DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR PURPOSE OR -NON-INFRINGING. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE COVERED -SOFTWARE IS WITH YOU. SHOULD ANY COVERED SOFTWARE PROVE DEFECTIVE IN ANY -RESPECT, YOU (NOT THE INITIAL DEVELOPER OR ANY OTHER CONTRIBUTOR) ASSUME THE -COST OF ANY NECESSARY SERVICING, REPAIR OR CORRECTION. THIS DISCLAIMER OF -WARRANTY CONSTITUTES AN ESSENTIAL PART OF THIS LICENSE. NO USE OF ANY COVERED -SOFTWARE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS DISCLAIMER. - -6. TERMINATION. - -6.1. This License and the rights granted hereunder will -terminate automatically if You fail to comply with terms herein and fail to cure -such breach within 30 days of becoming aware of the breach. Provisions which, by -their nature, must remain in effect beyond the termination of this License shall -survive. -6.2. If You assert a patent infringement claim (excluding declaratory -judgment actions) against Initial Developer or a Contributor (the Initial -Developer or Contributor against whom You assert such claim is referred to as -“Participant”) alleging that the Participant Software (meaning the -Contributor Version where the Participant is a Contributor or the Original -Software where the Participant is the Initial Developer) directly or indirectly -infringes any patent, then any and all rights granted directly or indirectly to -You by such Participant, the Initial Developer (if the Initial Developer is not -the Participant) and all Contributors under Sections 2.1 and/or 2.2 of this -License shall, upon 60 days notice from Participant terminate prospectively and -automatically at the expiration of such 60 day notice period, unless if within -such 60 day period You withdraw Your claim with respect to the Participant -Software against such Participant either unilaterally or pursuant to a written -agreement with Participant. -6.3. If You assert a patent infringement claim -against Participant alleging that the Participant Software directly or -indirectly infringes any patent where such claim is resolved (such as by license -or settlement) prior to the initiation of patent infringement litigation, then -the reasonable value of the licenses granted by such Participant under Sections -2.1 or 2.2 shall be taken into account in determining the amount or value of any -payment or license. -6.4. In the event of termination under Sections 6.1 or 6.2 -above, all end user licenses that have been validly granted by You or any -distributor hereunder prior to termination (excluding licenses granted to You by -any distributor) shall survive termination. - -7. LIMITATION OF LIABILITY. - -UNDER NO CIRCUMSTANCES AND UNDER NO LEGAL THEORY, WHETHER TORT -(INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE, SHALL YOU, THE INITIAL -DEVELOPER, ANY OTHER CONTRIBUTOR, OR ANY DISTRIBUTOR OF COVERED SOFTWARE, OR ANY -SUPPLIER OF ANY OF SUCH PARTIES, BE LIABLE TO ANY PERSON FOR ANY INDIRECT, -SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF ANY CHARACTER INCLUDING, -WITHOUT LIMITATION, DAMAGES FOR LOSS OF GOODWILL, WORK STOPPAGE, COMPUTER -FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER COMMERCIAL DAMAGES OR LOSSES, EVEN -IF SUCH PARTY SHALL HAVE BEEN INFORMED OF THE POSSIBILITY OF SUCH DAMAGES. THIS -LIMITATION OF LIABILITY SHALL NOT APPLY TO LIABILITY FOR DEATH OR PERSONAL -INJURY RESULTING FROM SUCH PARTY'S NEGLIGENCE TO THE EXTENT APPLICABLE LAW -PROHIBITS SUCH LIMITATION. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR -LIMITATION OF INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO THIS EXCLUSION AND -LIMITATION MAY NOT APPLY TO YOU. - -8. U.S. GOVERNMENT END USERS. - -The Covered -Software is a “commercial item,” as that term is defined in 48 C.F.R. 2.101 -(Oct. 1995), consisting of “commercial computer software” (as that term is -defined at 48 C.F.R. § 252.227-7014(a)(1)) and “commercial computer software -documentation” as such terms are used in 48 C.F.R. 12.212 (Sept. 1995). -Consistent with 48 C.F.R. 12.212 and 48 C.F.R. 227.7202-1 through 227.7202-4 -(June 1995), all U.S. Government End Users acquire Covered Software with only -those rights set forth herein. This U.S. Government Rights clause is in lieu of, -and supersedes, any other FAR, DFAR, or other clause or provision that addresses -Government rights in computer software under this License. - -9. MISCELLANEOUS. - -This License represents the complete agreement concerning -subject matter hereof. If any provision of this License is held to be -unenforceable, such provision shall be reformed only to the extent necessary to -make it enforceable. This License shall be governed by the law of the -jurisdiction specified in a notice contained within the Original Software -(except to the extent applicable law, if any, provides otherwise), excluding -such jurisdiction's conflict-of-law provisions. Any litigation relating to this -License shall be subject to the jurisdiction of the courts located in the -jurisdiction and venue specified in a notice contained within the Original -Software, with the losing party responsible for costs, including, without -limitation, court costs and reasonable attorneys' fees and expenses. The -application of the United Nations Convention on Contracts for the International -Sale of Goods is expressly excluded. Any law or regulation which provides that -the language of a contract shall be construed against the drafter shall not -apply to this License. You agree that You alone are responsible for compliance -with the United States export administration regulations (and the export control -laws and regulation of any other countries) when You use, distribute or -otherwise make available any Covered Software. - -10. RESPONSIBILITY FOR CLAIMS. - -As between Initial Developer and the Contributors, each party is -responsible for claims and damages arising, directly or indirectly, out of its -utilization of rights under this License and You agree to work with Initial -Developer and Contributors to distribute such responsibility on an equitable -basis. Nothing herein is intended or shall be deemed to constitute any admission -of liability. - -The binary distribution of this product bundles these dependencies under the -following license: -Protocol Buffer Java API 2.5.0 --------------------------------------------------------------------------------- -This license applies to all parts of Protocol Buffers except the following: - - - Atomicops support for generic gcc, located in - src/google/protobuf/stubs/atomicops_internals_generic_gcc.h. - This file is copyrighted by Red Hat Inc. - - - Atomicops support for AIX/POWER, located in - src/google/protobuf/stubs/atomicops_internals_power.h. - This file is copyrighted by Bloomberg Finance LP. - -Copyright 2014, Google Inc. All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: - - * Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above -copyright notice, this list of conditions and the following disclaimer -in the documentation and/or other materials provided with the -distribution. - * Neither the name of Google Inc. nor the names of its -contributors may be used to endorse or promote products derived from -this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -Code generated by the Protocol Buffer compiler is owned by the owner -of the input file used when generating it. This code is not -standalone and requires a support library to be linked with it. This -support library is itself covered by the above license. - -For: -XML Commons External Components XML APIs 1.3.04 --------------------------------------------------------------------------------- -By obtaining, using and/or copying this work, you (the licensee) agree that you -have read, understood, and will comply with the following terms and conditions. - -Permission to copy, modify, and distribute this software and its documentation, -with or without modification, for any purpose and without fee or royalty is -hereby granted, provided that you include the following on ALL copies of the -software and documentation or portions thereof, including modifications: -- The full text of this NOTICE in a location viewable to users of the -redistributed or derivative work. -- Any pre-existing intellectual property disclaimers, notices, or terms and -conditions. If none exist, the W3C Software Short Notice should be included -(hypertext is preferred, text is permitted) within the body of any redistributed -or derivative code. -- Notice of any changes or modifications to the files, including the date changes -were made. (We recommend you provide URIs to the location from which the code is -derived.) - -The binary distribution of this product bundles these dependencies under the -following license: -Eclipse JDT Core 3.1.1 --------------------------------------------------------------------------------- -(EPL v1.0) -Eclipse Public License - v 1.0 - -THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC -LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM -CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT. - -1. DEFINITIONS - -"Contribution" means: - -a) in the case of the initial Contributor, the initial code and documentation -distributed under this Agreement, and -b) in the case of each subsequent Contributor: -i) changes to the Program, and -ii) additions to the Program; -where such changes and/or additions to the Program originate from and are -distributed by that particular Contributor. A Contribution 'originates' from a -Contributor if it was added to the Program by such Contributor itself or anyone -acting on such Contributor's behalf. Contributions do not include additions to -the Program which: (i) are separate modules of software distributed in -conjunction with the Program under their own license agreement, and (ii) are not -derivative works of the Program. -"Contributor" means any person or entity that distributes the Program. - -"Licensed Patents" mean patent claims licensable by a Contributor which are -necessarily infringed by the use or sale of its Contribution alone or when -combined with the Program. - -"Program" means the Contributions distributed in accordance with this Agreement. - -"Recipient" means anyone who receives the Program under this Agreement, -including all Contributors. - -2. GRANT OF RIGHTS - -a) Subject to the terms of this Agreement, each Contributor hereby grants -Recipient a non-exclusive, worldwide, royalty-free copyright license to -reproduce, prepare derivative works of, publicly display, publicly perform, -distribute and sublicense the Contribution of such Contributor, if any, and such -derivative works, in source code and object code form. -b) Subject to the terms of this Agreement, each Contributor hereby grants -Recipient a non-exclusive, worldwide, royalty-free patent license under Licensed -Patents to make, use, sell, offer to sell, import and otherwise transfer the -Contribution of such Contributor, if any, in source code and object code form. -This patent license shall apply to the combination of the Contribution and the -Program if, at the time the Contribution is added by the Contributor, such -addition of the Contribution causes such combination to be covered by the -Licensed Patents. The patent license shall not apply to any other combinations -which include the Contribution. No hardware per se is licensed hereunder. -c) Recipient understands that although each Contributor grants the licenses to -its Contributions set forth herein, no assurances are provided by any -Contributor that the Program does not infringe the patent or other intellectual -property rights of any other entity. Each Contributor disclaims any liability to -Recipient for claims brought by any other entity based on infringement of -intellectual property rights or otherwise. As a condition to exercising the -rights and licenses granted hereunder, each Recipient hereby assumes sole -responsibility to secure any other intellectual property rights needed, if any. -For example, if a third party patent license is required to allow Recipient to -distribute the Program, it is Recipient's responsibility to acquire that license -before distributing the Program. -d) Each Contributor represents that to its knowledge it has sufficient copyright -rights in its Contribution, if any, to grant the copyright license set forth in -this Agreement. -3. REQUIREMENTS - -A Contributor may choose to distribute the Program in object code form under its -own license agreement, provided that: - -a) it complies with the terms and conditions of this Agreement; and -b) its license agreement: -i) effectively disclaims on behalf of all Contributors all warranties and -conditions, express and implied, including warranties or conditions of title and -non-infringement, and implied warranties or conditions of merchantability and -fitness for a particular purpose; -ii) effectively excludes on behalf of all Contributors all liability for -damages, including direct, indirect, special, incidental and consequential -damages, such as lost profits; -iii) states that any provisions which differ from this Agreement are offered by -that Contributor alone and not by any other party; and -iv) states that source code for the Program is available from such Contributor, -and informs licensees how to obtain it in a reasonable manner on or through a -medium customarily used for software exchange. -When the Program is made available in source code form: - -a) it must be made available under this Agreement; and -b) a copy of this Agreement must be included with each copy of the Program. -Contributors may not remove or alter any copyright notices contained within the -Program. - -Each Contributor must identify itself as the originator of its Contribution, if -any, in a manner that reasonably allows subsequent Recipients to identify the -originator of the Contribution. - -4. COMMERCIAL DISTRIBUTION - -Commercial distributors of software may accept certain responsibilities with -respect to end users, business partners and the like. While this license is -intended to facilitate the commercial use of the Program, the Contributor who -includes the Program in a commercial product offering should do so in a manner -which does not create potential liability for other Contributors. Therefore, if -a Contributor includes the Program in a commercial product offering, such -Contributor ("Commercial Contributor") hereby agrees to defend and indemnify -every other Contributor ("Indemnified Contributor") against any losses, damages -and costs (collectively "Losses") arising from claims, lawsuits and other legal -actions brought by a third party against the Indemnified Contributor to the -extent caused by the acts or omissions of such Commercial Contributor in -connection with its distribution of the Program in a commercial product -offering. The obligations in this section do not apply to any claims or Losses -relating to any actual or alleged intellectual property infringement. In order -to qualify, an Indemnified Contributor must: a) promptly notify the Commercial -Contributor in writing of such claim, and b) allow the Commercial Contributor to -control, and cooperate with the Commercial Contributor in, the defense and any -related settlement negotiations. The Indemnified Contributor may participate in -any such claim at its own expense. - -For example, a Contributor might include the Program in a commercial product -offering, Product X. That Contributor is then a Commercial Contributor. If that -Commercial Contributor then makes performance claims, or offers warranties -related to Product X, those performance claims and warranties are such -Commercial Contributor's responsibility alone. Under this section, the -Commercial Contributor would have to defend claims against the other -Contributors related to those performance claims and warranties, and if a court -requires any other Contributor to pay any damages as a result, the Commercial -Contributor must pay those damages. - -5. NO WARRANTY - -EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON AN -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR -IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE, -NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Each -Recipient is solely responsible for determining the appropriateness of using and -distributing the Program and assumes all risks associated with its exercise of -rights under this Agreement , including but not limited to the risks and costs -of program errors, compliance with applicable laws, damage to or loss of data, -programs or equipment, and unavailability or interruption of operations. - -6. DISCLAIMER OF LIABILITY - -EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY -CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, -SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION LOST -PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, -STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY -OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE EXERCISE OF ANY RIGHTS -GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. - -7. GENERAL - -If any provision of this Agreement is invalid or unenforceable under applicable -law, it shall not affect the validity or enforceability of the remainder of the -terms of this Agreement, and without further action by the parties hereto, such -provision shall be reformed to the minimum extent necessary to make such -provision valid and enforceable. - -If Recipient institutes patent litigation against any entity (including a -cross-claim or counterclaim in a lawsuit) alleging that the Program itself -(excluding combinations of the Program with other software or hardware) -infringes such Recipient's patent(s), then such Recipient's rights granted under -Section 2(b) shall terminate as of the date such litigation is filed. - -All Recipient's rights under this Agreement shall terminate if it fails to -comply with any of the material terms or conditions of this Agreement and does -not cure such failure in a reasonable period of time after becoming aware of -such noncompliance. If all Recipient's rights under this Agreement terminate, -Recipient agrees to cease use and distribution of the Program as soon as -reasonably practicable. However, Recipient's obligations under this Agreement -and any licenses granted by Recipient relating to the Program shall continue and -survive. - -Everyone is permitted to copy and distribute copies of this Agreement, but in -order to avoid inconsistency the Agreement is copyrighted and may only be -modified in the following manner. The Agreement Steward reserves the right to -publish new versions (including revisions) of this Agreement from time to time. -No one other than the Agreement Steward has the right to modify this Agreement. -The Eclipse Foundation is the initial Agreement Steward. The Eclipse Foundation -may assign the responsibility to serve as the Agreement Steward to a suitable -separate entity. Each new version of the Agreement will be given a -distinguishing version number. The Program (including Contributions) may always -be distributed subject to the version of the Agreement under which it was -received. In addition, after a new version of the Agreement is published, -Contributor may elect to distribute the Program (including its Contributions) -under the new version. Except as expressly stated in Sections 2(a) and 2(b) -above, Recipient receives no rights or licenses to the intellectual property of -any Contributor under this Agreement, whether expressly, by implication, -estoppel or otherwise. All rights in the Program not expressly granted under -this Agreement are reserved. - -This Agreement is governed by the laws of the State of New York and the -intellectual property laws of the United States of America. No party to this -Agreement will bring a legal action under this Agreement more than one year -after the cause of action arose. Each party waives its rights to a jury trial in -any resulting litigation. - -The binary distribution of this product bundles these dependencies under the -following license: -JSch 0.1.54 -ParaNamer Core 2.3 -JLine 0.9.94 -leveldbjni-all 1.8 -Hamcrest Core 1.3 -ASM Core 5.0.4 -ASM Commons 5.0.2 -ASM Tree 5.0.2 --------------------------------------------------------------------------------- -(3-clause BSD) -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - * Neither the name of the nor the - names of its contributors may be used to endorse or promote products - derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY -DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -The binary distribution of this product bundles these dependencies under the -following license: -FindBugs-jsr305 3.0.0 -dnsjava 2.1.7, Copyright (c) 1998-2011, Brian Wellington. All rights reserved. --------------------------------------------------------------------------------- -(2-clause BSD) -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: - -1. Redistributions of source code must retain the above copyright notice, this - list of conditions and the following disclaimer. -2. Redistributions in binary form must reproduce the above copyright notice, - this list of conditions and the following disclaimer in the documentation - and/or other materials provided with the distribution. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR -ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -The views and conclusions contained in the software and documentation are those -of the authors and should not be interpreted as representing official policies, -either expressed or implied, of the FreeBSD Project. - -The binary distribution of this product bundles these dependencies under the -following license: -"Java Concurrency in Practice" book annotations 1.0 --------------------------------------------------------------------------------- -(CCAL v2.5) -THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE COMMONS -PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY COPYRIGHT AND/OR -OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZED UNDER THIS -LICENSE OR COPYRIGHT LAW IS PROHIBITED. - -BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE -BOUND BY THE TERMS OF THIS LICENSE. THE LICENSOR GRANTS YOU THE RIGHTS CONTAINED -HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS. - -1. Definitions - -"Collective Work" means a work, such as a periodical issue, anthology or -encyclopedia, in which the Work in its entirety in unmodified form, along with a -number of other contributions, constituting separate and independent works in -themselves, are assembled into a collective whole. A work that constitutes a -Collective Work will not be considered a Derivative Work (as defined below) for -the purposes of this License. -"Derivative Work" means a work based upon the Work or upon the Work and other -pre-existing works, such as a translation, musical arrangement, dramatization, -fictionalization, motion picture version, sound recording, art reproduction, -abridgment, condensation, or any other form in which the Work may be recast, -transformed, or adapted, except that a work that constitutes a Collective Work -will not be considered a Derivative Work for the purpose of this License. For -the avoidance of doubt, where the Work is a musical composition or sound -recording, the synchronization of the Work in timed-relation with a moving image -("synching") will be considered a Derivative Work for the purpose of this -License. -"Licensor" means the individual or entity that offers the Work under the terms -of this License. -"Original Author" means the individual or entity who created the Work. -"Work" means the copyrightable work of authorship offered under the terms of -this License. -"You" means an individual or entity exercising rights under this License who has -not previously violated the terms of this License with respect to the Work, or -who has received express permission from the Licensor to exercise rights under -this License despite a previous violation. -2. Fair Use Rights. Nothing in this license is intended to reduce, limit, or -restrict any rights arising from fair use, first sale or other limitations on -the exclusive rights of the copyright owner under copyright law or other -applicable laws. - -3. License Grant. Subject to the terms and conditions of this License, Licensor -hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the -duration of the applicable copyright) license to exercise the rights in the Work -as stated below: - -to reproduce the Work, to incorporate the Work into one or more Collective -Works, and to reproduce the Work as incorporated in the Collective Works; -to create and reproduce Derivative Works; -to distribute copies or phonorecords of, display publicly, perform publicly, and -perform publicly by means of a digital audio transmission the Work including as -incorporated in Collective Works; -to distribute copies or phonorecords of, display publicly, perform publicly, and -perform publicly by means of a digital audio transmission Derivative Works. -For the avoidance of doubt, where the work is a musical composition: - -Performance Royalties Under Blanket Licenses. Licensor waives the exclusive -right to collect, whether individually or via a performance rights society (e.g. -ASCAP, BMI, SESAC), royalties for the public performance or public digital -performance (e.g. webcast) of the Work. -Mechanical Rights and Statutory Royalties. Licensor waives the exclusive right -to collect, whether individually or via a music rights agency or designated -agent (e.g. Harry Fox Agency), royalties for any phonorecord You create from the -Work ("cover version") and distribute, subject to the compulsory license created -by 17 USC Section 115 of the US Copyright Act (or the equivalent in other -jurisdictions). -Webcasting Rights and Statutory Royalties. For the avoidance of doubt, where the -Work is a sound recording, Licensor waives the exclusive right to collect, -whether individually or via a performance-rights society (e.g. SoundExchange), -royalties for the public digital performance (e.g. webcast) of the Work, subject -to the compulsory license created by 17 USC Section 114 of the US Copyright Act -(or the equivalent in other jurisdictions). -The above rights may be exercised in all media and formats whether now known or -hereafter devised. The above rights include the right to make such modifications -as are technically necessary to exercise the rights in other media and formats. -All rights not expressly granted by Licensor are hereby reserved. - -4. Restrictions.The license granted in Section 3 above is expressly made subject -to and limited by the following restrictions: - -You may distribute, publicly display, publicly perform, or publicly digitally -perform the Work only under the terms of this License, and You must include a -copy of, or the Uniform Resource Identifier for, this License with every copy or -phonorecord of the Work You distribute, publicly display, publicly perform, or -publicly digitally perform. You may not offer or impose any terms on the Work -that alter or restrict the terms of this License or the recipients' exercise of -the rights granted hereunder. You may not sublicense the Work. You must keep -intact all notices that refer to this License and to the disclaimer of -warranties. You may not distribute, publicly display, publicly perform, or -publicly digitally perform the Work with any technological measures that control -access or use of the Work in a manner inconsistent with the terms of this -License Agreement. The above applies to the Work as incorporated in a Collective -Work, but this does not require the Collective Work apart from the Work itself -to be made subject to the terms of this License. If You create a Collective -Work, upon notice from any Licensor You must, to the extent practicable, remove -from the Collective Work any credit as required by clause 4(b), as requested. If -You create a Derivative Work, upon notice from any Licensor You must, to the -extent practicable, remove from the Derivative Work any credit as required by -clause 4(b), as requested. -If you distribute, publicly display, publicly perform, or publicly digitally -perform the Work or any Derivative Works or Collective Works, You must keep -intact all copyright notices for the Work and provide, reasonable to the medium -or means You are utilizing: (i) the name of the Original Author (or pseudonym, -if applicable) if supplied, and/or (ii) if the Original Author and/or Licensor -designate another party or parties (e.g. a sponsor institute, publishing entity, -journal) for attribution in Licensor's copyright notice, terms of service or by -other reasonable means, the name of such party or parties; the title of the Work -if supplied; to the extent reasonably practicable, the Uniform Resource -Identifier, if any, that Licensor specifies to be associated with the Work, -unless such URI does not refer to the copyright notice or licensing information -for the Work; and in the case of a Derivative Work, a credit identifying the use -of the Work in the Derivative Work (e.g., "French translation of the Work by -Original Author," or "Screenplay based on original Work by Original Author"). -Such credit may be implemented in any reasonable manner; provided, however, that -in the case of a Derivative Work or Collective Work, at a minimum such credit -will appear where any other comparable authorship credit appears and in a manner -at least as prominent as such other comparable authorship credit. -5. Representations, Warranties and Disclaimer - -UNLESS OTHERWISE MUTUALLY AGREED TO BY THE PARTIES IN WRITING, LICENSOR OFFERS -THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING -THE WORK, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT -LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR -PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, -OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE. SOME -JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO SUCH -EXCLUSION MAY NOT APPLY TO YOU. - -6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, IN -NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY SPECIAL, -INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS -LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE -POSSIBILITY OF SUCH DAMAGES. - -7. Termination - -This License and the rights granted hereunder will terminate automatically upon -any breach by You of the terms of this License. Individuals or entities who have -received Derivative Works or Collective Works from You under this License, -however, will not have their licenses terminated provided such individuals or -entities remain in full compliance with those licenses. Sections 1, 2, 5, 6, 7, -and 8 will survive any termination of this License. -Subject to the above terms and conditions, the license granted here is perpetual -(for the duration of the applicable copyright in the Work). Notwithstanding the -above, Licensor reserves the right to release the Work under different license -terms or to stop distributing the Work at any time; provided, however that any -such election will not serve to withdraw this License (or any other license that -has been, or is required to be, granted under the terms of this License), and -this License will continue in full force and effect unless terminated as stated -above. -8. Miscellaneous - -Each time You distribute or publicly digitally perform the Work or a Collective -Work, the Licensor offers to the recipient a license to the Work on the same -terms and conditions as the license granted to You under this License. -Each time You distribute or publicly digitally perform a Derivative Work, -Licensor offers to the recipient a license to the original Work on the same -terms and conditions as the license granted to You under this License. -If any provision of this License is invalid or unenforceable under applicable -law, it shall not affect the validity or enforceability of the remainder of the -terms of this License, and without further action by the parties to this -agreement, such provision shall be reformed to the minimum extent necessary to -make such provision valid and enforceable. -No term or provision of this License shall be deemed waived and no breach -consented to unless such waiver or consent shall be in writing and signed by the -party to be charged with such waiver or consent. -This License constitutes the entire agreement between the parties with respect -to the Work licensed here. There are no understandings, agreements or -representations with respect to the Work not specified here. Licensor shall not -be bound by any additional provisions that may appear in any communication from -You. This License may not be modified without the mutual written agreement of -the Licensor and You. - -The binary distribution of this product bundles these dependencies under the -following license: -jamon-runtime 2.4.1 --------------------------------------------------------------------------------- -(MPL 2.0) - Mozilla Public License - Version 2.0 - -1. Definitions - -1.1. “Contributor” -means each individual or legal entity that creates, contributes to the creation -of, or owns Covered Software. - -1.2. “Contributor Version” -means the combination of the Contributions of others (if any) used by a -Contributor and that particular Contributor’s Contribution. - -1.3. “Contribution” -means Covered Software of a particular Contributor. - -1.4. “Covered Software” -means Source Code Form to which the initial Contributor has attached the notice -in Exhibit A, the Executable Form of such Source Code Form, and Modifications of -such Source Code Form, in each case including portions thereof. - -1.5. “Incompatible With Secondary Licenses” -means - -that the initial Contributor has attached the notice described in Exhibit B to -the Covered Software; or - -that the Covered Software was made available under the terms of version 1.1 or -earlier of the License, but not also under the terms of a Secondary License. - -1.6. “Executable Form” -means any form of the work other than Source Code Form. - -1.7. “Larger Work” -means a work that combines Covered Software with other material, in a separate -file or files, that is not Covered Software. - -1.8. “License” -means this document. - -1.9. “Licensable” -means having the right to grant, to the maximum extent possible, whether at the -time of the initial grant or subsequently, any and all of the rights conveyed by -this License. - -1.10. “Modifications” -means any of the following: - -any file in Source Code Form that results from an addition to, deletion from, or -modification of the contents of Covered Software; or - -any new file in Source Code Form that contains any Covered Software. - -1.11. “Patent Claims” of a Contributor -means any patent claim(s), including without limitation, method, process, and -apparatus claims, in any patent Licensable by such Contributor that would be -infringed, but for the grant of the License, by the making, using, selling, -offering for sale, having made, import, or transfer of either its Contributions -or its Contributor Version. - -1.12. “Secondary License” -means either the GNU General Public License, Version 2.0, the GNU Lesser General -Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, -or any later versions of those licenses. - -1.13. “Source Code Form” -means the form of the work preferred for making modifications. - -1.14. “You” (or “Your”) -means an individual or a legal entity exercising rights under this License. For -legal entities, “You” includes any entity that controls, is controlled by, -or is under common control with You. For purposes of this definition, -“control” means (a) the power, direct or indirect, to cause the direction or -management of such entity, whether by contract or otherwise, or (b) ownership of -more than fifty percent (50%) of the outstanding shares or beneficial ownership -of such entity. - -2. License Grants and Conditions - -2.1. Grants - -Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive -license: - -under intellectual property rights (other than patent or trademark) Licensable -by such Contributor to use, reproduce, make available, modify, display, perform, -distribute, and otherwise exploit its Contributions, either on an unmodified -basis, with Modifications, or as part of a Larger Work; and - -under Patent Claims of such Contributor to make, use, sell, offer for sale, have -made, import, and otherwise transfer either its Contributions or its Contributor -Version. - -2.2. Effective Date - -The licenses granted in Section 2.1 with respect to any Contribution become -effective for each Contribution on the date the Contributor first distributes -such Contribution. - -2.3. Limitations on Grant Scope - -The licenses granted in this Section 2 are the only rights granted under this -License. No additional rights or licenses will be implied from the distribution -or licensing of Covered Software under this License. Notwithstanding Section -2.1(b) above, no patent license is granted by a Contributor: - -for any code that a Contributor has removed from Covered Software; or - -for infringements caused by: (i) Your and any other third party’s -modifications of Covered Software, or (ii) the combination of its Contributions -with other software (except as part of its Contributor Version); or - -under Patent Claims infringed by Covered Software in the absence of its -Contributions. - -This License does not grant any rights in the trademarks, service marks, or -logos of any Contributor (except as may be necessary to comply with the notice -requirements in Section 3.4). - -2.4. Subsequent Licenses - -No Contributor makes additional grants as a result of Your choice to distribute -the Covered Software under a subsequent version of this License (see Section -10.2) or under the terms of a Secondary License (if permitted under the terms of -Section 3.3). - -2.5. Representation - -Each Contributor represents that the Contributor believes its Contributions are -its original creation(s) or it has sufficient rights to grant the rights to its -Contributions conveyed by this License. - -2.6. Fair Use - -This License is not intended to limit any rights You have under applicable -copyright doctrines of fair use, fair dealing, or other equivalents. - -2.7. Conditions - -Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in -Section 2.1. - -3. Responsibilities - -3.1. Distribution of Source Form - -All distribution of Covered Software in Source Code Form, including any -Modifications that You create or to which You contribute, must be under the -terms of this License. You must inform recipients that the Source Code Form of -the Covered Software is governed by the terms of this License, and how they can -obtain a copy of this License. You may not attempt to alter or restrict the -recipients’ rights in the Source Code Form. - -3.2. Distribution of Executable Form - -If You distribute Covered Software in Executable Form then: - -such Covered Software must also be made available in Source Code Form, as -described in Section 3.1, and You must inform recipients of the Executable Form -how they can obtain a copy of such Source Code Form by reasonable means in a -timely manner, at a charge no more than the cost of distribution to the -recipient; and - -You may distribute such Executable Form under the terms of this License, or -sublicense it under different terms, provided that the license for the -Executable Form does not attempt to limit or alter the recipients’ rights in -the Source Code Form under this License. - -3.3. Distribution of a Larger Work - -You may create and distribute a Larger Work under terms of Your choice, provided -that You also comply with the requirements of this License for the Covered -Software. If the Larger Work is a combination of Covered Software with a work -governed by one or more Secondary Licenses, and the Covered Software is not -Incompatible With Secondary Licenses, this License permits You to additionally -distribute such Covered Software under the terms of such Secondary License(s), -so that the recipient of the Larger Work may, at their option, further -distribute the Covered Software under the terms of either this License or such -Secondary License(s). - -3.4. Notices - -You may not remove or alter the substance of any license notices (including -copyright notices, patent notices, disclaimers of warranty, or limitations of -liability) contained within the Source Code Form of the Covered Software, except -that You may alter any license notices to the extent required to remedy known -factual inaccuracies. - -3.5. Application of Additional Terms - -You may choose to offer, and to charge a fee for, warranty, support, indemnity -or liability obligations to one or more recipients of Covered Software. However, -You may do so only on Your own behalf, and not on behalf of any Contributor. You -must make it absolutely clear that any such warranty, support, indemnity, or -liability obligation is offered by You alone, and You hereby agree to indemnify -every Contributor for any liability incurred by such Contributor as a result of -warranty, support, indemnity or liability terms You offer. You may include -additional disclaimers of warranty and limitations of liability specific to any -jurisdiction. - -4. Inability to Comply Due to Statute or Regulation - -If it is impossible for You to comply with any of the terms of this License with -respect to some or all of the Covered Software due to statute, judicial order, -or regulation then You must: (a) comply with the terms of this License to the -maximum extent possible; and (b) describe the limitations and the code they -affect. Such description must be placed in a text file included with all -distributions of the Covered Software under this License. Except to the extent -prohibited by statute or regulation, such description must be sufficiently -detailed for a recipient of ordinary skill to be able to understand it. - -5. Termination - -5.1. The rights granted under this License will terminate automatically if You -fail to comply with any of its terms. However, if You become compliant, then the -rights granted under this License from a particular Contributor are reinstated -(a) provisionally, unless and until such Contributor explicitly and finally -terminates Your grants, and (b) on an ongoing basis, if such Contributor fails -to notify You of the non-compliance by some reasonable means prior to 60 days -after You have come back into compliance. Moreover, Your grants from a -particular Contributor are reinstated on an ongoing basis if such Contributor -notifies You of the non-compliance by some reasonable means, this is the first -time You have received notice of non-compliance with this License from such -Contributor, and You become compliant prior to 30 days after Your receipt of the -notice. - -5.2. If You initiate litigation against any entity by asserting a patent -infringement claim (excluding declaratory judgment actions, counter-claims, and -cross-claims) alleging that a Contributor Version directly or indirectly -infringes any patent, then the rights granted to You by any and all Contributors -for the Covered Software under Section 2.1 of this License shall terminate. - -5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user -license agreements (excluding distributors and resellers) which have been -validly granted by You or Your distributors under this License prior to -termination shall survive termination. - -6. Disclaimer of Warranty - -Covered Software is provided under this License on an “as is” basis, without -warranty of any kind, either expressed, implied, or statutory, including, -without limitation, warranties that the Covered Software is free of defects, -merchantable, fit for a particular purpose or non-infringing. The entire risk as -to the quality and performance of the Covered Software is with You. Should any -Covered Software prove defective in any respect, You (not any Contributor) -assume the cost of any necessary servicing, repair, or correction. This -disclaimer of warranty constitutes an essential part of this License. No use of -any Covered Software is authorized under this License except under this -disclaimer. - -7. Limitation of Liability - -Under no circumstances and under no legal theory, whether tort (including -negligence), contract, or otherwise, shall any Contributor, or anyone who -distributes Covered Software as permitted above, be liable to You for any -direct, indirect, special, incidental, or consequential damages of any character -including, without limitation, damages for lost profits, loss of goodwill, work -stoppage, computer failure or malfunction, or any and all other commercial -damages or losses, even if such party shall have been informed of the -possibility of such damages. This limitation of liability shall not apply to -liability for death or personal injury resulting from such party’s negligence -to the extent applicable law prohibits such limitation. Some jurisdictions do -not allow the exclusion or limitation of incidental or consequential damages, so -this exclusion and limitation may not apply to You. - -8. Litigation - -Any litigation relating to this License may be brought only in the courts of a -jurisdiction where the defendant maintains its principal place of business and -such litigation shall be governed by laws of that jurisdiction, without -reference to its conflict-of-law provisions. Nothing in this Section shall -prevent a party’s ability to bring cross-claims or counter-claims. - -9. Miscellaneous - -This License represents the complete agreement concerning the subject matter -hereof. If any provision of this License is held to be unenforceable, such -provision shall be reformed only to the extent necessary to make it enforceable. -Any law or regulation which provides that the language of a contract shall be -construed against the drafter shall not be used to construe this License against -a Contributor. - -10. Versions of the License - -10.1. New Versions - -Mozilla Foundation is the license steward. Except as provided in Section 10.3, -no one other than the license steward has the right to modify or publish new -versions of this License. Each version will be given a distinguishing version -number. - -10.2. Effect of New Versions - -You may distribute the Covered Software under the terms of the version of the -License under which You originally received the Covered Software, or under the -terms of any subsequent version published by the license steward. - -10.3. Modified Versions - -If you create software not governed by this License, and you want to create a -new license for such software, you may create and use a modified version of this -License if you rename the license and remove any references to the name of the -license steward (except to note that such modified license differs from this -License). - -10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses - -If You choose to distribute Source Code Form that is Incompatible With Secondary -Licenses under the terms of this version of the License, the notice described in -Exhibit B of this License must be attached. - -Exhibit A - Source Code Form License Notice - -This Source Code Form is subject to the terms of the Mozilla Public License, v. -2.0. If a copy of the MPL was not distributed with this file, You can obtain one -at https://mozilla.org/MPL/2.0/. - -If it is not possible or desirable to put the notice in a particular file, then -You may include the notice in a location (such as a LICENSE file in a relevant -directory) where a recipient would be likely to look for such a notice. - -You may add additional accurate notices of copyright ownership. - -Exhibit B - “Incompatible With Secondary Licenses” Notice - -This Source Code Form is “Incompatible With Secondary Licenses”, as defined -by the Mozilla Public License, v. 2.0. - -The binary distribution of this product bundles these dependencies under the -following license: -JDOM 1.1 --------------------------------------------------------------------------------- -/*-- - - Copyright (C) 2000-2004 Jason Hunter & Brett McLaughlin. - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - 1. Redistributions of source code must retain the above copyright - notice, this list of conditions, and the following disclaimer. - - 2. Redistributions in binary form must reproduce the above copyright - notice, this list of conditions, and the disclaimer that follows - these conditions in the documentation and/or other materials - provided with the distribution. - - 3. The name "JDOM" must not be used to endorse or promote products - derived from this software without prior written permission. For - written permission, please contact . - - 4. Products derived from this software may not be called "JDOM", nor - may "JDOM" appear in their name, without prior written permission - from the JDOM Project Management . - - In addition, we request (but do not require) that you include in the - end-user documentation provided with the redistribution and/or in the - software itself an acknowledgement equivalent to the following: - "This product includes software developed by the - JDOM Project (http://www.jdom.org/)." - Alternatively, the acknowledgment may be graphical using the logos - available at http://www.jdom.org/images/logos. - - THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED - WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES - OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE - DISCLAIMED. IN NO EVENT SHALL THE JDOM AUTHORS OR THE PROJECT - CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF - USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND - ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, - OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT - OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - SUCH DAMAGE. - - This software consists of voluntary contributions made by many - individuals on behalf of the JDOM Project and was originally - created by Jason Hunter and - Brett McLaughlin . For more information - on the JDOM Project, please see . - - */ - -The binary distribution of this product bundles these dependencies under the -following license: -Hbase Server 1.2.4 --------------------------------------------------------------------------------- -This project bundles a derivative image for our Orca Logo. This image is -available under the Creative Commons By Attribution 3.0 License. - - Creative Commons Legal Code - - Attribution 3.0 Unported - - CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE - LEGAL SERVICES. DISTRIBUTION OF THIS LICENSE DOES NOT CREATE AN - ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS - INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES - REGARDING THE INFORMATION PROVIDED, AND DISCLAIMS LIABILITY FOR - DAMAGES RESULTING FROM ITS USE. - - License - - THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE - COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY - COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS - AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED. - - BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE - TO BE BOUND BY THE TERMS OF THIS LICENSE. TO THE EXTENT THIS LICENSE MAY - BE CONSIDERED TO BE A CONTRACT, THE LICENSOR GRANTS YOU THE RIGHTS - CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND - CONDITIONS. - - 1. Definitions - - a. "Adaptation" means a work based upon the Work, or upon the Work and - other pre-existing works, such as a translation, adaptation, - derivative work, arrangement of music or other alterations of a - literary or artistic work, or phonogram or performance and includes - cinematographic adaptations or any other form in which the Work may be - recast, transformed, or adapted including in any form recognizably - derived from the original, except that a work that constitutes a - Collection will not be considered an Adaptation for the purpose of - this License. For the avoidance of doubt, where the Work is a musical - work, performance or phonogram, the synchronization of the Work in - timed-relation with a moving image ("synching") will be considered an - Adaptation for the purpose of this License. - b. "Collection" means a collection of literary or artistic works, such as - encyclopedias and anthologies, or performances, phonograms or - broadcasts, or other works or subject matter other than works listed - in Section 1(f) below, which, by reason of the selection and - arrangement of their contents, constitute intellectual creations, in - which the Work is included in its entirety in unmodified form along - with one or more other contributions, each constituting separate and - independent works in themselves, which together are assembled into a - collective whole. A work that constitutes a Collection will not be - considered an Adaptation (as defined above) for the purposes of this - License. - c. "Distribute" means to make available to the public the original and - copies of the Work or Adaptation, as appropriate, through sale or - other transfer of ownership. - d. "Licensor" means the individual, individuals, entity or entities that - offer(s) the Work under the terms of this License. - e. "Original Author" means, in the case of a literary or artistic work, - the individual, individuals, entity or entities who created the Work - or if no individual or entity can be identified, the publisher; and in - addition (i) in the case of a performance the actors, singers, - musicians, dancers, and other persons who act, sing, deliver, declaim, - play in, interpret or otherwise perform literary or artistic works or - expressions of folklore; (ii) in the case of a phonogram the producer - being the person or legal entity who first fixes the sounds of a - performance or other sounds; and, (iii) in the case of broadcasts, the - organization that transmits the broadcast. - f. "Work" means the literary and/or artistic work offered under the terms - of this License including without limitation any production in the - literary, scientific and artistic domain, whatever may be the mode or - form of its expression including digital form, such as a book, - pamphlet and other writing; a lecture, address, sermon or other work - of the same nature; a dramatic or dramatico-musical work; a - choreographic work or entertainment in dumb show; a musical - composition with or without words; a cinematographic work to which are - assimilated works expressed by a process analogous to cinematography; - a work of drawing, painting, architecture, sculpture, engraving or - lithography; a photographic work to which are assimilated works - expressed by a process analogous to photography; a work of applied - art; an illustration, map, plan, sketch or three-dimensional work - relative to geography, topography, architecture or science; a - performance; a broadcast; a phonogram; a compilation of data to the - extent it is protected as a copyrightable work; or a work performed by - a variety or circus performer to the extent it is not otherwise - considered a literary or artistic work. - g. "You" means an individual or entity exercising rights under this - License who has not previously violated the terms of this License with - respect to the Work, or who has received express permission from the - Licensor to exercise rights under this License despite a previous - violation. - h. "Publicly Perform" means to perform public recitations of the Work and - to communicate to the public those public recitations, by any means or - process, including by wire or wireless means or public digital - performances; to make available to the public Works in such a way that - members of the public may access these Works from a place and at a - place individually chosen by them; to perform the Work to the public - by any means or process and the communication to the public of the - performances of the Work, including by public digital performance; to - broadcast and rebroadcast the Work by any means including signs, - sounds or images. - i. "Reproduce" means to make copies of the Work by any means including - without limitation by sound or visual recordings and the right of - fixation and reproducing fixations of the Work, including storage of a - protected performance or phonogram in digital form or other electronic - medium. - - 2. Fair Dealing Rights. Nothing in this License is intended to reduce, - limit, or restrict any uses free from copyright or rights arising from - limitations or exceptions that are provided for in connection with the - copyright protection under copyright law or other applicable laws. - - 3. License Grant. Subject to the terms and conditions of this License, - Licensor hereby grants You a worldwide, royalty-free, non-exclusive, - perpetual (for the duration of the applicable copyright) license to - exercise the rights in the Work as stated below: - - a. to Reproduce the Work, to incorporate the Work into one or more - Collections, and to Reproduce the Work as incorporated in the - Collections; - b. to create and Reproduce Adaptations provided that any such Adaptation, - including any translation in any medium, takes reasonable steps to - clearly label, demarcate or otherwise identify that changes were made - to the original Work. For example, a translation could be marked "The - original work was translated from English to Spanish," or a - modification could indicate "The original work has been modified."; - c. to Distribute and Publicly Perform the Work including as incorporated - in Collections; and, - d. to Distribute and Publicly Perform Adaptations. - e. For the avoidance of doubt: - - i. Non-waivable Compulsory License Schemes. In those jurisdictions in - which the right to collect royalties through any statutory or - compulsory licensing scheme cannot be waived, the Licensor - reserves the exclusive right to collect such royalties for any - exercise by You of the rights granted under this License; - ii. Waivable Compulsory License Schemes. In those jurisdictions in - which the right to collect royalties through any statutory or - compulsory licensing scheme can be waived, the Licensor waives the - exclusive right to collect such royalties for any exercise by You - of the rights granted under this License; and, - iii. Voluntary License Schemes. The Licensor waives the right to - collect royalties, whether individually or, in the event that the - Licensor is a member of a collecting society that administers - voluntary licensing schemes, via that society, from any exercise - by You of the rights granted under this License. - - The above rights may be exercised in all media and formats whether now - known or hereafter devised. The above rights include the right to make - such modifications as are technically necessary to exercise the rights in - other media and formats. Subject to Section 8(f), all rights not expressly - granted by Licensor are hereby reserved. - - 4. Restrictions. The license granted in Section 3 above is expressly made - subject to and limited by the following restrictions: - - a. You may Distribute or Publicly Perform the Work only under the terms - of this License. You must include a copy of, or the Uniform Resource - Identifier (URI) for, this License with every copy of the Work You - Distribute or Publicly Perform. You may not offer or impose any terms - on the Work that restrict the terms of this License or the ability of - the recipient of the Work to exercise the rights granted to that - recipient under the terms of the License. You may not sublicense the - Work. You must keep intact all notices that refer to this License and - to the disclaimer of warranties with every copy of the Work You - Distribute or Publicly Perform. When You Distribute or Publicly - Perform the Work, You may not impose any effective technological - measures on the Work that restrict the ability of a recipient of the - Work from You to exercise the rights granted to that recipient under - the terms of the License. This Section 4(a) applies to the Work as - incorporated in a Collection, but this does not require the Collection - apart from the Work itself to be made subject to the terms of this - License. If You create a Collection, upon notice from any Licensor You - must, to the extent practicable, remove from the Collection any credit - as required by Section 4(b), as requested. If You create an - Adaptation, upon notice from any Licensor You must, to the extent - practicable, remove from the Adaptation any credit as required by - Section 4(b), as requested. - b. If You Distribute, or Publicly Perform the Work or any Adaptations or - Collections, You must, unless a request has been made pursuant to - Section 4(a), keep intact all copyright notices for the Work and - provide, reasonable to the medium or means You are utilizing: (i) the - name of the Original Author (or pseudonym, if applicable) if supplied, - and/or if the Original Author and/or Licensor designate another party - or parties (e.g., a sponsor institute, publishing entity, journal) for - attribution ("Attribution Parties") in Licensor's copyright notice, - terms of service or by other reasonable means, the name of such party - or parties; (ii) the title of the Work if supplied; (iii) to the - extent reasonably practicable, the URI, if any, that Licensor - specifies to be associated with the Work, unless such URI does not - refer to the copyright notice or licensing information for the Work; - and (iv) , consistent with Section 3(b), in the case of an Adaptation, - a credit identifying the use of the Work in the Adaptation (e.g., - "French translation of the Work by Original Author," or "Screenplay - based on original Work by Original Author"). The credit required by - this Section 4 (b) may be implemented in any reasonable manner; - provided, however, that in the case of a Adaptation or Collection, at - a minimum such credit will appear, if a credit for all contributing - authors of the Adaptation or Collection appears, then as part of these - credits and in a manner at least as prominent as the credits for the - other contributing authors. For the avoidance of doubt, You may only - use the credit required by this Section for the purpose of attribution - in the manner set out above and, by exercising Your rights under this - License, You may not implicitly or explicitly assert or imply any - connection with, sponsorship or endorsement by the Original Author, - Licensor and/or Attribution Parties, as appropriate, of You or Your - use of the Work, without the separate, express prior written - permission of the Original Author, Licensor and/or Attribution - Parties. - c. Except as otherwise agreed in writing by the Licensor or as may be - otherwise permitted by applicable law, if You Reproduce, Distribute or - Publicly Perform the Work either by itself or as part of any - Adaptations or Collections, You must not distort, mutilate, modify or - take other derogatory action in relation to the Work which would be - prejudicial to the Original Author's honor or reputation. Licensor - agrees that in those jurisdictions (e.g. Japan), in which any exercise - of the right granted in Section 3(b) of this License (the right to - make Adaptations) would be deemed to be a distortion, mutilation, - modification or other derogatory action prejudicial to the Original - Author's honor and reputation, the Licensor will waive or not assert, - as appropriate, this Section, to the fullest extent permitted by the - applicable national law, to enable You to reasonably exercise Your - right under Section 3(b) of this License (right to make Adaptations) - but not otherwise. - - 5. Representations, Warranties and Disclaimer - - UNLESS OTHERWISE MUTUALLY AGREED TO BY THE PARTIES IN WRITING, LICENSOR - OFFERS THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY - KIND CONCERNING THE WORK, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, - INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, - FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF - LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, - WHETHER OR NOT DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION - OF IMPLIED WARRANTIES, SO SUCH EXCLUSION MAY NOT APPLY TO YOU. - - 6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE - LAW, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR - ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES - ARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS - BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. - - 7. Termination - - a. This License and the rights granted hereunder will terminate - automatically upon any breach by You of the terms of this License. - Individuals or entities who have received Adaptations or Collections - from You under this License, however, will not have their licenses - terminated provided such individuals or entities remain in full - compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will - survive any termination of this License. - b. Subject to the above terms and conditions, the license granted here is - perpetual (for the duration of the applicable copyright in the Work). - Notwithstanding the above, Licensor reserves the right to release the - Work under different license terms or to stop distributing the Work at - any time; provided, however that any such election will not serve to - withdraw this License (or any other license that has been, or is - required to be, granted under the terms of this License), and this - License will continue in full force and effect unless terminated as - stated above. - - 8. Miscellaneous - - a. Each time You Distribute or Publicly Perform the Work or a Collection, - the Licensor offers to the recipient a license to the Work on the same - terms and conditions as the license granted to You under this License. - b. Each time You Distribute or Publicly Perform an Adaptation, Licensor - offers to the recipient a license to the original Work on the same - terms and conditions as the license granted to You under this License. - c. If any provision of this License is invalid or unenforceable under - applicable law, it shall not affect the validity or enforceability of - the remainder of the terms of this License, and without further action - by the parties to this agreement, such provision shall be reformed to - the minimum extent necessary to make such provision valid and - enforceable. - d. No term or provision of this License shall be deemed waived and no - breach consented to unless such waiver or consent shall be in writing - and signed by the party to be charged with such waiver or consent. - e. This License constitutes the entire agreement between the parties with - respect to the Work licensed here. There are no understandings, - agreements or representations with respect to the Work not specified - here. Licensor shall not be bound by any additional provisions that - may appear in any communication from You. This License may not be - modified without the mutual written agreement of the Licensor and You. - f. The rights granted under, and the subject matter referenced, in this - License were drafted utilizing the terminology of the Berne Convention - for the Protection of Literary and Artistic Works (as amended on - September 28, 1979), the Rome Convention of 1961, the WIPO Copyright - Treaty of 1996, the WIPO Performances and Phonograms Treaty of 1996 - and the Universal Copyright Convention (as revised on July 24, 1971). - These rights and subject matter take effect in the relevant - jurisdiction in which the License terms are sought to be enforced - according to the corresponding provisions of the implementation of - those treaty provisions in the applicable national law. If the - standard suite of rights granted under applicable copyright law - includes additional rights not granted under this License, such - additional rights are deemed to be included in the License; this - License is not intended to restrict the license of any rights under - applicable law. - - - Creative Commons Notice - - Creative Commons is not a party to this License, and makes no warranty - whatsoever in connection with the Work. Creative Commons will not be - liable to You or any party on any legal theory for any damages - whatsoever, including without limitation any general, special, - incidental or consequential damages arising in connection to this - license. Notwithstanding the foregoing two (2) sentences, if Creative - Commons has expressly identified itself as the Licensor hereunder, it - shall have all rights and obligations of Licensor. - - Except for the limited purpose of indicating to the public that the - Work is licensed under the CCPL, Creative Commons does not authorize - the use by either party of the trademark "Creative Commons" or any - related trademark or logo of Creative Commons without the prior - written consent of Creative Commons. Any permitted use will be in - compliance with Creative Commons' then-current trademark usage - guidelines, as may be published on its website or otherwise made - available upon request from time to time. For the avoidance of doubt, - this trademark restriction does not form part of this License. - - Creative Commons may be contacted at https://creativecommons.org/. - --------------------------------------------------------------------------------- - -For: /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server -/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL - -xterm.js 3.8.0 -The source and binary distribution of this product bundles these dependencies -under the following license: - -Copyright (c) 2017-2018, The xterm.js authors (https://github.com/xtermjs/xterm.js) -Copyright (c) 2014-2016, SourceLair Private Company (https://www.sourcelair.com) -Copyright (c) 2012-2013, Christopher Jeffrey (https://github.com/chjj/) - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -THE SOFTWARE. - --------------------------------------------------------------------------------- - -For: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs -/server/datanode/checker/AbstractFuture.java and -hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs -/server/datanode/checker/TimeoutFuture.java - -Copyright (C) 2007 The Guava Authors - -Licensed under the Apache License, Version 2.0 (the "License"); you may not -use this file except in compliance with the License. You may obtain a copy of -the License at - -http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -License for the specific language governing permissions and limitations under -the License. - --------------------------------------------------------------------------------- - -Jline 3.9.0 -The binary distribution of this product bundles these dependencies under the -following license: - -Copyright (c) 2002-2018, the original author or authors. -All rights reserved. - -http://www.opensource.org/licenses/bsd-license.php - -Redistribution and use in source and binary forms, with or -without modification, are permitted provided that the following -conditions are met: - -Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. - -Redistributions in binary form must reproduce the above copyright -notice, this list of conditions and the following disclaimer -in the documentation and/or other materials provided with -the distribution. - -Neither the name of JLine nor the names of its contributors -may be used to endorse or promote products derived from this -software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, -BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY -AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO -EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE -FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, -OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, -PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED -AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT -LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING -IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED -OF THE POSSIBILITY OF SUCH DAMAGE. +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL diff --git a/NOTICE-binary b/NOTICE-binary new file mode 100644 index 0000000000000..2f8a9241a8d00 --- /dev/null +++ b/NOTICE-binary @@ -0,0 +1,840 @@ +Apache Hadoop +Copyright 2006 and onwards The Apache Software Foundation. + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +Export Control Notice +--------------------- + +This distribution includes cryptographic software. The country in +which you currently reside may have restrictions on the import, +possession, use, and/or re-export to another country, of +encryption software. BEFORE using any encryption software, please +check your country's laws, regulations and policies concerning the +import, possession, or use, and re-export of encryption software, to +see if this is permitted. See for more +information. + +The U.S. Government Department of Commerce, Bureau of Industry and +Security (BIS), has classified this software as Export Commodity +Control Number (ECCN) 5D002.C.1, which includes information security +software using or performing cryptographic functions with asymmetric +algorithms. The form and manner of this Apache Software Foundation +distribution makes it eligible for export under the License Exception +ENC Technology Software Unrestricted (TSU) exception (see the BIS +Export Administration Regulations, Section 740.13) for both object +code and source code. + +The following provides more details on the included cryptographic software: + +This software uses the SSL libraries from the Jetty project written +by mortbay.org. +Hadoop Yarn Server Web Proxy uses the BouncyCastle Java +cryptography APIs written by the Legion of the Bouncy Castle Inc. + +// ------------------------------------------------------------------ +// NOTICE file corresponding to the section 4d of The Apache License, +// Version 2.0, in this case for +// ------------------------------------------------------------------ + + +Apache Yetus +Copyright 2008-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +--- +Additional licenses for the Apache Yetus Source/Website: +--- + + +See LICENSE for terms. + + + +Apache Avro +Copyright 2010 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +C JSON parsing provided by Jansson and +written by Petri Lehtinen. The original software is +available from http://www.digip.org/jansson/. + + +AWS SDK for Java +Copyright 2010-2014 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +This product includes software developed by +Amazon Technologies, Inc (http://www.amazon.com/). + +********************** +THIRD PARTY COMPONENTS +********************** +This software includes third party software subject to the following copyrights: +- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty. +- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc. + +The licenses for these third party components are included in LICENSE.txt + + +Apache Commons BeanUtils +Copyright 2000-2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons CLI +Copyright 2001-2009 The Apache Software Foundation + +This product includes software developed by +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Codec +Copyright 2002-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java +contains test data from http://aspell.net/test/orig/batch0.tab. +Copyright (C) 2002 Kevin Atkinson (kevina@gnu.org) + +=============================================================================== + +The content of package org.apache.commons.codec.language.bm has been translated +from the original php source code available at http://stevemorse.org/phoneticinfo.htm +with permission from the original authors. +Original source copyright: +Copyright (c) 2008 Alexander Beider & Stephen P. Morse. + + +Apache Commons Collections +Copyright 2001-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Compress +Copyright 2002-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (https://www.apache.org/). + +The files in the package org.apache.commons.compress.archivers.sevenz +were derived from the LZMA SDK, version 9.20 (C/ and CPP/7zip/), +which has been placed in the public domain: + +"LZMA SDK is placed in the public domain." (http://www.7-zip.org/sdk.html) + + +Apache Commons Configuration +Copyright 2001-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons CSV +Copyright 2005-2014 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +src/main/resources/contract.txt +This file was downloaded from http://www.ferc.gov/docs-filing/eqr/soft-tools/sample-csv/contract.txt and contains neither copyright notice nor license. + +src/main/resources/transaction.txt +This file was downloaded from http://www.ferc.gov/docs-filing/eqr/soft-tools/sample-csv/transaction.txt and contains neither copyright notice nor license. + +src/test/resources/CSVFileParser/bom.csv +src/test/resources/CSVFileParser/test.csv +src/test/resources/CSVFileParser/test_default.txt +src/test/resources/CSVFileParser/test_default_comment.txt +src/test/resources/CSVFileParser/test_rfc4180.txt +src/test/resources/CSVFileParser/test_rfc4180_trim.txt +src/test/resources/CSVFileParser/testCSV85.csv +src/test/resources/CSVFileParser/testCSV85_default.txt +src/test/resources/CSVFileParser/testCSV85_ignoreEmpty.txt +These files are used as test data and test result specifications. + + +Apache Commons Daemon +Copyright 1999-2013 The Apache Software Foundation + +This product includes software developed by +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Digester +Copyright 2001-2008 The Apache Software Foundation + +This product includes software developed by +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons IO +Copyright 2002-2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Lang +Copyright 2001-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +This product includes software from the Spring Framework, +under the Apache License 2.0 (see: StringUtils.containsWhitespace()) + + +Apache Commons Logging +Copyright 2003-2013 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Math +Copyright 2001-2012 The Apache Software Foundation + +This product includes software developed by +The Apache Software Foundation (http://www.apache.org/). + +=============================================================================== + +The BracketFinder (package org.apache.commons.math3.optimization.univariate) +and PowellOptimizer (package org.apache.commons.math3.optimization.general) +classes are based on the Python code in module "optimize.py" (version 0.5) +developed by Travis E. Oliphant for the SciPy library (http://www.scipy.org/) +Copyright © 2003-2009 SciPy Developers. +=============================================================================== + +The LinearConstraint, LinearObjectiveFunction, LinearOptimizer, +RelationShip, SimplexSolver and SimplexTableau classes in package +org.apache.commons.math3.optimization.linear include software developed by +Benjamin McCann (http://www.benmccann.com) and distributed with +the following copyright: Copyright 2009 Google Inc. +=============================================================================== + +This product includes software developed by the +University of Chicago, as Operator of Argonne National +Laboratory. +The LevenbergMarquardtOptimizer class in package +org.apache.commons.math3.optimization.general includes software +translated from the lmder, lmpar and qrsolv Fortran routines +from the Minpack package +Minpack Copyright Notice (1999) University of Chicago. All rights reserved +=============================================================================== + +The GraggBulirschStoerIntegrator class in package +org.apache.commons.math3.ode.nonstiff includes software translated +from the odex Fortran routine developed by E. Hairer and G. Wanner. +Original source copyright: +Copyright (c) 2004, Ernst Hairer +=============================================================================== + +The EigenDecompositionImpl class in package +org.apache.commons.math3.linear includes software translated +from some LAPACK Fortran routines. Original source copyright: +Copyright (c) 1992-2008 The University of Tennessee. All rights reserved. +=============================================================================== + +The MersenneTwister class in package org.apache.commons.math3.random +includes software translated from the 2002-01-26 version of +the Mersenne-Twister generator written in C by Makoto Matsumoto and Takuji +Nishimura. Original source copyright: +Copyright (C) 1997 - 2002, Makoto Matsumoto and Takuji Nishimura, +All rights reserved +=============================================================================== + +The LocalizedFormatsTest class in the unit tests is an adapted version of +the OrekitMessagesTest class from the orekit library distributed under the +terms of the Apache 2 licence. Original source copyright: +Copyright 2010 CS Systèmes d'Information +=============================================================================== + +The HermiteInterpolator class and its corresponding test have been imported from +the orekit library distributed under the terms of the Apache 2 licence. Original +source copyright: +Copyright 2010-2012 CS Systèmes d'Information +=============================================================================== + +The creation of the package "o.a.c.m.analysis.integration.gauss" was inspired +by an original code donated by Sébastien Brisard. +=============================================================================== + + +The complete text of licenses and disclaimers associated with the the original +sources enumerated above at the time of code translation are in the LICENSE.txt +file. + + +Apache Commons Net +Copyright 2001-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Text +Copyright 2014-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Validator +Copyright 2001-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Curator +Copyright 2013-2014 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Ehcache V3 +Copyright 2014-2016 Terracotta, Inc. + +The product includes software from the Apache Commons Lang project, +under the Apache License 2.0 (see: org.ehcache.impl.internal.classes.commonslang) + + +Apache Geronimo +Copyright 2003-2018 The Apache Software Foundation + +This product includes software developed by +The Apache Software Foundation (http://www.apache.org/). + + +Copyright 2014 The gRPC Authors + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +----------------------------------------------------------------------- + +This product contains a modified portion of 'OkHttp', an open source +HTTP & SPDY client for Android and Java applications, which can be obtained +at: + + * LICENSE: + * okhttp/third_party/okhttp/LICENSE (Apache License 2.0) + * HOMEPAGE: + * https://github.com/square/okhttp + * LOCATION_IN_GRPC: + * okhttp/third_party/okhttp + +This product contains a modified portion of 'Netty', an open source +networking library, which can be obtained at: + + * LICENSE: + * netty/third_party/netty/LICENSE.txt (Apache License 2.0) + * HOMEPAGE: + * https://netty.io + * LOCATION_IN_GRPC: + * netty/third_party/netty + + +Apache HBase +Copyright 2007-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +-- +This product incorporates portions of the 'Hadoop' project + +Copyright 2007-2009 The Apache Software Foundation + +Licensed under the Apache License v2.0 +-- +Our Orca logo we got here: http://www.vectorfree.com/jumping-orca +It is licensed Creative Commons Attribution 3.0. +See https://creativecommons.org/licenses/by/3.0/us/ +We changed the logo by stripping the colored background, inverting +it and then rotating it some. + +Later we found that vectorfree.com image is not properly licensed. +The original is owned by vectorportal.com. The original was +relicensed so we could use it as Creative Commons Attribution 3.0. +The license is bundled with the download available here: +http://www.vectorportal.com/subcategory/205/KILLER-WHALE-FREE-VECTOR.eps/ifile/9136/detailtest.asp +-- +This product includes portions of the Bootstrap project v3.0.0 + +Copyright 2013 Twitter, Inc. + +Licensed under the Apache License v2.0 + +This product uses the Glyphicons Halflings icon set. + +http://glyphicons.com/ + +Copyright Jan Kovařík + +Licensed under the Apache License v2.0 as a part of the Bootstrap project. + +-- +This product includes portions of the Guava project v14 and v21, specifically +'hbase-common/src/main/java/org/apache/hadoop/hbase/io/LimitInputStream.java' +'hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java' + +Copyright (C) 2007 The Guava Authors + +Licensed under the Apache License, Version 2.0 + + +Apache HTrace +Copyright 2016 The Apache Software Foundation + +This product includes software developed at The Apache Software +Foundation (http://www.apache.org/). + +In addition, this product includes software dependencies. See +the accompanying LICENSE.txt for a listing of dependencies +that are NOT Apache licensed (with pointers to their licensing) + +Apache HTrace includes an Apache Thrift connector to Zipkin. Zipkin +is a distributed tracing system that is Apache 2.0 Licensed. +Copyright 2012 Twitter, Inc. + + +Apache HttpComponents Client +Copyright 1999-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache HttpComponents Core +Copyright 2005-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +============================================================== + Jetty Web Container + Copyright 1995-2017 Mort Bay Consulting Pty Ltd. +============================================================== + +The Jetty Web Container is Copyright Mort Bay Consulting Pty Ltd +unless otherwise noted. + +Jetty is dual licensed under both + + * The Apache 2.0 License + http://www.apache.org/licenses/LICENSE-2.0.html + + and + + * The Eclipse Public 1.0 License + http://www.eclipse.org/legal/epl-v10.html + +Jetty may be distributed under either license. + +------ +Eclipse + +The following artifacts are EPL. + * org.eclipse.jetty.orbit:org.eclipse.jdt.core + +The following artifacts are EPL and ASL2. + * org.eclipse.jetty.orbit:javax.security.auth.message + + +The following artifacts are EPL and CDDL 1.0. + * org.eclipse.jetty.orbit:javax.mail.glassfish + + +------ +Oracle + +The following artifacts are CDDL + GPLv2 with classpath exception. +https://glassfish.dev.java.net/nonav/public/CDDL+GPL.html + + * javax.servlet:javax.servlet-api + * javax.annotation:javax.annotation-api + * javax.transaction:javax.transaction-api + * javax.websocket:javax.websocket-api + +------ +Oracle OpenJDK + +If ALPN is used to negotiate HTTP/2 connections, then the following +artifacts may be included in the distribution or downloaded when ALPN +module is selected. + + * java.sun.security.ssl + +These artifacts replace/modify OpenJDK classes. The modififications +are hosted at github and both modified and original are under GPL v2 with +classpath exceptions. +http://openjdk.java.net/legal/gplv2+ce.html + + +------ +OW2 + +The following artifacts are licensed by the OW2 Foundation according to the +terms of http://asm.ow2.org/license.html + +org.ow2.asm:asm-commons +org.ow2.asm:asm + + +------ +Apache + +The following artifacts are ASL2 licensed. + +org.apache.taglibs:taglibs-standard-spec +org.apache.taglibs:taglibs-standard-impl + + +------ +MortBay + +The following artifacts are ASL2 licensed. Based on selected classes from +following Apache Tomcat jars, all ASL2 licensed. + +org.mortbay.jasper:apache-jsp + org.apache.tomcat:tomcat-jasper + org.apache.tomcat:tomcat-juli + org.apache.tomcat:tomcat-jsp-api + org.apache.tomcat:tomcat-el-api + org.apache.tomcat:tomcat-jasper-el + org.apache.tomcat:tomcat-api + org.apache.tomcat:tomcat-util-scan + org.apache.tomcat:tomcat-util + +org.mortbay.jasper:apache-el + org.apache.tomcat:tomcat-jasper-el + org.apache.tomcat:tomcat-el-api + + +------ +Mortbay + +The following artifacts are CDDL + GPLv2 with classpath exception. + +https://glassfish.dev.java.net/nonav/public/CDDL+GPL.html + +org.eclipse.jetty.toolchain:jetty-schemas + +------ +Assorted + +The UnixCrypt.java code implements the one way cryptography used by +Unix systems for simple password protection. Copyright 1996 Aki Yoshida, +modified April 2001 by Iris Van den Broeke, Daniel Deville. +Permission to use, copy, modify and distribute UnixCrypt +for non-commercial or commercial purposes and without fee is +granted provided that the copyright notice appears in all copies. + + +Apache Kafka +Copyright 2012 The Apache Software Foundation. + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Kerby +Copyright 2015-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache log4j +Copyright 2010 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Metrics +Copyright 2010-2013 Coda Hale and Yammer, Inc. + +This product includes software developed by Coda Hale and Yammer, Inc. + +This product includes code derived from the JSR-166 project (ThreadLocalRandom, Striped64, +LongAdder), which was released with the following comments: + + Written by Doug Lea with assistance from members of JCP JSR-166 + Expert Group and released to the public domain, as explained at + http://creativecommons.org/publicdomain/zero/1.0/ + + + + The Netty Project + ================= + +Please visit the Netty web site for more information: + + * http://netty.io/ + +Copyright 2014 The Netty Project + +The Netty Project licenses this file to you under the Apache License, +version 2.0 (the "License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at: + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +License for the specific language governing permissions and limitations +under the License. + +Also, please refer to each LICENSE..txt file, which is located in +the 'license' directory of the distribution file, for the license terms of the +components that this product depends on. + +------------------------------------------------------------------------------- +This product contains the extensions to Java Collections Framework which has +been derived from the works by JSR-166 EG, Doug Lea, and Jason T. Greene: + + * LICENSE: + * license/LICENSE.jsr166y.txt (Public Domain) + * HOMEPAGE: + * http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/ + * http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbosscache/experimental/jsr166/ + +This product contains a modified version of Robert Harder's Public Domain +Base64 Encoder and Decoder, which can be obtained at: + + * LICENSE: + * license/LICENSE.base64.txt (Public Domain) + * HOMEPAGE: + * http://iharder.sourceforge.net/current/java/base64/ + +This product contains a modified portion of 'Webbit', an event based +WebSocket and HTTP server, which can be obtained at: + + * LICENSE: + * license/LICENSE.webbit.txt (BSD License) + * HOMEPAGE: + * https://github.com/joewalnes/webbit + +This product contains a modified portion of 'SLF4J', a simple logging +facade for Java, which can be obtained at: + + * LICENSE: + * license/LICENSE.slf4j.txt (MIT License) + * HOMEPAGE: + * http://www.slf4j.org/ + +This product contains a modified portion of 'Apache Harmony', an open source +Java SE, which can be obtained at: + + * NOTICE: + * license/NOTICE.harmony.txt + * LICENSE: + * license/LICENSE.harmony.txt (Apache License 2.0) + * HOMEPAGE: + * http://archive.apache.org/dist/harmony/ + +This product contains a modified portion of 'jbzip2', a Java bzip2 compression +and decompression library written by Matthew J. Francis. It can be obtained at: + + * LICENSE: + * license/LICENSE.jbzip2.txt (MIT License) + * HOMEPAGE: + * https://code.google.com/p/jbzip2/ + +This product contains a modified portion of 'libdivsufsort', a C API library to construct +the suffix array and the Burrows-Wheeler transformed string for any input string of +a constant-size alphabet written by Yuta Mori. It can be obtained at: + + * LICENSE: + * license/LICENSE.libdivsufsort.txt (MIT License) + * HOMEPAGE: + * https://github.com/y-256/libdivsufsort + +This product contains a modified portion of Nitsan Wakart's 'JCTools', Java Concurrency Tools for the JVM, + which can be obtained at: + + * LICENSE: + * license/LICENSE.jctools.txt (ASL2 License) + * HOMEPAGE: + * https://github.com/JCTools/JCTools + +This product optionally depends on 'JZlib', a re-implementation of zlib in +pure Java, which can be obtained at: + + * LICENSE: + * license/LICENSE.jzlib.txt (BSD style License) + * HOMEPAGE: + * http://www.jcraft.com/jzlib/ + +This product optionally depends on 'Compress-LZF', a Java library for encoding and +decoding data in LZF format, written by Tatu Saloranta. It can be obtained at: + + * LICENSE: + * license/LICENSE.compress-lzf.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/ning/compress + +This product optionally depends on 'lz4', a LZ4 Java compression +and decompression library written by Adrien Grand. It can be obtained at: + + * LICENSE: + * license/LICENSE.lz4.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/jpountz/lz4-java + +This product optionally depends on 'lzma-java', a LZMA Java compression +and decompression library, which can be obtained at: + + * LICENSE: + * license/LICENSE.lzma-java.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/jponge/lzma-java + +This product contains a modified portion of 'jfastlz', a Java port of FastLZ compression +and decompression library written by William Kinney. It can be obtained at: + + * LICENSE: + * license/LICENSE.jfastlz.txt (MIT License) + * HOMEPAGE: + * https://code.google.com/p/jfastlz/ + +This product contains a modified portion of and optionally depends on 'Protocol Buffers', Google's data +interchange format, which can be obtained at: + + * LICENSE: + * license/LICENSE.protobuf.txt (New BSD License) + * HOMEPAGE: + * https://github.com/google/protobuf + +This product optionally depends on 'Bouncy Castle Crypto APIs' to generate +a temporary self-signed X.509 certificate when the JVM does not provide the +equivalent functionality. It can be obtained at: + + * LICENSE: + * license/LICENSE.bouncycastle.txt (MIT License) + * HOMEPAGE: + * http://www.bouncycastle.org/ + +This product optionally depends on 'Snappy', a compression library produced +by Google Inc, which can be obtained at: + + * LICENSE: + * license/LICENSE.snappy.txt (New BSD License) + * HOMEPAGE: + * https://github.com/google/snappy + +This product optionally depends on 'JBoss Marshalling', an alternative Java +serialization API, which can be obtained at: + + * LICENSE: + * license/LICENSE.jboss-marshalling.txt (GNU LGPL 2.1) + * HOMEPAGE: + * http://www.jboss.org/jbossmarshalling + +This product optionally depends on 'Caliper', Google's micro- +benchmarking framework, which can be obtained at: + + * LICENSE: + * license/LICENSE.caliper.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/google/caliper + +This product optionally depends on 'Apache Commons Logging', a logging +framework, which can be obtained at: + + * LICENSE: + * license/LICENSE.commons-logging.txt (Apache License 2.0) + * HOMEPAGE: + * http://commons.apache.org/logging/ + +This product optionally depends on 'Apache Log4J', a logging framework, which +can be obtained at: + + * LICENSE: + * license/LICENSE.log4j.txt (Apache License 2.0) + * HOMEPAGE: + * http://logging.apache.org/log4j/ + +This product optionally depends on 'Aalto XML', an ultra-high performance +non-blocking XML processor, which can be obtained at: + + * LICENSE: + * license/LICENSE.aalto-xml.txt (Apache License 2.0) + * HOMEPAGE: + * http://wiki.fasterxml.com/AaltoHome + +This product contains a modified version of 'HPACK', a Java implementation of +the HTTP/2 HPACK algorithm written by Twitter. It can be obtained at: + + * LICENSE: + * license/LICENSE.hpack.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/twitter/hpack + +This product contains a modified portion of 'Apache Commons Lang', a Java library +provides utilities for the java.lang API, which can be obtained at: + + * LICENSE: + * license/LICENSE.commons-lang.txt (Apache License 2.0) + * HOMEPAGE: + * https://commons.apache.org/proper/commons-lang/ + + +This product contains the Maven wrapper scripts from 'Maven Wrapper', that provides an easy way to ensure a user has everything necessary to run the Maven build. + + * LICENSE: + * license/LICENSE.mvn-wrapper.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/takari/maven-wrapper + + +This product includes software developed by Google + Snappy: http://code.google.com/p/snappy/ (New BSD License) + +This product includes software developed by Apache + PureJavaCrc32C from apache-hadoop-common http://hadoop.apache.org/ + (Apache 2.0 license) + +This library containd statically linked libstdc++. This inclusion is allowed by +"GCC RUntime Library Exception" +http://gcc.gnu.org/onlinedocs/libstdc++/manual/license.html + +== Contributors == + * Tatu Saloranta + * Providing benchmark suite + * Alec Wysoker + * Performance and memory usage improvement + + +Apache ZooKeeper +Copyright 2009-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). diff --git a/NOTICE.txt b/NOTICE.txt index b4329b8095a59..f6715f7beb085 100644 --- a/NOTICE.txt +++ b/NOTICE.txt @@ -1,637 +1,34 @@ -This product includes software developed by The Apache Software -Foundation (http://www.apache.org/). - -The binary distribution of this product bundles binaries of -org.iq80.leveldb:leveldb-api (https://github.com/dain/leveldb), which has the -following notices: -* Copyright 2011 Dain Sundstrom -* Copyright 2011 FuseSource Corp. http://fusesource.com - -The binary distribution of this product bundles binaries of -AWS SDK for Java - Bundle 1.11.375, -AWS Java SDK for AWS KMS 1.11.375, -AWS Java SDK for Amazon S3 1.11.375, -AWS Java SDK for AWS STS 1.11.375, -JMES Path Query library 1.0, -which has the following notices: - * This software includes third party software subject to the following - copyrights: - XML parsing and utility functions from JetS3t - Copyright - 2006-2009 James Murty. - JSON parsing and utility functions from JSON.org - - Copyright 2002 JSON.org. - PKCS#1 PEM encoded private key parsing and utility - functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc. - -The binary distribution of this product bundles binaries of -Gson 2.2.4, -which has the following notices: - - The Netty Project - ================= - -Please visit the Netty web site for more information: - - * http://netty.io/ - -Copyright 2014 The Netty Project - -The Netty Project licenses this file to you under the Apache License, -version 2.0 (the "License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at: - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -License for the specific language governing permissions and limitations -under the License. - -Also, please refer to each LICENSE..txt file, which is located in -the 'license' directory of the distribution file, for the license terms of the -components that this product depends on. - -------------------------------------------------------------------------------- -This product contains the extensions to Java Collections Framework which has -been derived from the works by JSR-166 EG, Doug Lea, and Jason T. Greene: - - * LICENSE: - * license/LICENSE.jsr166y.txt (Public Domain) - * HOMEPAGE: - * http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/ - * http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbosscache/experimental/jsr166/ - -This product contains a modified version of Robert Harder's Public Domain -Base64 Encoder and Decoder, which can be obtained at: - - * LICENSE: - * license/LICENSE.base64.txt (Public Domain) - * HOMEPAGE: - * http://iharder.sourceforge.net/current/java/base64/ - -This product contains a modified portion of 'Webbit', an event based -WebSocket and HTTP server, which can be obtained at: - - * LICENSE: - * license/LICENSE.webbit.txt (BSD License) - * HOMEPAGE: - * https://github.com/joewalnes/webbit - -This product contains a modified portion of 'SLF4J', a simple logging -facade for Java, which can be obtained at: - - * LICENSE: - * license/LICENSE.slf4j.txt (MIT License) - * HOMEPAGE: - * http://www.slf4j.org/ - -This product contains a modified portion of 'ArrayDeque', written by Josh -Bloch of Google, Inc: - - * LICENSE: - * license/LICENSE.deque.txt (Public Domain) - -This product contains a modified portion of 'Apache Harmony', an open source -Java SE, which can be obtained at: - - * LICENSE: - * license/LICENSE.harmony.txt (Apache License 2.0) - * HOMEPAGE: - * http://archive.apache.org/dist/harmony/ - -This product contains a modified version of Roland Kuhn's ASL2 -AbstractNodeQueue, which is based on Dmitriy Vyukov's non-intrusive MPSC queue. -It can be obtained at: - - * LICENSE: - * license/LICENSE.abstractnodequeue.txt (Public Domain) - * HOMEPAGE: - * https://github.com/akka/akka/blob/wip-2.2.3-for-scala-2.11/akka-actor/src/main/java/akka/dispatch/AbstractNodeQueue.java - -This product contains a modified portion of 'jbzip2', a Java bzip2 compression -and decompression library written by Matthew J. Francis. It can be obtained at: - - * LICENSE: - * license/LICENSE.jbzip2.txt (MIT License) - * HOMEPAGE: - * https://code.google.com/p/jbzip2/ - -This product contains a modified portion of 'libdivsufsort', a C API library to construct -the suffix array and the Burrows-Wheeler transformed string for any input string of -a constant-size alphabet written by Yuta Mori. It can be obtained at: - - * LICENSE: - * license/LICENSE.libdivsufsort.txt (MIT License) - * HOMEPAGE: - * https://code.google.com/p/libdivsufsort/ - -This product contains a modified portion of Nitsan Wakart's 'JCTools', Java Concurrency Tools for the JVM, - which can be obtained at: - - * LICENSE: - * license/LICENSE.jctools.txt (ASL2 License) - * HOMEPAGE: - * https://github.com/JCTools/JCTools - -This product optionally depends on 'JZlib', a re-implementation of zlib in -pure Java, which can be obtained at: - - * LICENSE: - * license/LICENSE.jzlib.txt (BSD style License) - * HOMEPAGE: - * http://www.jcraft.com/jzlib/ - -This product optionally depends on 'Compress-LZF', a Java library for encoding and -decoding data in LZF format, written by Tatu Saloranta. It can be obtained at: - - * LICENSE: - * license/LICENSE.compress-lzf.txt (Apache License 2.0) - * HOMEPAGE: - * https://github.com/ning/compress - -This product optionally depends on 'lz4', a LZ4 Java compression -and decompression library written by Adrien Grand. It can be obtained at: - - * LICENSE: - * license/LICENSE.lz4.txt (Apache License 2.0) - * HOMEPAGE: - * https://github.com/jpountz/lz4-java - -This product optionally depends on 'lzma-java', a LZMA Java compression -and decompression library, which can be obtained at: - - * LICENSE: - * license/LICENSE.lzma-java.txt (Apache License 2.0) - * HOMEPAGE: - * https://github.com/jponge/lzma-java - -This product contains a modified portion of 'jfastlz', a Java port of FastLZ compression -and decompression library written by William Kinney. It can be obtained at: - - * LICENSE: - * license/LICENSE.jfastlz.txt (MIT License) - * HOMEPAGE: - * https://code.google.com/p/jfastlz/ - -This product contains a modified portion of and optionally depends on 'Protocol Buffers', Google's data -interchange format, which can be obtained at: - - * LICENSE: - * license/LICENSE.protobuf.txt (New BSD License) - * HOMEPAGE: - * http://code.google.com/p/protobuf/ - -This product optionally depends on 'Bouncy Castle Crypto APIs' to generate -a temporary self-signed X.509 certificate when the JVM does not provide the -equivalent functionality. It can be obtained at: - - * LICENSE: - * license/LICENSE.bouncycastle.txt (MIT License) - * HOMEPAGE: - * http://www.bouncycastle.org/ - -This product optionally depends on 'Snappy', a compression library produced -by Google Inc, which can be obtained at: - - * LICENSE: - * license/LICENSE.snappy.txt (New BSD License) - * HOMEPAGE: - * http://code.google.com/p/snappy/ - -This product contains a modified portion of UnsignedBytes LexicographicalComparator -from Guava v21 project by Google Inc, which can be obtained at: - - * LICENSE: - * license/COPYING (Apache License 2.0) - * HOMEPAGE: - * https://github.com/google/guava - -This product optionally depends on 'JBoss Marshalling', an alternative Java -serialization API, which can be obtained at: - - * LICENSE: - * license/LICENSE.jboss-marshalling.txt (GNU LGPL 2.1) - * HOMEPAGE: - * http://www.jboss.org/jbossmarshalling - -This product optionally depends on 'Caliper', Google's micro- -benchmarking framework, which can be obtained at: - - * LICENSE: - * license/LICENSE.caliper.txt (Apache License 2.0) - * HOMEPAGE: - * http://code.google.com/p/caliper/ - -This product optionally depends on 'Apache Commons Logging', a logging -framework, which can be obtained at: - - * LICENSE: - * license/LICENSE.commons-logging.txt (Apache License 2.0) - * HOMEPAGE: - * http://commons.apache.org/logging/ - -This product optionally depends on 'Apache Log4J', a logging framework, which -can be obtained at: - - * LICENSE: - * license/LICENSE.log4j.txt (Apache License 2.0) - * HOMEPAGE: - * http://logging.apache.org/log4j/ - -This product optionally depends on 'Aalto XML', an ultra-high performance -non-blocking XML processor, which can be obtained at: - - * LICENSE: - * license/LICENSE.aalto-xml.txt (Apache License 2.0) - * HOMEPAGE: - * http://wiki.fasterxml.com/AaltoHome - -This product contains a modified version of 'HPACK', a Java implementation of -the HTTP/2 HPACK algorithm written by Twitter. It can be obtained at: - - * LICENSE: - * license/LICENSE.hpack.txt (Apache License 2.0) - * HOMEPAGE: - * https://github.com/twitter/hpack - -This product contains a modified portion of 'Apache Commons Lang', a Java library -provides utilities for the java.lang API, which can be obtained at: - - * LICENSE: - * license/LICENSE.commons-lang.txt (Apache License 2.0) - * HOMEPAGE: - * https://commons.apache.org/proper/commons-lang/ - -This product contains a modified portion of 'JDOM 1.1', which can be obtained at: - - * LICENSE: - * https://github.com/hunterhacker/jdom/blob/jdom-1.1/core/LICENSE.txt - * HOMEPAGE: - * http://www.jdom.org/ - -The binary distribution of this product bundles binaries of -Commons Codec 1.4, -which has the following notices: - * src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.javacontains test data from http://aspell.net/test/orig/batch0.tab.Copyright (C) 2002 Kevin Atkinson (kevina@gnu.org) - =============================================================================== - The content of package org.apache.commons.codec.language.bm has been translated - from the original php source code available at http://stevemorse.org/phoneticinfo.htm - with permission from the original authors. - Original source copyright:Copyright (c) 2008 Alexander Beider & Stephen P. Morse. - -The binary distribution of this product bundles binaries of -Commons Lang 2.6, -which has the following notices: - * This product includes software from the Spring Framework,under the Apache License 2.0 (see: StringUtils.containsWhitespace()) - -The binary distribution of this product bundles binaries of -Apache Log4j 1.2.17, -which has the following notices: - * ResolverUtil.java - Copyright 2005-2006 Tim Fennell - Dumbster SMTP test server - Copyright 2004 Jason Paul Kitchen - TypeUtil.java - Copyright 2002-2012 Ramnivas Laddad, Juergen Hoeller, Chris Beams - -The binary distribution of this product bundles binaries of -"Java Concurrency in Practice" book annotations 1.0, -which has the following notices: - * Copyright (c) 2005 Brian Goetz and Tim Peierls Released under the Creative - Commons Attribution License (http://creativecommons.org/licenses/by/2.5) - Official home: http://www.jcip.net Any republication or derived work - distributed in source code form must include this copyright and license - notice. - -The binary distribution of this product bundles binaries of -Jetty :: Http Utility 9.3.19., -Jetty :: IO Utility 9.3.19., -Jetty :: Security 9.3.19., -Jetty :: Server Core 9.3.19., -Jetty :: Servlet Handling 9.3.19., -Jetty :: Utilities 9.3.19., -Jetty :: Utilities :: Ajax, -Jetty :: Webapp Application Support 9.3.19., -Jetty :: XML utilities 9.3.19., -which has the following notices: - * ============================================================== - Jetty Web Container - Copyright 1995-2016 Mort Bay Consulting Pty Ltd. - ============================================================== - - The Jetty Web Container is Copyright Mort Bay Consulting Pty Ltd - unless otherwise noted. - - Jetty is dual licensed under both - - * The Apache 2.0 License - http://www.apache.org/licenses/LICENSE-2.0.html - - and - - * The Eclipse Public 1.0 License - http://www.eclipse.org/legal/epl-v10.html - - Jetty may be distributed under either license. - - ------ - Eclipse - - The following artifacts are EPL. - * org.eclipse.jetty.orbit:org.eclipse.jdt.core - - The following artifacts are EPL and ASL2. - * org.eclipse.jetty.orbit:javax.security.auth.message - - - The following artifacts are EPL and CDDL 1.0. - * org.eclipse.jetty.orbit:javax.mail.glassfish - - - ------ - Oracle - - The following artifacts are CDDL + GPLv2 with classpath exception. - https://glassfish.dev.java.net/nonav/public/CDDL+GPL.html - - * javax.servlet:javax.servlet-api - * javax.annotation:javax.annotation-api - * javax.transaction:javax.transaction-api - * javax.websocket:javax.websocket-api - - ------ - Oracle OpenJDK - - If ALPN is used to negotiate HTTP/2 connections, then the following - artifacts may be included in the distribution or downloaded when ALPN - module is selected. - - * java.sun.security.ssl - - These artifacts replace/modify OpenJDK classes. The modififications - are hosted at github and both modified and original are under GPL v2 with - classpath exceptions. - http://openjdk.java.net/legal/gplv2+ce.html - - - ------ - OW2 - - The following artifacts are licensed by the OW2 Foundation according to the - terms of http://asm.ow2.org/license.html - - org.ow2.asm:asm-commons - org.ow2.asm:asm - - - ------ - Apache - - The following artifacts are ASL2 licensed. - - org.apache.taglibs:taglibs-standard-spec - org.apache.taglibs:taglibs-standard-impl - - - ------ - MortBay - - The following artifacts are ASL2 licensed. Based on selected classes from - following Apache Tomcat jars, all ASL2 licensed. - - org.mortbay.jasper:apache-jsp - org.apache.tomcat:tomcat-jasper - org.apache.tomcat:tomcat-juli - org.apache.tomcat:tomcat-jsp-api - org.apache.tomcat:tomcat-el-api - org.apache.tomcat:tomcat-jasper-el - org.apache.tomcat:tomcat-api - org.apache.tomcat:tomcat-util-scan - org.apache.tomcat:tomcat-util - - org.mortbay.jasper:apache-el - org.apache.tomcat:tomcat-jasper-el - org.apache.tomcat:tomcat-el-api - - - ------ - Mortbay - - The following artifacts are CDDL + GPLv2 with classpath exception. - - https://glassfish.dev.java.net/nonav/public/CDDL+GPL.html - - org.eclipse.jetty.toolchain:jetty-schemas - - ------ - Assorted - - The UnixCrypt.java code implements the one way cryptography used by - Unix systems for simple password protection. Copyright 1996 Aki Yoshida, - modified April 2001 by Iris Van den Broeke, Daniel Deville. - Permission to use, copy, modify and distribute UnixCrypt - for non-commercial or commercial purposes and without fee is - granted provided that the copyright notice appears in all copies./ - -The binary distribution of this product bundles binaries of -Snappy for Java 1.0.4.1, -which has the following notices: - * This product includes software developed by Google - Snappy: http://code.google.com/p/snappy/ (New BSD License) - - This product includes software developed by Apache - PureJavaCrc32C from apache-hadoop-common http://hadoop.apache.org/ - (Apache 2.0 license) - - This library containd statically linked libstdc++. This inclusion is allowed by - "GCC RUntime Library Exception" - http://gcc.gnu.org/onlinedocs/libstdc++/manual/license.html - - == Contributors == - * Tatu Saloranta - * Providing benchmark suite - * Alec Wysoker - * Performance and memory usage improvement - -The binary distribution of this product bundles binaries of -Xerces2 Java Parser 2.9.1, -which has the following notices: - * ========================================================================= - == NOTICE file corresponding to section 4(d) of the Apache License, == - == Version 2.0, in this case for the Apache Xerces Java distribution. == - ========================================================================= - - Apache Xerces Java - Copyright 1999-2007 The Apache Software Foundation - - This product includes software developed at - The Apache Software Foundation (http://www.apache.org/). - - Portions of this software were originally based on the following: - - software copyright (c) 1999, IBM Corporation., http://www.ibm.com. - - software copyright (c) 1999, Sun Microsystems., http://www.sun.com. - - voluntary contributions made by Paul Eng on behalf of the - Apache Software Foundation that were originally developed at iClick, Inc., - software copyright (c) 1999. - -The binary distribution of this product bundles binaries of -Logback Classic Module 1.1.2, -Logback Core Module 1.1.2, -which has the following notices: - * Logback: the reliable, generic, fast and flexible logging framework. - Copyright (C) 1999-2012, QOS.ch. All rights reserved. - -The binary distribution of this product bundles binaries of -Apache HBase - Annotations 1.2.6, -Apache HBase - Client 1.2.6, -Apache HBase - Common 1.2.6, -Apache HBase - Hadoop Compatibility 1.2.6, -Apache HBase - Hadoop Two Compatibility 1.2.6, -Apache HBase - Prefix Tree 1.2.6, -Apache HBase - Procedure 1.2.6, -Apache HBase - Protocol 1.2.6, -Apache HBase - Server 1.2.6, -which has the following notices: - * Apache HBase - Copyright 2007-2015 The Apache Software Foundation - - This product includes software developed at - The Apache Software Foundation (http://www.apache.org/). - - -- - This product incorporates portions of the 'Hadoop' project - - Copyright 2007-2009 The Apache Software Foundation - - Licensed under the Apache License v2.0 - -- - Our Orca logo we got here: http://www.vectorfree.com/jumping-orca - It is licensed Creative Commons Attribution 3.0. - See https://creativecommons.org/licenses/by/3.0/us/ - We changed the logo by stripping the colored background, inverting - it and then rotating it some. - - Later we found that vectorfree.com image is not properly licensed. - The original is owned by vectorportal.com. The original was - relicensed so we could use it as Creative Commons Attribution 3.0. - The license is bundled with the download available here: - http://www.vectorportal.com/subcategory/205/KILLER-WHALE-FREE-VECTOR.eps/ifile/9136/detailtest.asp - -- - This product includes portions of the Bootstrap project v3.0.0 - - Copyright 2013 Twitter, Inc. - - Licensed under the Apache License v2.0 - - This product uses the Glyphicons Halflings icon set. - - http://glyphicons.com/ - - Copyright Jan Kovařík - - Licensed under the Apache License v2.0 as a part of the Bootstrap project. - - -- - This product includes portions of the Guava project v14, specifically - 'hbase-common/src/main/java/org/apache/hadoop/hbase/io/LimitInputStream.java' - - Copyright (C) 2007 The Guava Authors - - Licensed under the Apache License, Version 2.0 - -The binary distribution of this product bundles binaries of -Phoenix Core 4.7.0, -which has the following notices: - Apache Phoenix - Copyright 2013-2016 The Apache Software Foundation - - This product includes software developed by The Apache Software - Foundation (http://www.apache.org/). - - This also includes: - - The phoenix-spark module has been adapted from the phoenix-spark library - distributed under the terms of the Apache 2 license. Original source copyright: - Copyright 2014 Simply Measured, Inc. - Copyright 2015 Interset Software Inc. - - The file bin/daemon.py is based on the file of the same name in python-daemon 2.0.5 - (https://pypi.python.org/pypi/python-daemon/). Original source copyright: - # Copyright © 2008–2015 Ben Finney - # Copyright © 2007–2008 Robert Niederreiter, Jens Klein - # Copyright © 2004–2005 Chad J. Schroeder - # Copyright © 2003 Clark Evans - # Copyright © 2002 Noah Spurrier - # Copyright © 2001 Jürgen Hermann - -The binary distribution of this product bundles binaries of -Plexus Cipher: encryption/decryption Component 1.4, -which has the following notices: - * The code in this component contains a class - Base64 taken from http://juliusdavies.ca/svn/not-yet-commons-ssl/tags/commons-ssl-0.3.10/src/java/org/apache/commons/ssl/Base64.java - which is Apache license: http://www.apache.org/licenses/LICENSE-2.0 - - The PBE key processing routine PBECipher.createCipher() is adopted from http://juliusdavies.ca/svn/not-yet-commons-ssl/tags/commons-ssl-0.3.10/src/java/org/apache/commons/ssl/OpenSSL.java - which is also Apache APL-2.0 license: http://www.apache.org/licenses/LICENSE-2.0 - -The binary distribution of this product bundles binaries of -software.amazon.ion:ion-java 1.0.1, -which has the following notices: - * Amazon Ion Java Copyright 2007-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved. - -The binary distribution of this product bundles binaries of -joda-time:joda-time:2.9.9 -which has the following notices: - * ============================================================================= - = NOTICE file corresponding to section 4d of the Apache License Version 2.0 = - ============================================================================= - This product includes software developed by - Joda.org (http://www.joda.org/). - -The binary distribution of this product bundles binaries of -Ehcache 3.3.1, -which has the following notices: - * Ehcache V3 Copyright 2014-2016 Terracotta, Inc. - -The binary distribution of this product bundles binaries of -snakeyaml (https://bitbucket.org/asomov/snakeyaml), -which has the following notices: - * Copyright (c) 2008, http://www.snakeyaml.org - -The binary distribution of this product bundles binaries of -swagger-annotations (https://github.com/swagger-api/swagger-core), -which has the following notices: - * Copyright 2016 SmartBear Software - -The binary distribution of this product bundles binaries of -metrics-core 3.2.4 -which has the following notices: - * Copyright 2010-2013 Coda Hale and Yammer, Inc. - - This product includes software developed by Coda Hale and Yammer, Inc. - - This product includes code derived from the JSR-166 project (ThreadLocalRandom, Striped64, - LongAdder), which was released with the following comments: - - Written by Doug Lea with assistance from members of JCP JSR-166 - Expert Group and released to the public domain, as explained at - http://creativecommons.org/publicdomain/zero/1.0/ - -The source and binary distribution of this product bundles binaries of -xterm.js 3.8.0 -which has the following notices: -Copyright (c) 2017-2018, The xterm.js authors (https://github.com/xtermjs/xterm.js) -Copyright (c) 2014-2016, SourceLair Private Company (https://www.sourcelair.com) -Copyright (c) 2012-2013, Christopher Jeffrey (https://github.com/chjj/) - -The source and binary distribution of this product bundles modified version of - github.com/awslabs/aws-js-s3-explorer licensed under Apache 2.0 license - with the following notice: - -AWS JavaScript S3 Explorer -Copyright 2014-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. - -The binary distribution of this product bundles binaries of -jline 3.9.0 (https://github.com/jline/jline3) - - * LICENSE: - * license/LICENSE.jline3.txt (BSD License) - * HOMEPAGE: - * https://github.com/jline/jline3 +Apache Hadoop +Copyright 2006 and onwards The Apache Software Foundation. + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +Export Control Notice +--------------------- + +This distribution includes cryptographic software. The country in +which you currently reside may have restrictions on the import, +possession, use, and/or re-export to another country, of +encryption software. BEFORE using any encryption software, please +check your country's laws, regulations and policies concerning the +import, possession, or use, and re-export of encryption software, to +see if this is permitted. See for more +information. + +The U.S. Government Department of Commerce, Bureau of Industry and +Security (BIS), has classified this software as Export Commodity +Control Number (ECCN) 5D002.C.1, which includes information security +software using or performing cryptographic functions with asymmetric +algorithms. The form and manner of this Apache Software Foundation +distribution makes it eligible for export under the License Exception +ENC Technology Software Unrestricted (TSU) exception (see the BIS +Export Administration Regulations, Section 740.13) for both object +code and source code. + +The following provides more details on the included cryptographic software: + +This software uses the SSL libraries from the Jetty project written +by mortbay.org. +Hadoop Yarn Server Web Proxy uses the BouncyCastle Java +cryptography APIs written by the Legion of the Bouncy Castle Inc. diff --git a/README.txt b/README.txt index f11b1b441afd0..8d37cc92a7ecb 100644 --- a/README.txt +++ b/README.txt @@ -4,30 +4,4 @@ For the latest information about Hadoop, please visit our website at: and our wiki, at: - http://wiki.apache.org/hadoop/ - -This distribution includes cryptographic software. The country in -which you currently reside may have restrictions on the import, -possession, use, and/or re-export to another country, of -encryption software. BEFORE using any encryption software, please -check your country's laws, regulations and policies concerning the -import, possession, or use, and re-export of encryption software, to -see if this is permitted. See for more -information. - -The U.S. Government Department of Commerce, Bureau of Industry and -Security (BIS), has classified this software as Export Commodity -Control Number (ECCN) 5D002.C.1, which includes information security -software using or performing cryptographic functions with asymmetric -algorithms. The form and manner of this Apache Software Foundation -distribution makes it eligible for export under the License Exception -ENC Technology Software Unrestricted (TSU) exception (see the BIS -Export Administration Regulations, Section 740.13) for both object -code and source code. - -The following provides more details on the included cryptographic -software: - Hadoop Core uses the SSL libraries from the Jetty project written -by mortbay.org. - Hadoop Yarn Server Web Proxy uses the BouncyCastle Java -cryptography APIs written by the Legion of the Bouncy Castle Inc. + https://cwiki.apache.org/confluence/display/HADOOP/ diff --git a/dev-support/bin/create-release b/dev-support/bin/create-release index 35447982903e2..d14c0073a5f17 100755 --- a/dev-support/bin/create-release +++ b/dev-support/bin/create-release @@ -641,7 +641,7 @@ function signartifacts for i in ${ARTIFACTS_DIR}/*; do ${GPG} --use-agent --armor --output "${i}.asc" --detach-sig "${i}" - ${GPG} --print-mds "${i}" > "${i}.mds" + sha512sum --tag "${i}" > "${i}.sha512" done if [[ "${ASFRELEASE}" = true ]]; then diff --git a/dev-support/bin/dist-copynativelibs b/dev-support/bin/dist-copynativelibs index 67d2edf22d304..4a783f086a4dc 100755 --- a/dev-support/bin/dist-copynativelibs +++ b/dev-support/bin/dist-copynativelibs @@ -96,6 +96,12 @@ for i in "$@"; do --isalbundle=*) ISALBUNDLE=${i#*=} ;; + --pmdklib=*) + PMDKLIB=${i#*=} + ;; + --pmdkbundle=*) + PMDKBUNDLE=${i#*=} + ;; --opensslbinbundle=*) OPENSSLBINBUNDLE=${i#*=} ;; @@ -153,6 +159,8 @@ if [[ -d "${LIB_DIR}" ]]; then bundle_native_lib "${OPENSSLLIBBUNDLE}" "openssl.lib" "crypto" "${OPENSSLLIB}" bundle_native_lib "${ISALBUNDLE}" "isal.lib" "isa" "${ISALLIB}" + + bundle_native_lib "${PMDKBUNDLE}" "pmdk.lib" "pmdk" "${PMDKLIB}" fi # Windows diff --git a/dev-support/bin/dist-layout-stitching b/dev-support/bin/dist-layout-stitching index 119ee6022709b..20e8cf27805e2 100755 --- a/dev-support/bin/dist-layout-stitching +++ b/dev-support/bin/dist-layout-stitching @@ -122,6 +122,10 @@ run mkdir "hadoop-${VERSION}" run cd "hadoop-${VERSION}" run cp -p "${ROOT}/LICENSE.txt" . run cp -p "${ROOT}/NOTICE.txt" . +run cp -p "${ROOT}/LICENSE-binary" . +run mkdir licenses-binary +run cp -p "${ROOT}/licenses-binary"/* licenses-binary/ +run cp -p "${ROOT}/NOTICE-binary" . run cp -p "${ROOT}/README.txt" . # Copy hadoop-common first so that it have always have all dependencies. @@ -148,6 +152,7 @@ run cp -p "${ROOT}/hadoop-client-modules/hadoop-client-runtime/target/hadoop-cli run cp -p "${ROOT}/hadoop-client-modules/hadoop-client-minicluster/target/hadoop-client-minicluster-${VERSION}.jar" share/hadoop/client/ run copy "${ROOT}/hadoop-tools/hadoop-tools-dist/target/hadoop-tools-dist-${VERSION}" . +run copy "${ROOT}/hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-dist/target/hadoop-dynamometer-dist-${VERSION}" . echo diff --git a/dev-support/bin/yetus-wrapper b/dev-support/bin/yetus-wrapper index ae05d426b2c76..b0f71f105d85e 100755 --- a/dev-support/bin/yetus-wrapper +++ b/dev-support/bin/yetus-wrapper @@ -68,20 +68,30 @@ function yetus_abs return 1 } +function version_ge() +{ + test "$(echo "$@" | tr " " "\n" | sort -rV | head -n 1)" == "$1"; +} WANTED="$1" shift ARGV=("$@") -HADOOP_YETUS_VERSION=${HADOOP_YETUS_VERSION:-0.8.0} +HADOOP_YETUS_VERSION=${HADOOP_YETUS_VERSION:-0.10.0} BIN=$(yetus_abs "${BASH_SOURCE-$0}") BINDIR=$(dirname "${BIN}") +## HADOOP_YETUS_VERSION >= 0.9.0 the tarball named with apache-yetus prefix +if version_ge "${HADOOP_YETUS_VERSION}" "0.9.0"; then + YETUS_PREFIX=apache-yetus +else + YETUS_PREFIX=yetus +fi + ### ### if YETUS_HOME is set, then try to use it ### -if [[ -n "${YETUS_HOME}" - && -x "${YETUS_HOME}/bin/${WANTED}" ]]; then +if [[ -n "${YETUS_HOME}" && -x "${YETUS_HOME}/bin/${WANTED}" ]]; then exec "${YETUS_HOME}/bin/${WANTED}" "${ARGV[@]}" fi @@ -105,8 +115,8 @@ HADOOP_PATCHPROCESS=${mytmpdir} ## ## if we've already DL'd it, then short cut ## -if [[ -x "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then - exec "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}" +if [[ -x "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then + exec "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}" fi ## @@ -114,7 +124,7 @@ fi ## BASEURL="https://archive.apache.org/dist/yetus/${HADOOP_YETUS_VERSION}/" -TARBALL="yetus-${HADOOP_YETUS_VERSION}-bin.tar" +TARBALL="${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}-bin.tar" GPGBIN=$(command -v gpg) CURLBIN=$(command -v curl) @@ -166,9 +176,9 @@ if ! (gunzip -c "${TARBALL}.gz" | tar xpf -); then exit 1 fi -if [[ -x "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then +if [[ -x "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then popd >/dev/null - exec "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}" + exec "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}" fi ## diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile index e71e51c970c66..65cada2784df9 100644 --- a/dev-support/docker/Dockerfile +++ b/dev-support/docker/Dockerfile @@ -106,12 +106,12 @@ ENV CMAKE_HOME /opt/cmake ENV PATH "${PATH}:/opt/cmake/bin" ###### -# Install Google Protobuf 2.5.0 (2.6.0 ships with Xenial) +# Install Google Protobuf 3.7.1 (2.6.0 ships with Xenial) ###### # hadolint ignore=DL3003 RUN mkdir -p /opt/protobuf-src \ && curl -L -s -S \ - https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz \ + https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ -o /opt/protobuf.tar.gz \ && tar xzf /opt/protobuf.tar.gz --strip-components 1 -C /opt/protobuf-src \ && cd /opt/protobuf-src \ @@ -200,7 +200,6 @@ RUN curl -L -s -S \ ### ENV MAVEN_OPTS -Xms256m -Xmx1536m - ### # Everything past this point is either not needed for testing or breaks Yetus. # So tell Yetus not to read the rest of the file: @@ -208,10 +207,11 @@ ENV MAVEN_OPTS -Xms256m -Xmx1536m ### # Hugo static website generator (for new hadoop site and Ozone docs) -RUN curl -L -o hugo.deb https://github.com/gohugoio/hugo/releases/download/v0.30.2/hugo_0.30.2_Linux-64bit.deb \ +RUN curl -L -o hugo.deb https://github.com/gohugoio/hugo/releases/download/v0.58.3/hugo_0.58.3_Linux-64bit.deb \ && dpkg --install hugo.deb \ && rm hugo.deb + # Add a welcome message and environment checks. COPY hadoop_env_checks.sh /root/hadoop_env_checks.sh RUN chmod 755 /root/hadoop_env_checks.sh diff --git a/hadoop-assemblies/pom.xml b/hadoop-assemblies/pom.xml index aabef34579609..b0fd7325c6eb1 100644 --- a/hadoop-assemblies/pom.xml +++ b/hadoop-assemblies/pom.xml @@ -18,7 +18,7 @@ + https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml index 88cd210947721..df48f314b43b7 100644 --- a/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml +++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml @@ -131,6 +131,19 @@ /include + + . + + LICENSE-binary + NOTICE-binary + + + + ./license-binary + + * + + diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-blockgen.xml b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-blockgen.xml new file mode 100644 index 0000000000000..8d35141f64abe --- /dev/null +++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-blockgen.xml @@ -0,0 +1,35 @@ + + + hadoop-dynamometer-blockgen + + dir + + false + + + + ${basedir}/src/main/bash + dynamometer-blockgen/bin + 0755 + + + + diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-infra.xml b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-infra.xml new file mode 100644 index 0000000000000..a8d4c09b9520d --- /dev/null +++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-infra.xml @@ -0,0 +1,35 @@ + + + hadoop-dynamometer-infra + + dir + + false + + + + ${basedir}/src/main/bash + dynamometer-infra/bin + 0755 + + + + diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-workload.xml b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-workload.xml new file mode 100644 index 0000000000000..80cf304840eb0 --- /dev/null +++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-workload.xml @@ -0,0 +1,35 @@ + + + hadoop-dynamometer-workload + + dir + + false + + + + ${basedir}/src/main/bash + dynamometer-workload/bin + 0755 + + + + diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml new file mode 100644 index 0000000000000..448035262e12d --- /dev/null +++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml @@ -0,0 +1,73 @@ + + + hadoop-dynamometer + + dir + + false + + + ../hadoop-dynamometer-blockgen/target + /share/hadoop/${hadoop.component}/sources + + *-sources.jar + + + + ../hadoop-dynamometer-blockgen/target/hadoop-dynamometer-blockgen-${project.version}/dynamometer-blockgen + /share/hadoop/${hadoop.component}/dynamometer/dynamometer-blockgen + + + ../hadoop-dynamometer-workload/target + /share/hadoop/${hadoop.component}/sources + + *-sources.jar + + + + ../hadoop-dynamometer-workload/target/hadoop-dynamometer-workload-${project.version}/dynamometer-workload + /share/hadoop/${hadoop.component}/dynamometer/dynamometer-workload + + + ../hadoop-dynamometer-infra/target + /share/hadoop/${hadoop.component}/sources + + *-sources.jar + + + + ../hadoop-dynamometer-infra/target/hadoop-dynamometer-infra-${project.version}/dynamometer-infra + /share/hadoop/${hadoop.component}/dynamometer/dynamometer-infra + + + + + /share/hadoop/${hadoop.component}/lib + false + runtime + false + + + org.slf4j:slf4j-api + org.slf4j:slf4j-log4j12 + + + + diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-src-with-hdds.xml b/hadoop-assemblies/src/main/resources/assemblies/hadoop-src-submarine.xml similarity index 100% rename from hadoop-assemblies/src/main/resources/assemblies/hadoop-src-with-hdds.xml rename to hadoop-assemblies/src/main/resources/assemblies/hadoop-src-submarine.xml diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-src.xml b/hadoop-assemblies/src/main/resources/assemblies/hadoop-src.xml index f0a8d44376c5a..b47b4bcc333fb 100644 --- a/hadoop-assemblies/src/main/resources/assemblies/hadoop-src.xml +++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-src.xml @@ -31,6 +31,12 @@ NOTICE.txt + + ./licenses + + * + + . true @@ -52,6 +58,7 @@ **/SecurityAuth.audit* hadoop-ozone/** hadoop-hdds/** + hadoop-submarine/** diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml b/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml index 7d34154838d55..054d8c0ace2bd 100644 --- a/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml +++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml @@ -209,7 +209,7 @@ org.apache.hadoop:hadoop-common org.apache.hadoop:hadoop-hdfs - org.apache.hadoop:hadoop-mapreduce + org.apache.hadoop:hadoop-client org.apache.hadoop:hadoop-pipes diff --git a/hadoop-build-tools/pom.xml b/hadoop-build-tools/pom.xml index b94dd32038953..ed4c0ef9ce9ff 100644 --- a/hadoop-build-tools/pom.xml +++ b/hadoop-build-tools/pom.xml @@ -14,7 +14,7 @@ --> + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> hadoop-main org.apache.hadoop diff --git a/hadoop-client-modules/hadoop-client-api/pom.xml b/hadoop-client-modules/hadoop-client-api/pom.xml index 9e4406d39d1b1..7ee7b85fec937 100644 --- a/hadoop-client-modules/hadoop-client-api/pom.xml +++ b/hadoop-client-modules/hadoop-client-api/pom.xml @@ -13,7 +13,7 @@ limitations under the License. See accompanying LICENSE file. --> + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-client-modules/hadoop-client-check-invariants/pom.xml b/hadoop-client-modules/hadoop-client-check-invariants/pom.xml index f0629867d5120..757b374ec0ab0 100644 --- a/hadoop-client-modules/hadoop-client-check-invariants/pom.xml +++ b/hadoop-client-modules/hadoop-client-check-invariants/pom.xml @@ -1,4 +1,4 @@ - + + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -154,6 +154,7 @@ true + ; hadoop-client-artifacts diff --git a/hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh b/hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh index 84efe7e08ce1c..7242ade356fda 100644 --- a/hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh +++ b/hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh @@ -15,13 +15,24 @@ # See the License for the specific language governing permissions and # limitations under the License. -# Usage: $0 [/path/to/some/example.jar:/path/to/another/example/created.jar] +# Usage: $0 [/path/to/some/example.jar;/path/to/another/example/created.jar] # # accepts a single command line argument with a colon separated list of # paths to jars to check. Iterates through each such passed jar and checks # all the contained paths to make sure they follow the below constructed # safe list. +# We use +=, which is a bash 3.1+ feature +if [[ -z "${BASH_VERSINFO[0]}" ]] \ + || [[ "${BASH_VERSINFO[0]}" -lt 3 ]] \ + || [[ "${BASH_VERSINFO[0]}" -eq 3 && "${BASH_VERSINFO[1]}" -lt 1 ]]; then + echo "bash v3.1+ is required. Sorry." + exit 1 +fi + +set -e +set -o pipefail + # we have to allow the directories that lead to the org/apache/hadoop dir allowed_expr="(^org/$|^org/apache/$" # We allow the following things to exist in our client artifacts: @@ -60,9 +71,23 @@ allowed_expr+="|^jetty-dir.css$" allowed_expr+=")" declare -i bad_artifacts=0 declare -a bad_contents -IFS=: read -r -d '' -a artifact_list < <(printf '%s\0' "$1") +declare -a artifact_list +while IFS='' read -r -d ';' line; do artifact_list+=("$line"); done < <(printf '%s;' "$1") +if [ "${#artifact_list[@]}" -eq 0 ]; then + echo "[ERROR] No artifacts passed in." + exit 1 +fi + +jar_list_failed () +{ + echo "[ERROR] Listing jar contents for file '${artifact}' failed." + exit 1 +} +trap jar_list_failed SIGUSR1 + for artifact in "${artifact_list[@]}"; do - bad_contents=($(jar tf "${artifact}" | grep -v -E "${allowed_expr}")) + # Note: On Windows the output from jar tf may contain \r\n's. Normalize to \n. + while IFS='' read -r line; do bad_contents+=("$line"); done < <( ( jar tf "${artifact}" | sed 's/\\r//' || kill -SIGUSR1 $$ ) | grep -v -E "${allowed_expr}" ) if [ ${#bad_contents[@]} -gt 0 ]; then echo "[ERROR] Found artifact with unexpected contents: '${artifact}'" echo " Please check the following and either correct the build or update" diff --git a/hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml b/hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml index c02ce5658dcc4..08b4fb27befd9 100644 --- a/hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml +++ b/hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml @@ -1,4 +1,4 @@ - + + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -164,6 +164,7 @@ hadoop-client-api,hadoop-client-runtime true + ; hadoop-client-artifacts diff --git a/hadoop-client-modules/hadoop-client-check-test-invariants/src/test/resources/ensure-jars-have-correct-contents.sh b/hadoop-client-modules/hadoop-client-check-test-invariants/src/test/resources/ensure-jars-have-correct-contents.sh index f8c6a15675709..08f9202972735 100644 --- a/hadoop-client-modules/hadoop-client-check-test-invariants/src/test/resources/ensure-jars-have-correct-contents.sh +++ b/hadoop-client-modules/hadoop-client-check-test-invariants/src/test/resources/ensure-jars-have-correct-contents.sh @@ -15,13 +15,24 @@ # See the License for the specific language governing permissions and # limitations under the License. -# Usage: $0 [/path/to/some/example.jar:/path/to/another/example/created.jar] +# Usage: $0 [/path/to/some/example.jar;/path/to/another/example/created.jar] # # accepts a single command line argument with a colon separated list of # paths to jars to check. Iterates through each such passed jar and checks # all the contained paths to make sure they follow the below constructed # safe list. +# We use +=, which is a bash 3.1+ feature +if [[ -z "${BASH_VERSINFO[0]}" ]] \ + || [[ "${BASH_VERSINFO[0]}" -lt 3 ]] \ + || [[ "${BASH_VERSINFO[0]}" -eq 3 && "${BASH_VERSINFO[1]}" -lt 1 ]]; then + echo "bash v3.1+ is required. Sorry." + exit 1 +fi + +set -e +set -o pipefail + # we have to allow the directories that lead to the org/apache/hadoop dir allowed_expr="(^org/$|^org/apache/$" # We allow the following things to exist in our client artifacts: @@ -30,7 +41,7 @@ allowed_expr="(^org/$|^org/apache/$" allowed_expr+="|^org/apache/hadoop/" # * whatever in the "META-INF" directory allowed_expr+="|^META-INF/" -# * whatever under the "webapps" directory; for minicluster UIs +# * whatever under the "webapps" directory; for things shipped by yarn allowed_expr+="|^webapps/" # * Hadoop's default configuration files, which have the form # "_module_-default.xml" @@ -54,9 +65,23 @@ allowed_expr+="|^librocksdbjni-linux-ppc64le.so" allowed_expr+=")" declare -i bad_artifacts=0 declare -a bad_contents -IFS=: read -r -d '' -a artifact_list < <(printf '%s\0' "$1") +declare -a artifact_list +while IFS='' read -r -d ';' line; do artifact_list+=("$line"); done < <(printf '%s;' "$1") +if [ "${#artifact_list[@]}" -eq 0 ]; then + echo "[ERROR] No artifacts passed in." + exit 1 +fi + +jar_list_failed () +{ + echo "[ERROR] Listing jar contents for file '${artifact}' failed." + exit 1 +} +trap jar_list_failed SIGUSR1 + for artifact in "${artifact_list[@]}"; do - bad_contents=($(jar tf "${artifact}" | grep -v -E "${allowed_expr}")) + # Note: On Windows the output from jar tf may contain \r\n's. Normalize to \n. + while IFS='' read -r line; do bad_contents+=("$line"); done < <( ( jar tf "${artifact}" | sed 's/\\r//' || kill -SIGUSR1 $$ ) | grep -v -E "${allowed_expr}" ) if [ ${#bad_contents[@]} -gt 0 ]; then echo "[ERROR] Found artifact with unexpected contents: '${artifact}'" echo " Please check the following and either correct the build or update" diff --git a/hadoop-client-modules/hadoop-client-integration-tests/pom.xml b/hadoop-client-modules/hadoop-client-integration-tests/pom.xml index 558372d5ae42b..1a14549250c3e 100644 --- a/hadoop-client-modules/hadoop-client-integration-tests/pom.xml +++ b/hadoop-client-modules/hadoop-client-integration-tests/pom.xml @@ -13,7 +13,7 @@ limitations under the License. See accompanying LICENSE file. --> + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-client-modules/hadoop-client-minicluster/pom.xml b/hadoop-client-modules/hadoop-client-minicluster/pom.xml index 9007462771486..52595d93523e9 100644 --- a/hadoop-client-modules/hadoop-client-minicluster/pom.xml +++ b/hadoop-client-modules/hadoop-client-minicluster/pom.xml @@ -13,7 +13,7 @@ limitations under the License. See accompanying LICENSE file. --> + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -143,7 +143,7 @@ hadoop-yarn-common - org.fusesource.leveldbjni + ${leveldbjni.group} leveldbjni-all @@ -484,7 +484,7 @@ hadoop-yarn-server-common - org.fusesource.leveldbjni + ${leveldbjni.group} leveldbjni-all @@ -811,6 +811,15 @@ */** + + + org.eclipse.jetty:jetty-client + + */** + + @@ -939,6 +948,13 @@ **/pom.xml + + javax/websocket/ + ${shaded.dependency.prefix}.javax.websocket. + + **/pom.xml + + jersey/ ${shaded.dependency.prefix}.jersey. diff --git a/hadoop-client-modules/hadoop-client-runtime/pom.xml b/hadoop-client-modules/hadoop-client-runtime/pom.xml index 1e98a61a94c74..552cd9c1d88c7 100644 --- a/hadoop-client-modules/hadoop-client-runtime/pom.xml +++ b/hadoop-client-modules/hadoop-client-runtime/pom.xml @@ -13,7 +13,7 @@ limitations under the License. See accompanying LICENSE file. --> + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -161,6 +161,9 @@ org.eclipse.jetty.websocket:* org.eclipse.jetty:jetty-servlet org.eclipse.jetty:jetty-security + org.eclipse.jetty:jetty-client + org.eclipse.jetty:jetty-http + org.eclipse.jetty:jetty-xml org.ow2.asm:* org.bouncycastle:* @@ -229,6 +232,13 @@ update* + + com.google.protobuf:protobuf-java + + google/protobuf/*.proto + google/protobuf/**/*.proto + + diff --git a/hadoop-client-modules/hadoop-client/pom.xml b/hadoop-client-modules/hadoop-client/pom.xml index 86b23fe82f4b8..9216a2e54a397 100644 --- a/hadoop-client-modules/hadoop-client/pom.xml +++ b/hadoop-client-modules/hadoop-client/pom.xml @@ -13,7 +13,7 @@ limitations under the License. See accompanying LICENSE file. --> + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-client-modules/pom.xml b/hadoop-client-modules/pom.xml index 3273240a730f2..0895e31ca307f 100644 --- a/hadoop-client-modules/pom.xml +++ b/hadoop-client-modules/pom.xml @@ -13,7 +13,7 @@ limitations under the License. See accompanying LICENSE file. --> + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml b/hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml index 16e9582434eac..b5e35b079f9fd 100644 --- a/hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml +++ b/hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml @@ -13,7 +13,7 @@ limitations under the License. See accompanying LICENSE file. --> + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-cloud-storage-project/hadoop-cos/dev-support/findbugs-exclude.xml b/hadoop-cloud-storage-project/hadoop-cos/dev-support/findbugs-exclude.xml new file mode 100644 index 0000000000000..40d78d0cd6cec --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/dev-support/findbugs-exclude.xml @@ -0,0 +1,18 @@ + + + diff --git a/hadoop-cloud-storage-project/hadoop-cos/pom.xml b/hadoop-cloud-storage-project/hadoop-cos/pom.xml new file mode 100644 index 0000000000000..839bd04c9b643 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/pom.xml @@ -0,0 +1,140 @@ + + + + + 4.0.0 + + org.apache.hadoop + hadoop-project + 3.3.0-SNAPSHOT + ../../hadoop-project + + hadoop-cos + Apache Hadoop Tencent COS Support + + This module contains code to support integration with Tencent COS. + It also declares the dependencies needed to work with COS. + + jar + + + UTF-8 + true + + + + + tests-off + + + src/test/resources/auth-keys.xml + + + + true + + + + tests-on + + + src/test/resources/auth-keys.xml + + + + false + + + + + + + + org.codehaus.mojo + findbugs-maven-plugin + + true + true + ${basedir}/dev-support/findbugs-exclude.xml + + Max + + + + org.apache.maven.plugins + maven-surefire-plugin + + 3600 + + + + + + + + junit + junit + test + + + + com.qcloud + cos_api + 5.4.9 + compile + + + + org.apache.hadoop + hadoop-common + provided + + + + org.apache.hadoop + hadoop-common + test + test-jar + + + + org.apache.hadoop + hadoop-yarn-server-tests + test + test-jar + + + + org.apache.hadoop + hadoop-mapreduce-client-hs + test + + + + org.apache.hadoop + hadoop-distcp + test + + + org.apache.hadoop + hadoop-distcp + test + test-jar + + + + diff --git a/hadoop-cloud-storage-project/hadoop-cos/site/markdown/cloud-storage/index.md b/hadoop-cloud-storage-project/hadoop-cos/site/markdown/cloud-storage/index.md new file mode 100644 index 0000000000000..7049b3f0f013f --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/site/markdown/cloud-storage/index.md @@ -0,0 +1,367 @@ + + +# Integeration of Tencent COS in Hadoop + +## Introduction + +[Tencent COS](https://intl.cloud.tencent.com/product/cos) is a famous object storage system provided by Tencent Corp. Hadoop-COS is a client that makes the upper computing systems based on HDFS be able to use the COS as its underlying storage system. The big data-processing systems that have been identified for support are: Hadoop MR, Spark, Alluxio and etc. In addition, Druid also can use COS as its deep storage by configuring HDFS-Load-Plugin integerating with HADOOP-COS. + + +## Features + +- Support Hadoop MapReduce and Spark write data into COS and read from it directly. + +- Implements the interfaces of the Hadoop file system and provides the pseudo-hierarchical directory structure same as HDFS. + +- Supports multipart uploads for a large file. Single file supports up to 19TB + +- High performance and high availability. The performance difference between Hadoop-COS and HDFS is not more than 30%. + + +> Notes: + +> Object Storage is not a file system and it has some limitations: + +> 1. Object storage is a key-value storage and it does not support hierarchical directory naturally. Usually, using the directory separatory in object key to simulate the hierarchical directory, such as "/hadoop/data/words.dat". + +> 2. COS Object storage can not support the object's append operation currently. It means that you can not append content to the end of an existing object(file). + +> 3. Both `delete` and `rename` operations are non-atomic, which means that the operations are interrupted, the operation result may be inconsistent state. + +> 4. Object storages have different authorization models: + +> - Directory permissions are reported as 777. + +> - File permissions are reported as 666. + +> - File owner is reported as the local current user. + +> - File group is also reported as the local current user. + +> 5. Supports multipart uploads for a large file(up to 40TB), but the number of part is limited as 10000. + +> 6. The number of files listed each time is limited to 1000. + + +## Quick Start + +### Concepts + +- **Bucket**: A container for storing data in COS. Its name is made up of user-defined bucketname and user appid. + +- **Appid**: Unique resource identifier for the user dimension. + +- **SecretId**: ID used to authenticate the user + +- **SecretKey**: Key used to authenticate the user + +- **Region**: The region where a bucket locates. + +- **CosN**: Hadoop-COS uses `cosn` as its URI scheme, so CosN is often used to refer to Hadoop-COS. + + +### Usage + +#### System Requirements + +Linux kernel 2.6+ + + +#### Dependencies + +- cos_api (version 5.4.10 or later ) +- cos-java-sdk (version 2.0.6 recommended) +- joda-time (version 2.9.9 recommended) +- httpClient (version 4.5.1 or later recommended) +- Jackson: jackson-core, jackson-databind, jackson-annotations (version 2.9.8 or later) +- bcprov-jdk15on (version 1.59 recommended) + + +#### Configure Properties + +##### URI and Region Properties + +If you plan to use COS as the default file system for Hadoop or other big data systems, you need to configure `fs.defaultFS` as the URI of Hadoop-COS in core-site.xml. Hadoop-COS uses `cosn` as its URI scheme, and the bucket as its URI host. At the same time, you need to explicitly set `fs.cosn.userinfo.region` to indicate the region your bucket locates. + +**NOTE**: + +- For Hadoop-COS, `fs.defaultFS` is an option. If you are only temporarily using the COS as a data source for Hadoop, you do not need to set the property, just specify the full URI when you use it. For example: `hadoop fs -ls cosn://testBucket-125236746/testDir/test.txt`. + +- `fs.cosn.userinfo.region` is an required property for Hadoop-COS. The reason is that Hadoop-COS must know the region of the using bucket in order to accurately construct a URL to access it. + +- COS supports multi-region storage, and different regions have different access domains by default. It is recommended to choose the nearest storage region according to your own business scenarios, so as to improve the object upload and download speed. You can find the available region from [https://intl.cloud.tencent.com/document/product/436/6224](https://intl.cloud.tencent.com/document/product/436/6224) + +The following is an example for the configuration format: + +```xml + + fs.defaultFS + cosn:// + + Optional: If you don't want to use CosN as the default file system, you don't need to configure it. + + + + + fs.cosn.bucket.region + ap-xxx + The region where the bucket is located + + +``` + + +##### User Authentication Properties + +Each user needs to properly configure the credentials ( User's secreteId and secretKey ) properly to access the object stored in COS. These credentials can be obtained from the official console provided by Tencent Cloud. + +```xml + + fs.cosn.credentials.provider + org.apache.hadoop.fs.auth.SimpleCredentialProvider + + + This option allows the user to specify how to get the credentials. + Comma-separated class names of credential provider classes which implement + com.qcloud.cos.auth.COSCredentialsProvider: + + 1.org.apache.hadoop.fs.auth.SimpleCredentialProvider: Obtain the secret id and secret key + from fs.cosn.userinfo.secretId and fs.cosn.userinfo.secretKey in core-site.xml + 2.org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider: Obtain the secret id and secret key from system environment variables named COS_SECRET_ID and COS_SECRET_KEY + + If unspecified, the default order of credential providers is: + 1. org.apache.hadoop.fs.auth.SimpleCredentialProvider + 2. org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider + + + + + + fs.cosn.userinfo.secretId + xxxxxxxxxxxxxxxxxxxxxxxxx + Tencent Cloud Secret Id + + + + fs.cosn.userinfo.secretKey + xxxxxxxxxxxxxxxxxxxxxxxx + Tencent Cloud Secret Key + + +``` + + +##### Integration Properties + +You need to explicitly specify the A and B options in order for Hadoop to properly integrate the COS as the underlying file system + +Only correctly set `fs.cosn.impl` and `fs.AbstractFileSystem.cosn.impl` to enable Hadoop to integrate COS as its underlying file system. `fs.cosn.impl` must be set as `org.apache.hadoop.fs.cos.CosFileSystem` and `fs.AbstractFileSystem.cosn.impl` must be set as `org.apache.hadoop.fs.cos.CosN`. + +```xml + + fs.cosn.impl + org.apache.hadoop.fs.cosn.CosNFileSystem + The implementation class of the CosN Filesystem + + + + fs.AbstractFileSystem.cosn.impl + org.apache.hadoop.fs.cos.CosN + The implementation class of the CosN AbstractFileSystem. + + +``` + +##### Other Runtime Properties + +Hadoop-COS provides rich runtime properties to set, and most of these do not require custom values because a well-run default value provided for them. + +**It is important to note that**: + +- Hadoop-COS will generate some temporary files and consumes some disk space. All temporary files would be placed in the directory specified by option `fs.cosn.tmp.dir` (Default: /tmp/hadoop_cos); + +- The default block size is 8MB and it means that you can only upload a single file up to 78GB into the COS blob storage system. That is mainly due to the fact that the multipart-upload can only support up to 10,000 blocks. For this reason, if needing to support larger single files, you must increase the block size accordingly by setting the property `fs.cosn.block.size`. For example, the size of the largest single file is 1TB, the block size is at least greater than or equal to (1 \* 1024 \* 1024 \* 1024 \* 1024)/10000 = 109951163. Currently, the maximum support file is 19TB (block size: 2147483648) + +```xml + + fs.cosn.tmp.dir + /tmp/hadoop_cos + Temporary files would be placed here. + + + + fs.cosn.buffer.size + 33554432 + The total size of the buffer pool. + + + + fs.cosn.block.size + 8388608 + + Block size to use cosn filesysten, which is the part size for MultipartUpload. Considering the COS supports up to 10000 blocks, user should estimate the maximum size of a single file. For example, 8MB part size can allow writing a 78GB single file. + + + + + fs.cosn.maxRetries + 3 + + The maximum number of retries for reading or writing files to COS, before throwing a failure to the application. + + + + + fs.cosn.retry.interval.seconds + 3 + The number of seconds to sleep between each COS retry. + + +``` + + +##### Properties Summary + +| properties | description | default value | required | +|:----------:|:-----------|:-------------:|:--------:| +| fs.defaultFS | Configure the default file system used by Hadoop.| None | NO | +| fs.cosn.credentials.provider | This option allows the user to specify how to get the credentials. Comma-separated class names of credential provider classes which implement com.qcloud.cos.auth.COSCredentialsProvider:
1. org.apache.hadoop.fs.cos.auth.SimpleCredentialProvider: Obtain the secret id and secret key from `fs.cosn.userinfo.secretId` and `fs.cosn.userinfo.secretKey` in core-site.xml;
2. org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider: Obtain the secret id and secret key from system environment variables named `COSN_SECRET_ID` and `COSN_SECRET_KEY`.

If unspecified, the default order of credential providers is:
1. org.apache.hadoop.fs.auth.SimpleCredentialProvider;
2. org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider. | None | NO | +| fs.cosn.userinfo.secretId/secretKey | The API key information of your account | None | YES | +| fs.cosn.bucket.region | The region where the bucket is located. | None | YES | +| fs.cosn.impl | The implementation class of the CosN filesystem. | None | YES | +| fs.AbstractFileSystem.cosn.impl | The implementation class of the CosN AbstractFileSystem. | None | YES | +| fs.cosn.tmp.dir | Temporary files generated by cosn would be stored here during the program running. | /tmp/hadoop_cos | NO | +| fs.cosn.buffer.size | The total size of the buffer pool. Require greater than or equal to block size. | 33554432 | NO | +| fs.cosn.block.size | The size of file block. Considering the limitation that each file can be divided into a maximum of 10,000 to upload, the option must be set according to the maximum size of used single file. For example, 8MB part size can allow writing a 78GB single file. | 8388608 | NO | +| fs.cosn.upload_thread_pool | Number of threads used for concurrent uploads when files are streamed to COS. | CPU core number * 3 | NO | +| fs.cosn.read.ahead.block.size | The size of each read-ahead block. | 524288 (512KB) | NO | +| fs.cosn.read.ahead.queue.size | The length of readahead queue. | 10 | NO | +| fs.cosn.maxRetries | The maxium number of retries for reading or writing files to COS, before throwing a failure to the application. | 3 | NO | +| fs.cosn.retry.interval.seconds | The number of seconds to sleep between each retry | 3 | NO | + + +#### Command Usage + +Command format: `hadoop fs -ls -R cosn://bucket-appid/` or `hadoop fs -ls -R /`, the latter requires the defaultFs option to be set as `cosn`. + + +#### Example + +Use CosN as the underlying file system to run the WordCount routine: + +```shell +bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-x.x.x.jar wordcount cosn://example/mr/input.txt cosn://example/mr/output +``` + +If setting CosN as the default file system for Hadoop, you can run it as follows: + +```shell +bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-x.x.x.jar wordcount /mr/input.txt /mr/output +``` + +## Testing the hadoop-cos Module + +To test CosN filesystem, the following two files which pass in authentication details to the test runner are needed. + +1. auth-keys.xml +2. core-site.xml + +These two files need to be created under the `hadoop-cloud-storage-project/hadoop-cos/src/test/resource` directory. + + +### `auth-key.xml` + +COS credentials can specified in `auth-key.xml`. At the same time, it is also a trigger for the CosN filesystem tests. +COS bucket URL should be provided by specifying the option: `test.fs.cosn.name`. + +An example of the `auth-keys.xml` is as follow: + +```xml + + + test.fs.cosn.name + cosn://testbucket-12xxxxxx + + + fs.cosn.bucket.region + ap-xxx + The region where the bucket is located + + + fs.cosn.userinfo.secretId + AKIDXXXXXXXXXXXXXXXXXXXX + + + fs.cosn.userinfo.secretKey + xxxxxxxxxxxxxxxxxxxxxxxxx + + + + +``` + +Without this file, all tests in this module will be skipped. + +### `core-site.xml` + +This file pre-exists and sources the configurations created in auth-keys.xml. +For most cases, no modification is needed, unless a specific, non-default property needs to be set during the testing. + +### `contract-test-options.xml` + +All configurations related to support contract tests need to be specified in `contract-test-options.xml`. Here is an example of `contract-test-options.xml`. + +```xml + + + + + + fs.contract.test.fs.cosn + cosn://testbucket-12xxxxxx + + + + fs.cosn.bucket.region + ap-xxx + The region where the bucket is located + + + + +``` + +If the option `fs.contract.test.fs.cosn` not definded in the file, all contract tests will be skipped. + +## Other issues + +### Performance Loss + +The IO performance of COS is lower than HDFS in principle, even on virtual clusters running on Tencent CVM. + +The main reason can be attributed to the following points: + +- HDFS replicates data for faster query. + +- HDFS is significantly faster for many “metadata” operations: listing the contents of a directory, calling getFileStatus() on path, creating or deleting directories. + +- HDFS stores the data on the local hard disks, avoiding network traffic if the code can be executed on that host. But access to the object storing in COS requires access to network almost each time. It is a critical point in damaging IO performance. Hadoop-COS also do a lot of optimization work for it, such as the pre-read queue, the upload buffer pool, the concurrent upload thread pool, etc. + +- File IO performing many seek calls/positioned read calls will also encounter performance problems due to the size of the HTTP requests made. Despite the pre-read cache optimizations, a large number of random reads can still cause frequent network requests. + +- On HDFS, both the `rename` and `mv` for a directory or a file are an atomic and O(1)-level operation, but in COS, the operation need to combine `copy` and `delete` sequentially. Therefore, performing rename and move operations on a COS object is not only low performance, but also difficult to guarantee data consistency. + +At present, using the COS blob storage system through Hadoop-COS occurs about 20% ~ 25% performance loss compared to HDFS. But, the cost of using COS is lower than HDFS, which includes both storage and maintenance costs. diff --git a/hadoop-submarine/hadoop-submarine-tony-runtime/src/site/resources/css/site.css b/hadoop-cloud-storage-project/hadoop-cos/site/resources/css/site.css similarity index 100% rename from hadoop-submarine/hadoop-submarine-tony-runtime/src/site/resources/css/site.css rename to hadoop-cloud-storage-project/hadoop-cos/site/resources/css/site.css diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/BufferPool.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/BufferPool.java new file mode 100644 index 0000000000000..a4ee4d5be9ac8 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/BufferPool.java @@ -0,0 +1,245 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.File; +import java.io.IOException; +import java.io.RandomAccessFile; +import java.nio.ByteBuffer; +import java.nio.MappedByteBuffer; +import java.nio.channels.FileChannel; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.hadoop.conf.Configuration; + +/** + * BufferPool class is used to manage the buffers during program execution. + * It is provided in a thread-safe singleton mode,and + * keeps the program's memory and disk consumption at a stable value. + */ +public final class BufferPool { + private static final Logger LOG = + LoggerFactory.getLogger(BufferPool.class); + + private static BufferPool ourInstance = new BufferPool(); + + /** + * Use this method to get the instance of BufferPool. + * + * @return the instance of BufferPool + */ + public static BufferPool getInstance() { + return ourInstance; + } + + private BlockingQueue bufferPool = null; + private long singleBufferSize = 0; + private File diskBufferDir = null; + + private AtomicBoolean isInitialize = new AtomicBoolean(false); + + private BufferPool() { + } + + private File createDir(String dirPath) throws IOException { + File dir = new File(dirPath); + if (null != dir) { + if (!dir.exists()) { + LOG.debug("Buffer dir: [{}] does not exists. create it first.", + dirPath); + if (dir.mkdirs()) { + if (!dir.setWritable(true) || !dir.setReadable(true) + || !dir.setExecutable(true)) { + LOG.warn("Set the buffer dir: [{}]'s permission [writable," + + "readable, executable] failed.", dir.getAbsolutePath()); + } + LOG.debug("Buffer dir: [{}] is created successfully.", + dir.getAbsolutePath()); + } else { + // Once again, check if it has been created successfully. + // Prevent problems created by multiple processes at the same time. + if (!dir.exists()) { + throw new IOException("buffer dir:" + dir.getAbsolutePath() + + " is created unsuccessfully"); + } + } + } else { + LOG.debug("buffer dir: {} already exists.", dirPath); + } + } else { + throw new IOException("creating buffer dir: " + dir.getAbsolutePath() + + "unsuccessfully."); + } + + return dir; + } + + /** + * Create buffers correctly by reading the buffer file directory, + * buffer pool size,and file block size in the configuration. + * + * @param conf Provides configurations for the Hadoop runtime + * @throws IOException Configuration errors, + * insufficient or no access for memory or + * disk space may cause this exception + */ + public synchronized void initialize(Configuration conf) + throws IOException { + if (this.isInitialize.get()) { + return; + } + this.singleBufferSize = conf.getLong(CosNConfigKeys.COSN_BLOCK_SIZE_KEY, + CosNConfigKeys.DEFAULT_BLOCK_SIZE); + + // The block size of CosN can only support up to 2GB. + if (this.singleBufferSize < Constants.MIN_PART_SIZE + || this.singleBufferSize > Constants.MAX_PART_SIZE) { + String exceptionMsg = String.format( + "The block size of CosN is limited to %d to %d", + Constants.MIN_PART_SIZE, Constants.MAX_PART_SIZE); + throw new IOException(exceptionMsg); + } + + long memoryBufferLimit = conf.getLong( + CosNConfigKeys.COSN_UPLOAD_BUFFER_SIZE_KEY, + CosNConfigKeys.DEFAULT_UPLOAD_BUFFER_SIZE); + + this.diskBufferDir = this.createDir(conf.get( + CosNConfigKeys.COSN_BUFFER_DIR_KEY, + CosNConfigKeys.DEFAULT_BUFFER_DIR)); + + int bufferPoolSize = (int) (memoryBufferLimit / this.singleBufferSize); + if (0 == bufferPoolSize) { + throw new IOException( + String.format("The total size of the buffer [%d] is " + + "smaller than a single block [%d]." + + "please consider increase the buffer size " + + "or decrease the block size", + memoryBufferLimit, this.singleBufferSize)); + } + this.bufferPool = new LinkedBlockingQueue<>(bufferPoolSize); + for (int i = 0; i < bufferPoolSize; i++) { + this.bufferPool.add(ByteBuffer.allocateDirect( + (int) this.singleBufferSize)); + } + + this.isInitialize.set(true); + } + + /** + * Check if the buffer pool has been initialized. + * + * @throws IOException if the buffer pool is not initialized + */ + private void checkInitialize() throws IOException { + if (!this.isInitialize.get()) { + throw new IOException( + "The buffer pool has not been initialized yet"); + } + } + + /** + * Obtain a buffer from this buffer pool through the method. + * + * @param bufferSize expected buffer size to get + * @return a buffer wrapper that satisfies the bufferSize. + * @throws IOException if the buffer pool not initialized, + * or the bufferSize parameter is not within + * the range[1MB to the single buffer size] + */ + public ByteBufferWrapper getBuffer(int bufferSize) throws IOException { + this.checkInitialize(); + if (bufferSize > 0 && bufferSize <= this.singleBufferSize) { + ByteBufferWrapper byteBufferWrapper = this.getByteBuffer(); + if (null == byteBufferWrapper) { + // Use a disk buffer when the memory buffer is not enough + byteBufferWrapper = this.getMappedBuffer(); + } + return byteBufferWrapper; + } else { + String exceptionMsg = String.format( + "Parameter buffer size out of range: 1048576 to %d", + this.singleBufferSize + ); + throw new IOException(exceptionMsg); + } + } + + /** + * Get a ByteBufferWrapper from the buffer pool. + * + * @return a new byte buffer wrapper + * @throws IOException if the buffer pool is not initialized + */ + private ByteBufferWrapper getByteBuffer() throws IOException { + this.checkInitialize(); + ByteBuffer buffer = this.bufferPool.poll(); + return buffer == null ? null : new ByteBufferWrapper(buffer); + } + + /** + * Get a mapped buffer from the buffer pool. + * + * @return a new mapped buffer + * @throws IOException If the buffer pool is not initialized. + * or some I/O error occurs + */ + private ByteBufferWrapper getMappedBuffer() throws IOException { + this.checkInitialize(); + File tmpFile = File.createTempFile(Constants.BLOCK_TMP_FILE_PREFIX, + Constants.BLOCK_TMP_FILE_SUFFIX, this.diskBufferDir); + tmpFile.deleteOnExit(); + RandomAccessFile raf = new RandomAccessFile(tmpFile, "rw"); + raf.setLength(this.singleBufferSize); + MappedByteBuffer buf = raf.getChannel().map( + FileChannel.MapMode.READ_WRITE, 0, this.singleBufferSize); + return new ByteBufferWrapper(buf, raf, tmpFile); + } + + /** + * return the byte buffer wrapper to the buffer pool. + * + * @param byteBufferWrapper the byte buffer wrapper getting from the pool + * @throws InterruptedException if interrupted while waiting + * @throws IOException some io error occurs + */ + public void returnBuffer(ByteBufferWrapper byteBufferWrapper) + throws InterruptedException, IOException { + if (null == this.bufferPool || null == byteBufferWrapper) { + return; + } + + if (byteBufferWrapper.isDiskBuffer()) { + byteBufferWrapper.close(); + } else { + ByteBuffer byteBuffer = byteBufferWrapper.getByteBuffer(); + if (null != byteBuffer) { + byteBuffer.clear(); + LOG.debug("Return the buffer to the buffer pool."); + if (!this.bufferPool.offer(byteBuffer)) { + LOG.error("Return the buffer to buffer pool failed."); + } + } + } + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferInputStream.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferInputStream.java new file mode 100644 index 0000000000000..440a7deda09d8 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferInputStream.java @@ -0,0 +1,89 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.InvalidMarkException; + +/** + * The input stream class is used for buffered files. + * The purpose of providing this class is to optimize buffer read performance. + */ +public class ByteBufferInputStream extends InputStream { + private ByteBuffer byteBuffer; + private boolean isClosed; + + public ByteBufferInputStream(ByteBuffer byteBuffer) throws IOException { + if (null == byteBuffer) { + throw new IOException("byte buffer is null"); + } + this.byteBuffer = byteBuffer; + this.isClosed = false; + } + + @Override + public int read() throws IOException { + if (null == this.byteBuffer) { + throw new IOException("this byte buffer for InputStream is null"); + } + if (!this.byteBuffer.hasRemaining()) { + return -1; + } + return this.byteBuffer.get() & 0xFF; + } + + @Override + public synchronized void mark(int readLimit) { + if (!this.markSupported()) { + return; + } + this.byteBuffer.mark(); + // Parameter readLimit is ignored + } + + @Override + public boolean markSupported() { + return true; + } + + @Override + public synchronized void reset() throws IOException { + if (this.isClosed) { + throw new IOException("Closed in InputStream"); + } + try { + this.byteBuffer.reset(); + } catch (InvalidMarkException e) { + throw new IOException("Invalid mark"); + } + } + + @Override + public int available() { + return this.byteBuffer.remaining(); + } + + @Override + public void close() { + this.byteBuffer.rewind(); + this.byteBuffer = null; + this.isClosed = true; + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferOutputStream.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferOutputStream.java new file mode 100644 index 0000000000000..9e6a6fc5f3663 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferOutputStream.java @@ -0,0 +1,74 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.IOException; +import java.io.OutputStream; +import java.nio.ByteBuffer; + +/** + * The input stream class is used for buffered files. + * The purpose of providing this class is to optimize buffer write performance. + */ +public class ByteBufferOutputStream extends OutputStream { + private ByteBuffer byteBuffer; + private boolean isFlush; + private boolean isClosed; + + public ByteBufferOutputStream(ByteBuffer byteBuffer) throws IOException { + if (null == byteBuffer) { + throw new IOException("byte buffer is null"); + } + this.byteBuffer = byteBuffer; + this.byteBuffer.clear(); + this.isFlush = false; + this.isClosed = false; + } + + @Override + public void write(int b) { + byte[] singleBytes = new byte[1]; + singleBytes[0] = (byte) b; + this.byteBuffer.put(singleBytes, 0, 1); + this.isFlush = false; + } + + @Override + public void flush() { + if (this.isFlush) { + return; + } + this.isFlush = true; + } + + @Override + public void close() throws IOException { + if (this.isClosed) { + return; + } + if (null == this.byteBuffer) { + throw new IOException("Can not close a null object"); + } + + this.flush(); + this.byteBuffer.flip(); + this.byteBuffer = null; + this.isFlush = false; + this.isClosed = true; + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferWrapper.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferWrapper.java new file mode 100644 index 0000000000000..a7d1c5fffce0e --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferWrapper.java @@ -0,0 +1,103 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.File; +import java.io.IOException; +import java.io.RandomAccessFile; +import java.nio.ByteBuffer; +import java.nio.MappedByteBuffer; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.hadoop.util.CleanerUtil; + +/** + * The wrapper for memory buffers and disk buffers. + */ +public class ByteBufferWrapper { + private static final Logger LOG = + LoggerFactory.getLogger(ByteBufferWrapper.class); + private ByteBuffer byteBuffer; + private File file; + private RandomAccessFile randomAccessFile; + + ByteBufferWrapper(ByteBuffer byteBuffer) { + this(byteBuffer, null, null); + } + + ByteBufferWrapper(ByteBuffer byteBuffer, RandomAccessFile randomAccessFile, + File file) { + this.byteBuffer = byteBuffer; + this.file = file; + this.randomAccessFile = randomAccessFile; + } + + public ByteBuffer getByteBuffer() { + return this.byteBuffer; + } + + boolean isDiskBuffer() { + return this.file != null && this.randomAccessFile != null; + } + + private void munmap(MappedByteBuffer buffer) { + if (CleanerUtil.UNMAP_SUPPORTED) { + try { + CleanerUtil.getCleaner().freeBuffer(buffer); + } catch (IOException e) { + LOG.warn("Failed to unmap the buffer", e); + } + } else { + LOG.trace(CleanerUtil.UNMAP_NOT_SUPPORTED_REASON); + } + } + + void close() throws IOException { + if (null != this.byteBuffer) { + this.byteBuffer.clear(); + } + + IOException exception = null; + // catch all exceptions, and try to free up resources that can be freed. + try { + if (null != randomAccessFile) { + this.randomAccessFile.close(); + } + } catch (IOException e) { + LOG.error("Close the random access file occurs an exception.", e); + exception = e; + } + + if (this.byteBuffer instanceof MappedByteBuffer) { + munmap((MappedByteBuffer) this.byteBuffer); + } + + if (null != this.file && this.file.exists()) { + if (!this.file.delete()) { + LOG.warn("Delete the tmp file: [{}] failed.", + this.file.getAbsolutePath()); + } + } + + if (null != exception) { + throw exception; + } + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/Constants.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/Constants.java new file mode 100644 index 0000000000000..f67e07ecff370 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/Constants.java @@ -0,0 +1,43 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +/** + * constant definition. + */ +public final class Constants { + private Constants() { + } + + public static final String BLOCK_TMP_FILE_PREFIX = "cos_"; + public static final String BLOCK_TMP_FILE_SUFFIX = "_local_block"; + + // The maximum number of files listed in a single COS list request. + public static final int COS_MAX_LISTING_LENGTH = 999; + + // The maximum number of parts supported by a multipart uploading. + public static final int MAX_PART_NUM = 10000; + + // The maximum size of a part + public static final long MAX_PART_SIZE = (long) 2 * Unit.GB; + // The minimum size of a part + public static final long MIN_PART_SIZE = (long) Unit.MB; + + public static final String COSN_SECRET_ID_ENV = "COSN_SECRET_ID"; + public static final String COSN_SECRET_KEY_ENV = "COSN_SECRET_KEY"; +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosN.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosN.java new file mode 100644 index 0000000000000..990fcbd56b3bb --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosN.java @@ -0,0 +1,41 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.IOException; +import java.net.URI; +import java.net.URISyntaxException; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.DelegateToFileSystem; + +/** + * CosN implementation for the Hadoop's AbstractFileSystem. + * This implementation delegates to the CosNFileSystem {@link CosNFileSystem}. + */ +public class CosN extends DelegateToFileSystem { + public CosN(URI theUri, Configuration conf) + throws IOException, URISyntaxException { + super(theUri, new CosNFileSystem(), conf, CosNFileSystem.SCHEME, false); + } + + @Override + public int getUriDefaultPort() { + return -1; + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNConfigKeys.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNConfigKeys.java new file mode 100644 index 0000000000000..4d98d5f7e2961 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNConfigKeys.java @@ -0,0 +1,86 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.fs.CommonConfigurationKeys; + +/** + * This class contains constants for configuration keys used in COS. + */ +@InterfaceAudience.Private +@InterfaceStability.Unstable +public class CosNConfigKeys extends CommonConfigurationKeys { + public static final String USER_AGENT = "fs.cosn.user.agent"; + public static final String DEFAULT_USER_AGENT = "cos-hadoop-plugin-v5.3"; + + public static final String COSN_CREDENTIALS_PROVIDER = + "fs.cosn.credentials.provider"; + public static final String COSN_SECRET_ID_KEY = "fs.cosn.userinfo.secretId"; + public static final String COSN_SECRET_KEY_KEY = "fs.cosn.userinfo.secretKey"; + public static final String COSN_REGION_KEY = "fs.cosn.bucket.region"; + public static final String COSN_ENDPOINT_SUFFIX_KEY = + "fs.cosn.bucket.endpoint_suffix"; + + public static final String COSN_USE_HTTPS_KEY = "fs.cosn.useHttps"; + public static final boolean DEFAULT_USE_HTTPS = false; + + public static final String COSN_BUFFER_DIR_KEY = "fs.cosn.tmp.dir"; + public static final String DEFAULT_BUFFER_DIR = "/tmp/hadoop_cos"; + + public static final String COSN_UPLOAD_BUFFER_SIZE_KEY = + "fs.cosn.buffer.size"; + public static final long DEFAULT_UPLOAD_BUFFER_SIZE = 32 * Unit.MB; + + public static final String COSN_BLOCK_SIZE_KEY = "fs.cosn.block.size"; + public static final long DEFAULT_BLOCK_SIZE = 8 * Unit.MB; + + public static final String COSN_MAX_RETRIES_KEY = "fs.cosn.maxRetries"; + public static final int DEFAULT_MAX_RETRIES = 3; + public static final String COSN_RETRY_INTERVAL_KEY = + "fs.cosn.retry.interval.seconds"; + public static final long DEFAULT_RETRY_INTERVAL = 3; + + public static final String UPLOAD_THREAD_POOL_SIZE_KEY = + "fs.cosn.upload_thread_pool"; + public static final int DEFAULT_UPLOAD_THREAD_POOL_SIZE = 1; + + public static final String COPY_THREAD_POOL_SIZE_KEY = + "fs.cosn.copy_thread_pool"; + public static final int DEFAULT_COPY_THREAD_POOL_SIZE = 1; + + /** + * This is the maximum time that excess idle threads will wait for new tasks + * before terminating. The time unit for it is second. + */ + public static final String THREAD_KEEP_ALIVE_TIME_KEY = + "fs.cosn.threads.keep_alive_time"; + // The default keep_alive_time is 60 seconds. + public static final long DEFAULT_THREAD_KEEP_ALIVE_TIME = 60L; + + public static final String READ_AHEAD_BLOCK_SIZE_KEY = + "fs.cosn.read.ahead.block.size"; + public static final long DEFAULT_READ_AHEAD_BLOCK_SIZE = 512 * Unit.KB; + public static final String READ_AHEAD_QUEUE_SIZE = + "fs.cosn.read.ahead.queue.size"; + public static final int DEFAULT_READ_AHEAD_QUEUE_SIZE = 5; + + public static final String MAX_CONNECTION_NUM = "fs.cosn.max.connection.num"; + public static final int DEFAULT_MAX_CONNECTION_NUM = 2048; +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNCopyFileContext.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNCopyFileContext.java new file mode 100644 index 0000000000000..39a2e91351f23 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNCopyFileContext.java @@ -0,0 +1,66 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.locks.Condition; +import java.util.concurrent.locks.ReentrantLock; + +/** + * The context of the copy task, including concurrency control, + * asynchronous acquisition of copy results and etc. + */ +public class CosNCopyFileContext { + + private final ReentrantLock lock = new ReentrantLock(); + private Condition readyCondition = lock.newCondition(); + + private AtomicBoolean copySuccess = new AtomicBoolean(true); + private AtomicInteger copiesFinish = new AtomicInteger(0); + + public void lock() { + this.lock.lock(); + } + + public void unlock() { + this.lock.unlock(); + } + + public void awaitAllFinish(int waitCopiesFinish) throws InterruptedException { + while (this.copiesFinish.get() != waitCopiesFinish) { + this.readyCondition.await(); + } + } + + public void signalAll() { + this.readyCondition.signalAll(); + } + + public boolean isCopySuccess() { + return this.copySuccess.get(); + } + + public void setCopySuccess(boolean copySuccess) { + this.copySuccess.set(copySuccess); + } + + public void incCopiesFinish() { + this.copiesFinish.addAndGet(1); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNCopyFileTask.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNCopyFileTask.java new file mode 100644 index 0000000000000..33d38b80e2e0a --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNCopyFileTask.java @@ -0,0 +1,68 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.IOException; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Used by {@link CosNFileSystem} as an task that submitted + * to the thread pool to accelerate the copy progress. + * Each task is responsible for copying the source key to the destination. + */ +public class CosNCopyFileTask implements Runnable { + private static final Logger LOG = + LoggerFactory.getLogger(CosNCopyFileTask.class); + + private NativeFileSystemStore store; + private String srcKey; + private String dstKey; + private CosNCopyFileContext cosCopyFileContext; + + public CosNCopyFileTask(NativeFileSystemStore store, String srcKey, + String dstKey, CosNCopyFileContext cosCopyFileContext) { + this.store = store; + this.srcKey = srcKey; + this.dstKey = dstKey; + this.cosCopyFileContext = cosCopyFileContext; + } + + @Override + public void run() { + boolean fail = false; + LOG.info(Thread.currentThread().getName() + "copying..."); + try { + this.store.copy(srcKey, dstKey); + } catch (IOException e) { + LOG.warn("Exception thrown when copy from {} to {}, exception:{}", + this.srcKey, this.dstKey, e); + fail = true; + } finally { + this.cosCopyFileContext.lock(); + if (fail) { + cosCopyFileContext.setCopySuccess(false); + } + cosCopyFileContext.incCopiesFinish(); + cosCopyFileContext.signalAll(); + this.cosCopyFileContext.unlock(); + } + } + +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNFileReadTask.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNFileReadTask.java new file mode 100644 index 0000000000000..a5dcdda07120b --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNFileReadTask.java @@ -0,0 +1,125 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.IOException; +import java.io.InputStream; +import java.util.HashMap; +import java.util.Map; +import java.util.concurrent.TimeUnit; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.io.IOUtils; +import org.apache.hadoop.io.retry.RetryPolicies; +import org.apache.hadoop.io.retry.RetryPolicy; + +/** + * Used by {@link CosNInputStream} as an asynchronous task + * submitted to the thread pool. + * Each task is responsible for reading a part of a large file. + * It is used to pre-read the data from COS to accelerate file reading process. + */ +public class CosNFileReadTask implements Runnable { + private static final Logger LOG = + LoggerFactory.getLogger(CosNFileReadTask.class); + + private final String key; + private final NativeFileSystemStore store; + private final CosNInputStream.ReadBuffer readBuffer; + + private RetryPolicy retryPolicy; + + public CosNFileReadTask( + Configuration conf, + String key, NativeFileSystemStore store, + CosNInputStream.ReadBuffer readBuffer) { + this.key = key; + this.store = store; + this.readBuffer = readBuffer; + + RetryPolicy defaultPolicy = + RetryPolicies.retryUpToMaximumCountWithFixedSleep( + conf.getInt( + CosNConfigKeys.COSN_MAX_RETRIES_KEY, + CosNConfigKeys.DEFAULT_MAX_RETRIES), + conf.getLong( + CosNConfigKeys.COSN_RETRY_INTERVAL_KEY, + CosNConfigKeys.DEFAULT_RETRY_INTERVAL), + TimeUnit.SECONDS); + Map, RetryPolicy> retryPolicyMap = + new HashMap<>(); + retryPolicyMap.put(IOException.class, defaultPolicy); + retryPolicyMap.put( + IndexOutOfBoundsException.class, RetryPolicies.TRY_ONCE_THEN_FAIL); + retryPolicyMap.put( + NullPointerException.class, RetryPolicies.TRY_ONCE_THEN_FAIL); + + this.retryPolicy = RetryPolicies.retryByException( + defaultPolicy, retryPolicyMap); + } + + @Override + public void run() { + int retries = 0; + RetryPolicy.RetryAction retryAction; + LOG.info(Thread.currentThread().getName() + "read ..."); + try { + this.readBuffer.lock(); + do { + try { + InputStream inputStream = this.store.retrieveBlock(this.key, + this.readBuffer.getStart(), this.readBuffer.getEnd()); + IOUtils.readFully(inputStream, this.readBuffer.getBuffer(), 0, + readBuffer.getBuffer().length); + inputStream.close(); + this.readBuffer.setStatus(CosNInputStream.ReadBuffer.SUCCESS); + break; + } catch (IOException e) { + this.readBuffer.setStatus(CosNInputStream.ReadBuffer.ERROR); + LOG.warn( + "Exception occurs when retrieve the block range start: " + + String.valueOf(this.readBuffer.getStart()) + " end: " + + this.readBuffer.getEnd()); + try { + retryAction = this.retryPolicy.shouldRetry( + e, retries++, 0, true); + if (retryAction.action + == RetryPolicy.RetryAction.RetryDecision.RETRY) { + Thread.sleep(retryAction.delayMillis); + } + } catch (Exception e1) { + String errMsg = String.format("Exception occurs when retry[%s] " + + "to retrieve the block range start: %s, end:%s", + this.retryPolicy.toString(), + String.valueOf(this.readBuffer.getStart()), + String.valueOf(this.readBuffer.getEnd())); + LOG.error(errMsg, e1); + break; + } + } + } while (retryAction.action == + RetryPolicy.RetryAction.RetryDecision.RETRY); + this.readBuffer.signalAll(); + } finally { + this.readBuffer.unLock(); + } + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNFileSystem.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNFileSystem.java new file mode 100644 index 0000000000000..333b34929ecda --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNFileSystem.java @@ -0,0 +1,814 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.net.URI; +import java.util.ArrayList; +import java.util.EnumSet; +import java.util.List; +import java.util.Map; +import java.util.HashMap; +import java.util.Set; +import java.util.TreeSet; +import java.util.concurrent.TimeUnit; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import com.google.common.util.concurrent.ListeningExecutorService; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.BufferedFSInputStream; +import org.apache.hadoop.fs.CreateFlag; +import org.apache.hadoop.fs.FSDataInputStream; +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.FileAlreadyExistsException; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.PathIOException; +import org.apache.hadoop.fs.permission.FsPermission; +import org.apache.hadoop.io.retry.RetryPolicies; +import org.apache.hadoop.io.retry.RetryPolicy; +import org.apache.hadoop.io.retry.RetryProxy; +import org.apache.hadoop.util.BlockingThreadPoolExecutorService; +import org.apache.hadoop.util.Progressable; + +/** + * The core CosN Filesystem implementation. + */ +@InterfaceAudience.Private +@InterfaceStability.Stable +public class CosNFileSystem extends FileSystem { + static final Logger LOG = LoggerFactory.getLogger(CosNFileSystem.class); + + public static final String SCHEME = "cosn"; + public static final String PATH_DELIMITER = Path.SEPARATOR; + + private URI uri; + private String bucket; + private NativeFileSystemStore store; + private Path workingDir; + private String owner = "Unknown"; + private String group = "Unknown"; + + private ListeningExecutorService boundedIOThreadPool; + private ListeningExecutorService boundedCopyThreadPool; + + public CosNFileSystem() { + } + + public CosNFileSystem(NativeFileSystemStore store) { + this.store = store; + } + + /** + * Return the protocol scheme for the FileSystem. + * + * @return cosn + */ + @Override + public String getScheme() { + return CosNFileSystem.SCHEME; + } + + @Override + public void initialize(URI name, Configuration conf) throws IOException { + super.initialize(name, conf); + this.bucket = name.getHost(); + if (this.store == null) { + this.store = createDefaultStore(conf); + } + this.store.initialize(name, conf); + setConf(conf); + this.uri = URI.create(name.getScheme() + "://" + name.getAuthority()); + this.workingDir = new Path("/user", + System.getProperty("user.name")).makeQualified( + this.uri, + this.getWorkingDirectory()); + this.owner = getOwnerId(); + this.group = getGroupId(); + LOG.debug("owner:" + owner + ", group:" + group); + + BufferPool.getInstance().initialize(this.getConf()); + + // initialize the thread pool + int uploadThreadPoolSize = this.getConf().getInt( + CosNConfigKeys.UPLOAD_THREAD_POOL_SIZE_KEY, + CosNConfigKeys.DEFAULT_UPLOAD_THREAD_POOL_SIZE + ); + int readAheadPoolSize = this.getConf().getInt( + CosNConfigKeys.READ_AHEAD_QUEUE_SIZE, + CosNConfigKeys.DEFAULT_READ_AHEAD_QUEUE_SIZE + ); + int ioThreadPoolSize = uploadThreadPoolSize + readAheadPoolSize / 3; + long threadKeepAlive = this.getConf().getLong( + CosNConfigKeys.THREAD_KEEP_ALIVE_TIME_KEY, + CosNConfigKeys.DEFAULT_THREAD_KEEP_ALIVE_TIME + ); + this.boundedIOThreadPool = BlockingThreadPoolExecutorService.newInstance( + ioThreadPoolSize / 2, ioThreadPoolSize, + threadKeepAlive, TimeUnit.SECONDS, + "cos-transfer-thread-pool"); + int copyThreadPoolSize = this.getConf().getInt( + CosNConfigKeys.COPY_THREAD_POOL_SIZE_KEY, + CosNConfigKeys.DEFAULT_COPY_THREAD_POOL_SIZE + ); + this.boundedCopyThreadPool = BlockingThreadPoolExecutorService.newInstance( + CosNConfigKeys.DEFAULT_COPY_THREAD_POOL_SIZE, copyThreadPoolSize, + 60L, TimeUnit.SECONDS, + "cos-copy-thread-pool"); + } + + private static NativeFileSystemStore createDefaultStore(Configuration conf) { + NativeFileSystemStore store = new CosNativeFileSystemStore(); + RetryPolicy basePolicy = RetryPolicies.retryUpToMaximumCountWithFixedSleep( + conf.getInt(CosNConfigKeys.COSN_MAX_RETRIES_KEY, + CosNConfigKeys.DEFAULT_MAX_RETRIES), + conf.getLong(CosNConfigKeys.COSN_RETRY_INTERVAL_KEY, + CosNConfigKeys.DEFAULT_RETRY_INTERVAL), + TimeUnit.SECONDS); + Map, RetryPolicy> exceptionToPolicyMap = + new HashMap<>(); + + exceptionToPolicyMap.put(IOException.class, basePolicy); + RetryPolicy methodPolicy = RetryPolicies.retryByException( + RetryPolicies.TRY_ONCE_THEN_FAIL, + exceptionToPolicyMap); + Map methodNameToPolicyMap = new HashMap<>(); + methodNameToPolicyMap.put("storeFile", methodPolicy); + methodNameToPolicyMap.put("rename", methodPolicy); + + return (NativeFileSystemStore) RetryProxy.create( + NativeFileSystemStore.class, store, methodNameToPolicyMap); + } + + private String getOwnerId() { + return System.getProperty("user.name"); + } + + private String getGroupId() { + return System.getProperty("user.name"); + } + + private String getOwnerInfo(boolean getOwnerId) { + String ownerInfoId = ""; + try { + String userName = System.getProperty("user.name"); + String command = "id -u " + userName; + if (!getOwnerId) { + command = "id -g " + userName; + } + Process child = Runtime.getRuntime().exec(command); + child.waitFor(); + + // Get the input stream and read from it + InputStream in = child.getInputStream(); + StringBuilder strBuffer = new StringBuilder(); + int c; + while ((c = in.read()) != -1) { + strBuffer.append((char) c); + } + in.close(); + ownerInfoId = strBuffer.toString(); + } catch (IOException | InterruptedException e) { + LOG.error("Getting owner info occurs a exception", e); + } + return ownerInfoId; + } + + private static String pathToKey(Path path) { + if (path.toUri().getScheme() != null && path.toUri().getPath().isEmpty()) { + // allow uris without trailing slash after bucket to refer to root, + // like cosn://mybucket + return ""; + } + if (!path.isAbsolute()) { + throw new IllegalArgumentException("Path must be absolute: " + path); + } + String ret = path.toUri().getPath(); + if (ret.endsWith("/") && (ret.indexOf("/") != ret.length() - 1)) { + ret = ret.substring(0, ret.length() - 1); + } + return ret; + } + + private static Path keyToPath(String key) { + if (!key.startsWith(PATH_DELIMITER)) { + return new Path("/" + key); + } else { + return new Path(key); + } + } + + private Path makeAbsolute(Path path) { + if (path.isAbsolute()) { + return path; + } + return new Path(workingDir, path); + } + + /** + * This optional operation is not yet supported. + */ + @Override + public FSDataOutputStream append(Path f, int bufferSize, + Progressable progress) throws IOException { + throw new IOException("Not supported"); + } + + @Override + public FSDataOutputStream create(Path f, FsPermission permission, + boolean overwrite, int bufferSize, short replication, long blockSize, + Progressable progress) throws IOException { + FileStatus fileStatus; + + try { + fileStatus = getFileStatus(f); + if (fileStatus.isDirectory()) { + throw new FileAlreadyExistsException(f + " is a directory"); + } + if (!overwrite) { + // path references a file and overwrite is disabled + throw new FileAlreadyExistsException(f + " already exists"); + } + + } catch (FileNotFoundException e) { + LOG.debug("Creating a new file: [{}] in COS.", f); + } + + Path absolutePath = makeAbsolute(f); + String key = pathToKey(absolutePath); + return new FSDataOutputStream( + new CosNOutputStream(getConf(), store, key, blockSize, + this.boundedIOThreadPool), statistics); + } + + private boolean rejectRootDirectoryDelete(boolean isEmptyDir, + boolean recursive) throws PathIOException { + if (isEmptyDir) { + return true; + } + if (recursive) { + return false; + } else { + throw new PathIOException(this.bucket, "Can not delete root path"); + } + } + + @Override + public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, + EnumSet flags, int bufferSize, short replication, + long blockSize, Progressable progress) throws IOException { + Path parent = f.getParent(); + if (null != parent) { + if (!getFileStatus(parent).isDirectory()) { + throw new FileAlreadyExistsException("Not a directory: " + parent); + } + } + + return create(f, permission, flags.contains(CreateFlag.OVERWRITE), + bufferSize, replication, blockSize, progress); + } + + @Override + public boolean delete(Path f, boolean recursive) throws IOException { + LOG.debug("Ready to delete path: [{}]. recursive: [{}].", f, recursive); + FileStatus status; + try { + status = getFileStatus(f); + } catch (FileNotFoundException e) { + LOG.debug("Ready to delete the file: [{}], but it does not exist.", f); + return false; + } + Path absolutePath = makeAbsolute(f); + String key = pathToKey(absolutePath); + if (key.compareToIgnoreCase("/") == 0) { + FileStatus[] fileStatuses = listStatus(f); + return this.rejectRootDirectoryDelete( + fileStatuses.length == 0, recursive); + } + + if (status.isDirectory()) { + if (!key.endsWith(PATH_DELIMITER)) { + key += PATH_DELIMITER; + } + if (!recursive && listStatus(f).length > 0) { + String errMsg = String.format("Can not delete the directory: [%s], as" + + " it is not empty and option recursive is false.", f); + throw new IOException(errMsg); + } + + createParent(f); + + String priorLastKey = null; + do { + PartialListing listing = store.list( + key, + Constants.COS_MAX_LISTING_LENGTH, + priorLastKey, + true); + for (FileMetadata file : listing.getFiles()) { + store.delete(file.getKey()); + } + for (FileMetadata commonPrefix : listing.getCommonPrefixes()) { + store.delete(commonPrefix.getKey()); + } + priorLastKey = listing.getPriorLastKey(); + } while (priorLastKey != null); + try { + store.delete(key); + } catch (Exception e) { + LOG.error("Deleting the COS key: [{}] occurs an exception.", key, e); + } + + } else { + LOG.debug("Delete the file: {}", f); + createParent(f); + store.delete(key); + } + return true; + } + + @Override + public FileStatus getFileStatus(Path f) throws IOException { + Path absolutePath = makeAbsolute(f); + String key = pathToKey(absolutePath); + + if (key.length() == 0) { + // root always exists + return newDirectory(absolutePath); + } + + LOG.debug("Call the getFileStatus to obtain the metadata for " + + "the file: [{}].", f); + + FileMetadata meta = store.retrieveMetadata(key); + if (meta != null) { + if (meta.isFile()) { + LOG.debug("Path: [{}] is a file. COS key: [{}]", f, key); + return newFile(meta, absolutePath); + } else { + LOG.debug("Path: [{}] is a dir. COS key: [{}]", f, key); + return newDirectory(meta, absolutePath); + } + } + + if (!key.endsWith(PATH_DELIMITER)) { + key += PATH_DELIMITER; + } + + // Considering that the object store's directory is a common prefix in + // the object key, it needs to check the existence of the path by listing + // the COS key. + LOG.debug("List COS key: [{}] to check the existence of the path.", key); + PartialListing listing = store.list(key, 1); + if (listing.getFiles().length > 0 + || listing.getCommonPrefixes().length > 0) { + if (LOG.isDebugEnabled()) { + LOG.debug("Path: [{}] is a directory. COS key: [{}]", f, key); + } + return newDirectory(absolutePath); + } + + throw new FileNotFoundException( + "No such file or directory '" + absolutePath + "'"); + } + + @Override + public URI getUri() { + return uri; + } + + /** + *

+ * If f is a file, this method will make a single call to COS. + * If f is a directory, + * this method will make a maximum of ( n / 199) + 2 calls to cos, + * where n is the total number of files + * and directories contained directly in f. + *

+ */ + @Override + public FileStatus[] listStatus(Path f) throws IOException { + Path absolutePath = makeAbsolute(f); + String key = pathToKey(absolutePath); + + if (key.length() > 0) { + FileStatus fileStatus = this.getFileStatus(f); + if (fileStatus.isFile()) { + return new FileStatus[]{fileStatus}; + } + } + + if (!key.endsWith(PATH_DELIMITER)) { + key += PATH_DELIMITER; + } + + URI pathUri = absolutePath.toUri(); + Set status = new TreeSet<>(); + String priorLastKey = null; + do { + PartialListing listing = store.list( + key, Constants.COS_MAX_LISTING_LENGTH, priorLastKey, false); + for (FileMetadata fileMetadata : listing.getFiles()) { + Path subPath = keyToPath(fileMetadata.getKey()); + if (fileMetadata.getKey().equals(key)) { + // this is just the directory we have been asked to list. + LOG.debug("The file list contains the COS key [{}] to be listed.", + key); + } else { + status.add(newFile(fileMetadata, subPath)); + } + } + + for (FileMetadata commonPrefix : listing.getCommonPrefixes()) { + Path subPath = keyToPath(commonPrefix.getKey()); + String relativePath = pathUri.relativize(subPath.toUri()).getPath(); + status.add( + newDirectory(commonPrefix, new Path(absolutePath, relativePath))); + } + priorLastKey = listing.getPriorLastKey(); + } while (priorLastKey != null); + + return status.toArray(new FileStatus[status.size()]); + } + + private FileStatus newFile(FileMetadata meta, Path path) { + return new FileStatus(meta.getLength(), false, 1, getDefaultBlockSize(), + meta.getLastModified(), 0, null, this.owner, this.group, + path.makeQualified(this.getUri(), this.getWorkingDirectory())); + } + + private FileStatus newDirectory(Path path) { + return new FileStatus(0, true, 1, 0, 0, 0, null, this.owner, this.group, + path.makeQualified(this.getUri(), this.getWorkingDirectory())); + } + + private FileStatus newDirectory(FileMetadata meta, Path path) { + if (meta == null) { + return newDirectory(path); + } + return new FileStatus(0, true, 1, 0, meta.getLastModified(), + 0, null, this.owner, this.group, + path.makeQualified(this.getUri(), this.getWorkingDirectory())); + } + + /** + * Validate the path from the bottom up. + * + * @param path The path to be validated + * @throws FileAlreadyExistsException The specified path is an existing file + * @throws IOException Getting the file status of the + * specified path occurs + * an IOException. + */ + private void validatePath(Path path) throws IOException { + Path parent = path.getParent(); + do { + try { + FileStatus fileStatus = getFileStatus(parent); + if (fileStatus.isDirectory()) { + break; + } else { + throw new FileAlreadyExistsException(String.format( + "Can't make directory for path '%s', it is a file.", parent)); + } + } catch (FileNotFoundException e) { + LOG.debug("The Path: [{}] does not exist.", path); + } + parent = parent.getParent(); + } while (parent != null); + } + + @Override + public boolean mkdirs(Path f, FsPermission permission) throws IOException { + try { + FileStatus fileStatus = getFileStatus(f); + if (fileStatus.isDirectory()) { + return true; + } else { + throw new FileAlreadyExistsException("Path is a file: " + f); + } + } catch (FileNotFoundException e) { + validatePath(f); + } + + return mkDirRecursively(f, permission); + } + + /** + * Recursively create a directory. + * + * @param f Absolute path to the directory. + * @param permission Directory permissions. Permission does not work for + * the CosN filesystem currently. + * @return Return true if the creation was successful, throw a IOException. + * @throws IOException The specified path already exists or an error + * creating the path. + */ + public boolean mkDirRecursively(Path f, FsPermission permission) + throws IOException { + Path absolutePath = makeAbsolute(f); + List paths = new ArrayList<>(); + do { + paths.add(absolutePath); + absolutePath = absolutePath.getParent(); + } while (absolutePath != null); + + for (Path path : paths) { + if (path.equals(new Path(CosNFileSystem.PATH_DELIMITER))) { + break; + } + try { + FileStatus fileStatus = getFileStatus(path); + if (fileStatus.isFile()) { + throw new FileAlreadyExistsException( + String.format("Can't make directory for path: %s, " + + "since it is a file.", f)); + } + if (fileStatus.isDirectory()) { + break; + } + } catch (FileNotFoundException e) { + LOG.debug("Making dir: [{}] in COS", f); + + String folderPath = pathToKey(makeAbsolute(f)); + if (!folderPath.endsWith(PATH_DELIMITER)) { + folderPath += PATH_DELIMITER; + } + store.storeEmptyFile(folderPath); + } + } + return true; + } + + private boolean mkdir(Path f) throws IOException { + try { + FileStatus fileStatus = getFileStatus(f); + if (fileStatus.isFile()) { + throw new FileAlreadyExistsException( + String.format( + "Can't make directory for path '%s' since it is a file.", f)); + } + } catch (FileNotFoundException e) { + if (LOG.isDebugEnabled()) { + LOG.debug("Make directory: [{}] in COS.", f); + } + + String folderPath = pathToKey(makeAbsolute(f)); + if (!folderPath.endsWith(PATH_DELIMITER)) { + folderPath += PATH_DELIMITER; + } + store.storeEmptyFile(folderPath); + } + return true; + } + + @Override + public FSDataInputStream open(Path f, int bufferSize) throws IOException { + FileStatus fs = getFileStatus(f); // will throw if the file doesn't + // exist + if (fs.isDirectory()) { + throw new FileNotFoundException("'" + f + "' is a directory"); + } + LOG.info("Open the file: [{}] for reading.", f); + Path absolutePath = makeAbsolute(f); + String key = pathToKey(absolutePath); + long fileSize = store.getFileLength(key); + return new FSDataInputStream(new BufferedFSInputStream( + new CosNInputStream(this.getConf(), store, statistics, key, fileSize, + this.boundedIOThreadPool), bufferSize)); + } + + @Override + public boolean rename(Path src, Path dst) throws IOException { + LOG.debug("Rename source path: [{}] to dest path: [{}].", src, dst); + + // Renaming the root directory is not allowed + if (src.isRoot()) { + LOG.debug("Cannot rename the root directory of a filesystem."); + return false; + } + + // check the source path whether exists or not + FileStatus srcFileStatus = this.getFileStatus(src); + + // Source path and destination path are not allowed to be the same + if (src.equals(dst)) { + LOG.debug("Source path and dest path refer to " + + "the same file or directory: [{}].", dst); + throw new IOException("Source path and dest path refer " + + "the same file or directory"); + } + + // It is not allowed to rename a parent directory to its subdirectory + Path dstParentPath; + for (dstParentPath = dst.getParent(); + null != dstParentPath && !src.equals(dstParentPath); + dstParentPath = dstParentPath.getParent()) { + // Recursively find the common parent path of the source and + // destination paths. + LOG.debug("Recursively find the common parent directory of the source " + + "and destination paths. The currently found parent path: {}", + dstParentPath); + } + + if (null != dstParentPath) { + LOG.debug("It is not allowed to rename a parent directory:[{}] " + + "to its subdirectory:[{}].", src, dst); + throw new IOException(String.format( + "It is not allowed to rename a parent directory: %s " + + "to its subdirectory: %s", src, dst)); + } + + FileStatus dstFileStatus; + try { + dstFileStatus = this.getFileStatus(dst); + + // The destination path exists and is a file, + // and the rename operation is not allowed. + if (dstFileStatus.isFile()) { + throw new FileAlreadyExistsException(String.format( + "File: %s already exists", dstFileStatus.getPath())); + } else { + // The destination path is an existing directory, + // and it is checked whether there is a file or directory + // with the same name as the source path under the destination path + dst = new Path(dst, src.getName()); + FileStatus[] statuses; + try { + statuses = this.listStatus(dst); + } catch (FileNotFoundException e) { + statuses = null; + } + if (null != statuses && statuses.length > 0) { + LOG.debug("Cannot rename source file: [{}] to dest file: [{}], " + + "because the file already exists.", src, dst); + throw new FileAlreadyExistsException( + String.format( + "File: %s already exists", dst + ) + ); + } + } + } catch (FileNotFoundException e) { + // destination path not exists + Path tempDstParentPath = dst.getParent(); + FileStatus dstParentStatus = this.getFileStatus(tempDstParentPath); + if (!dstParentStatus.isDirectory()) { + throw new IOException(String.format( + "Cannot rename %s to %s, %s is a file", src, dst, dst.getParent() + )); + } + // The default root directory is definitely there. + } + + boolean result; + if (srcFileStatus.isDirectory()) { + result = this.copyDirectory(src, dst); + } else { + result = this.copyFile(src, dst); + } + + if (!result) { + //Since rename is a non-atomic operation, after copy fails, + // it is not allowed to delete the data of the original path. + return false; + } else { + return this.delete(src, true); + } + } + + private boolean copyFile(Path srcPath, Path dstPath) throws IOException { + String srcKey = pathToKey(srcPath); + String dstKey = pathToKey(dstPath); + this.store.copy(srcKey, dstKey); + return true; + } + + private boolean copyDirectory(Path srcPath, Path dstPath) throws IOException { + String srcKey = pathToKey(srcPath); + if (!srcKey.endsWith(PATH_DELIMITER)) { + srcKey += PATH_DELIMITER; + } + String dstKey = pathToKey(dstPath); + if (!dstKey.endsWith(PATH_DELIMITER)) { + dstKey += PATH_DELIMITER; + } + + if (dstKey.startsWith(srcKey)) { + throw new IOException( + "can not copy a directory to a subdirectory of self"); + } + + this.store.storeEmptyFile(dstKey); + CosNCopyFileContext copyFileContext = new CosNCopyFileContext(); + + int copiesToFinishes = 0; + String priorLastKey = null; + do { + PartialListing objectList = this.store.list( + srcKey, Constants.COS_MAX_LISTING_LENGTH, priorLastKey, true); + for (FileMetadata file : objectList.getFiles()) { + this.boundedCopyThreadPool.execute(new CosNCopyFileTask( + this.store, + file.getKey(), + dstKey.concat(file.getKey().substring(srcKey.length())), + copyFileContext)); + copiesToFinishes++; + if (!copyFileContext.isCopySuccess()) { + break; + } + } + priorLastKey = objectList.getPriorLastKey(); + } while (null != priorLastKey); + + copyFileContext.lock(); + try { + copyFileContext.awaitAllFinish(copiesToFinishes); + } catch (InterruptedException e) { + LOG.warn("interrupted when wait copies to finish"); + } finally { + copyFileContext.lock(); + } + + return copyFileContext.isCopySuccess(); + } + + private void createParent(Path path) throws IOException { + Path parent = path.getParent(); + if (parent != null) { + String parentKey = pathToKey(parent); + LOG.debug("Create parent key: {}", parentKey); + if (!parentKey.equals(PATH_DELIMITER)) { + String key = pathToKey(makeAbsolute(parent)); + if (key.length() > 0) { + try { + store.storeEmptyFile(key + PATH_DELIMITER); + } catch (IOException e) { + LOG.debug("Store a empty file in COS failed.", e); + throw e; + } + } + } + } + } + + @Override + @SuppressWarnings("deprecation") + public long getDefaultBlockSize() { + return getConf().getLong( + CosNConfigKeys.COSN_BLOCK_SIZE_KEY, + CosNConfigKeys.DEFAULT_BLOCK_SIZE); + } + + /** + * Set the working directory to the given directory. + */ + @Override + public void setWorkingDirectory(Path newDir) { + workingDir = newDir; + } + + @Override + public Path getWorkingDirectory() { + return workingDir; + } + + @Override + public String getCanonicalServiceName() { + // Does not support Token + return null; + } + + @Override + public void close() throws IOException { + try { + this.store.close(); + this.boundedIOThreadPool.shutdown(); + this.boundedCopyThreadPool.shutdown(); + } finally { + super.close(); + } + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNInputStream.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNInputStream.java new file mode 100644 index 0000000000000..e759b55a559ad --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNInputStream.java @@ -0,0 +1,365 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.EOFException; +import java.io.IOException; +import java.util.ArrayDeque; +import java.util.Queue; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.locks.Condition; +import java.util.concurrent.locks.ReentrantLock; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FSExceptionMessages; +import org.apache.hadoop.fs.FSInputStream; +import org.apache.hadoop.fs.FileSystem; + +/** + * The input stream for the COS blob store. + * Optimized sequential read flow based on a forward read-ahead queue + */ +public class CosNInputStream extends FSInputStream { + private static final Logger LOG = + LoggerFactory.getLogger(CosNInputStream.class); + + /** + * This class is used by {@link CosNInputStream} + * and {@link CosNFileReadTask} to buffer data that read from COS blob store. + */ + public static class ReadBuffer { + public static final int INIT = 1; + public static final int SUCCESS = 0; + public static final int ERROR = -1; + + private final ReentrantLock lock = new ReentrantLock(); + private Condition readyCondition = lock.newCondition(); + + private byte[] buffer; + private int status; + private long start; + private long end; + + public ReadBuffer(long start, long end) { + this.start = start; + this.end = end; + this.buffer = new byte[(int) (this.end - this.start) + 1]; + this.status = INIT; + } + + public void lock() { + this.lock.lock(); + } + + public void unLock() { + this.lock.unlock(); + } + + public void await(int waitStatus) throws InterruptedException { + while (this.status == waitStatus) { + readyCondition.await(); + } + } + + public void signalAll() { + readyCondition.signalAll(); + } + + public byte[] getBuffer() { + return this.buffer; + } + + public int getStatus() { + return this.status; + } + + public void setStatus(int status) { + this.status = status; + } + + public long getStart() { + return start; + } + + public long getEnd() { + return end; + } + } + + private FileSystem.Statistics statistics; + private final Configuration conf; + private final NativeFileSystemStore store; + private final String key; + private long position = 0; + private long nextPos = 0; + private long fileSize; + private long partRemaining; + private final long preReadPartSize; + private final int maxReadPartNumber; + private byte[] buffer; + private boolean closed; + + private final ExecutorService readAheadExecutorService; + private final Queue readBufferQueue; + + public CosNInputStream(Configuration conf, NativeFileSystemStore store, + FileSystem.Statistics statistics, String key, long fileSize, + ExecutorService readAheadExecutorService) { + super(); + this.conf = conf; + this.store = store; + this.statistics = statistics; + this.key = key; + this.fileSize = fileSize; + this.preReadPartSize = conf.getLong( + CosNConfigKeys.READ_AHEAD_BLOCK_SIZE_KEY, + CosNConfigKeys.DEFAULT_READ_AHEAD_BLOCK_SIZE); + this.maxReadPartNumber = conf.getInt( + CosNConfigKeys.READ_AHEAD_QUEUE_SIZE, + CosNConfigKeys.DEFAULT_READ_AHEAD_QUEUE_SIZE); + + this.readAheadExecutorService = readAheadExecutorService; + this.readBufferQueue = new ArrayDeque<>(this.maxReadPartNumber); + this.closed = false; + } + + private synchronized void reopen(long pos) throws IOException { + long partSize; + + if (pos < 0) { + throw new EOFException(FSExceptionMessages.NEGATIVE_SEEK); + } else if (pos > this.fileSize) { + throw new EOFException(FSExceptionMessages.CANNOT_SEEK_PAST_EOF); + } else { + if (pos + this.preReadPartSize > this.fileSize) { + partSize = this.fileSize - pos; + } else { + partSize = this.preReadPartSize; + } + } + + this.buffer = null; + + boolean isRandomIO = true; + if (pos == this.nextPos) { + isRandomIO = false; + } else { + while (this.readBufferQueue.size() != 0) { + if (this.readBufferQueue.element().getStart() != pos) { + this.readBufferQueue.poll(); + } else { + break; + } + } + } + + this.nextPos = pos + partSize; + + int currentBufferQueueSize = this.readBufferQueue.size(); + long lastByteStart; + if (currentBufferQueueSize == 0) { + lastByteStart = pos - partSize; + } else { + ReadBuffer[] readBuffers = + this.readBufferQueue.toArray( + new ReadBuffer[currentBufferQueueSize]); + lastByteStart = readBuffers[currentBufferQueueSize - 1].getStart(); + } + + int maxLen = this.maxReadPartNumber - currentBufferQueueSize; + for (int i = 0; i < maxLen && i < (currentBufferQueueSize + 1) * 2; i++) { + if (lastByteStart + partSize * (i + 1) > this.fileSize) { + break; + } + + long byteStart = lastByteStart + partSize * (i + 1); + long byteEnd = byteStart + partSize - 1; + if (byteEnd >= this.fileSize) { + byteEnd = this.fileSize - 1; + } + + ReadBuffer readBuffer = new ReadBuffer(byteStart, byteEnd); + if (readBuffer.getBuffer().length == 0) { + readBuffer.setStatus(ReadBuffer.SUCCESS); + } else { + this.readAheadExecutorService.execute( + new CosNFileReadTask( + this.conf, this.key, this.store, readBuffer)); + } + + this.readBufferQueue.add(readBuffer); + if (isRandomIO) { + break; + } + } + + ReadBuffer readBuffer = this.readBufferQueue.poll(); + if (null != readBuffer) { + readBuffer.lock(); + try { + readBuffer.await(ReadBuffer.INIT); + if (readBuffer.getStatus() == ReadBuffer.ERROR) { + this.buffer = null; + } else { + this.buffer = readBuffer.getBuffer(); + } + } catch (InterruptedException e) { + LOG.warn("An interrupted exception occurred " + + "when waiting a read buffer."); + } finally { + readBuffer.unLock(); + } + } + + if (null == this.buffer) { + throw new IOException("Null IO stream"); + } + + this.position = pos; + this.partRemaining = partSize; + } + + @Override + public void seek(long pos) throws IOException { + if (pos < 0) { + throw new EOFException(FSExceptionMessages.NEGATIVE_SEEK); + } + if (pos > this.fileSize) { + throw new EOFException(FSExceptionMessages.CANNOT_SEEK_PAST_EOF); + } + + if (this.position == pos) { + return; + } + if (pos > position && pos < this.position + partRemaining) { + long len = pos - this.position; + this.position = pos; + this.partRemaining -= len; + } else { + this.reopen(pos); + } + } + + @Override + public long getPos() { + return this.position; + } + + @Override + public boolean seekToNewSource(long targetPos) { + // Currently does not support to seek the offset of a new source + return false; + } + + @Override + public int read() throws IOException { + if (this.closed) { + throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED); + } + + if (this.partRemaining <= 0 && this.position < this.fileSize) { + this.reopen(this.position); + } + + int byteRead = -1; + if (this.partRemaining != 0) { + byteRead = this.buffer[ + (int) (this.buffer.length - this.partRemaining)] & 0xff; + } + if (byteRead >= 0) { + this.position++; + this.partRemaining--; + if (null != this.statistics) { + this.statistics.incrementBytesRead(byteRead); + } + } + + return byteRead; + } + + @Override + public int read(byte[] b, int off, int len) throws IOException { + if (this.closed) { + throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED); + } + + if (len == 0) { + return 0; + } + + if (off < 0 || len < 0 || len > b.length) { + throw new IndexOutOfBoundsException(); + } + + int bytesRead = 0; + while (position < fileSize && bytesRead < len) { + if (partRemaining <= 0) { + reopen(position); + } + + int bytes = 0; + for (int i = this.buffer.length - (int) partRemaining; + i < this.buffer.length; i++) { + b[off + bytesRead] = this.buffer[i]; + bytes++; + bytesRead++; + if (off + bytesRead >= len) { + break; + } + } + + if (bytes > 0) { + this.position += bytes; + this.partRemaining -= bytes; + } else if (this.partRemaining != 0) { + throw new IOException( + "Failed to read from stream. Remaining: " + this.partRemaining); + } + } + if (null != this.statistics && bytesRead > 0) { + this.statistics.incrementBytesRead(bytesRead); + } + + return bytesRead == 0 ? -1 : bytesRead; + } + + @Override + public int available() throws IOException { + if (this.closed) { + throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED); + } + + long remaining = this.fileSize - this.position; + if (remaining > Integer.MAX_VALUE) { + return Integer.MAX_VALUE; + } + return (int)remaining; + } + + @Override + public void close() { + if (this.closed) { + return; + } + this.closed = true; + this.buffer = null; + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNOutputStream.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNOutputStream.java new file mode 100644 index 0000000000000..c437dde613d2c --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNOutputStream.java @@ -0,0 +1,284 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Set; +import java.util.concurrent.Callable; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.io.IOException; +import java.io.OutputStream; +import java.security.DigestOutputStream; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import com.google.common.util.concurrent.Futures; +import com.google.common.util.concurrent.ListenableFuture; +import com.google.common.util.concurrent.ListeningExecutorService; +import com.google.common.util.concurrent.MoreExecutors; +import com.qcloud.cos.model.PartETag; + +import org.apache.hadoop.conf.Configuration; + +/** + * The output stream for the COS blob store. + * Implement streaming upload to COS based on the multipart upload function. + * ( the maximum size of each part is 5GB) + * Support up to 40TB single file by multipart upload (each part is 5GB). + * Improve the upload performance of writing large files by using byte buffers + * and a fixed thread pool. + */ +public class CosNOutputStream extends OutputStream { + private static final Logger LOG = + LoggerFactory.getLogger(CosNOutputStream.class); + + private final Configuration conf; + private final NativeFileSystemStore store; + private MessageDigest digest; + private long blockSize; + private String key; + private int currentBlockId = 0; + private Set blockCacheBuffers = new HashSet<>(); + private ByteBufferWrapper currentBlockBuffer; + private OutputStream currentBlockOutputStream; + private String uploadId = null; + private ListeningExecutorService executorService; + private List> etagList = new LinkedList<>(); + private int blockWritten = 0; + private boolean closed = false; + + public CosNOutputStream(Configuration conf, NativeFileSystemStore store, + String key, long blockSize, ExecutorService executorService) + throws IOException { + this.conf = conf; + this.store = store; + this.key = key; + this.blockSize = blockSize; + if (this.blockSize < Constants.MIN_PART_SIZE) { + LOG.warn( + String.format( + "The minimum size of a single block is limited to %d.", + Constants.MIN_PART_SIZE)); + this.blockSize = Constants.MIN_PART_SIZE; + } + if (this.blockSize > Constants.MAX_PART_SIZE) { + LOG.warn( + String.format( + "The maximum size of a single block is limited to %d.", + Constants.MAX_PART_SIZE)); + this.blockSize = Constants.MAX_PART_SIZE; + } + + // Use a blocking thread pool with fair scheduling + this.executorService = MoreExecutors.listeningDecorator(executorService); + + try { + this.currentBlockBuffer = + BufferPool.getInstance().getBuffer((int) this.blockSize); + } catch (IOException e) { + throw new IOException("Getting a buffer size: " + + String.valueOf(this.blockSize) + + " from buffer pool occurs an exception: ", e); + } + + try { + this.digest = MessageDigest.getInstance("MD5"); + this.currentBlockOutputStream = new DigestOutputStream( + new ByteBufferOutputStream(this.currentBlockBuffer.getByteBuffer()), + this.digest); + } catch (NoSuchAlgorithmException e) { + this.digest = null; + this.currentBlockOutputStream = + new ByteBufferOutputStream(this.currentBlockBuffer.getByteBuffer()); + } + } + + @Override + public void flush() throws IOException { + this.currentBlockOutputStream.flush(); + } + + @Override + public synchronized void close() throws IOException { + if (this.closed) { + return; + } + this.currentBlockOutputStream.flush(); + this.currentBlockOutputStream.close(); + LOG.info("The output stream has been close, and " + + "begin to upload the last block: [{}].", this.currentBlockId); + this.blockCacheBuffers.add(this.currentBlockBuffer); + if (this.blockCacheBuffers.size() == 1) { + byte[] md5Hash = this.digest == null ? null : this.digest.digest(); + store.storeFile(this.key, + new ByteBufferInputStream(this.currentBlockBuffer.getByteBuffer()), + md5Hash, this.currentBlockBuffer.getByteBuffer().remaining()); + } else { + PartETag partETag = null; + if (this.blockWritten > 0) { + LOG.info("Upload the last part..., blockId: [{}], written bytes: [{}]", + this.currentBlockId, this.blockWritten); + partETag = store.uploadPart( + new ByteBufferInputStream(currentBlockBuffer.getByteBuffer()), + key, uploadId, currentBlockId + 1, + currentBlockBuffer.getByteBuffer().remaining()); + } + final List futurePartETagList = this.waitForFinishPartUploads(); + if (null == futurePartETagList) { + throw new IOException("Failed to multipart upload to cos, abort it."); + } + List tmpPartEtagList = new LinkedList<>(futurePartETagList); + if (null != partETag) { + tmpPartEtagList.add(partETag); + } + store.completeMultipartUpload(this.key, this.uploadId, tmpPartEtagList); + } + try { + BufferPool.getInstance().returnBuffer(this.currentBlockBuffer); + } catch (InterruptedException e) { + LOG.error("An exception occurred " + + "while returning the buffer to the buffer pool.", e); + } + LOG.info("The outputStream for key: [{}] has been uploaded.", key); + this.blockWritten = 0; + this.closed = true; + } + + private List waitForFinishPartUploads() throws IOException { + try { + LOG.info("Wait for all parts to finish their uploading."); + return Futures.allAsList(this.etagList).get(); + } catch (InterruptedException e) { + LOG.error("Interrupt the part upload.", e); + return null; + } catch (ExecutionException e) { + LOG.error("Cancelling futures."); + for (ListenableFuture future : this.etagList) { + future.cancel(true); + } + (store).abortMultipartUpload(this.key, this.uploadId); + LOG.error("Multipart upload with id: [{}] to COS key: [{}]", + this.uploadId, this.key, e); + throw new IOException("Multipart upload with id: " + + this.uploadId + " to " + this.key, e); + } + } + + private void uploadPart() throws IOException { + this.currentBlockOutputStream.flush(); + this.currentBlockOutputStream.close(); + this.blockCacheBuffers.add(this.currentBlockBuffer); + + if (this.currentBlockId == 0) { + uploadId = (store).getUploadId(key); + } + + ListenableFuture partETagListenableFuture = + this.executorService.submit( + new Callable() { + private final ByteBufferWrapper buf = currentBlockBuffer; + private final String localKey = key; + private final String localUploadId = uploadId; + private final int blockId = currentBlockId; + + @Override + public PartETag call() throws Exception { + if (LOG.isDebugEnabled()) { + LOG.debug("{} is uploading a part.", + Thread.currentThread().getName()); + } + PartETag partETag = (store).uploadPart( + new ByteBufferInputStream(this.buf.getByteBuffer()), + this.localKey, this.localUploadId, + this.blockId + 1, this.buf.getByteBuffer().remaining()); + BufferPool.getInstance().returnBuffer(this.buf); + return partETag; + } + }); + this.etagList.add(partETagListenableFuture); + try { + this.currentBlockBuffer = + BufferPool.getInstance().getBuffer((int) this.blockSize); + } catch (IOException e) { + String errMsg = String.format("Getting a buffer [size:%d] from " + + "the buffer pool failed.", this.blockSize); + throw new IOException(errMsg, e); + } + this.currentBlockId++; + if (null != this.digest) { + this.digest.reset(); + this.currentBlockOutputStream = new DigestOutputStream( + new ByteBufferOutputStream(this.currentBlockBuffer.getByteBuffer()), + this.digest); + } else { + this.currentBlockOutputStream = + new ByteBufferOutputStream(this.currentBlockBuffer.getByteBuffer()); + } + } + + @Override + public void write(byte[] b, int off, int len) throws IOException { + if (this.closed) { + throw new IOException("block stream has been closed."); + } + + while (len > 0) { + long writeBytes; + if (this.blockWritten + len > this.blockSize) { + writeBytes = this.blockSize - this.blockWritten; + } else { + writeBytes = len; + } + + this.currentBlockOutputStream.write(b, off, (int) writeBytes); + this.blockWritten += writeBytes; + if (this.blockWritten >= this.blockSize) { + this.uploadPart(); + this.blockWritten = 0; + } + len -= writeBytes; + off += writeBytes; + } + } + + @Override + public void write(byte[] b) throws IOException { + this.write(b, 0, b.length); + } + + @Override + public void write(int b) throws IOException { + if (this.closed) { + throw new IOException("block stream has been closed."); + } + + byte[] singleBytes = new byte[1]; + singleBytes[0] = (byte) b; + this.currentBlockOutputStream.write(singleBytes, 0, 1); + this.blockWritten += 1; + if (this.blockWritten >= this.blockSize) { + this.uploadPart(); + this.blockWritten = 0; + } + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNUtils.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNUtils.java new file mode 100644 index 0000000000000..39981caba24bb --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNUtils.java @@ -0,0 +1,167 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.IOException; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.lang.reflect.Modifier; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import com.qcloud.cos.auth.COSCredentialsProvider; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.cosn.auth.COSCredentialProviderList; +import org.apache.hadoop.fs.cosn.auth.EnvironmentVariableCredentialProvider; +import org.apache.hadoop.fs.cosn.auth.SimpleCredentialProvider; + +/** + * Utility methods for CosN code. + */ +public final class CosNUtils { + private static final Logger LOG = LoggerFactory.getLogger(CosNUtils.class); + + static final String INSTANTIATION_EXCEPTION + = "instantiation exception"; + static final String NOT_COS_CREDENTIAL_PROVIDER + = "is not cos credential provider"; + static final String ABSTRACT_CREDENTIAL_PROVIDER + = "is abstract and therefore cannot be created"; + + private CosNUtils() { + } + + public static COSCredentialProviderList createCosCredentialsProviderSet( + Configuration conf) throws IOException { + COSCredentialProviderList credentialProviderList = + new COSCredentialProviderList(); + + Class[] cosClasses = CosNUtils.loadCosProviderClasses( + conf, + CosNConfigKeys.COSN_CREDENTIALS_PROVIDER); + if (0 == cosClasses.length) { + credentialProviderList.add(new SimpleCredentialProvider(conf)); + credentialProviderList.add(new EnvironmentVariableCredentialProvider()); + } else { + for (Class credClass : cosClasses) { + credentialProviderList.add(createCOSCredentialProvider( + conf, + credClass)); + } + } + + return credentialProviderList; + } + + public static Class[] loadCosProviderClasses( + Configuration conf, + String key, + Class... defaultValue) throws IOException { + try { + return conf.getClasses(key, defaultValue); + } catch (RuntimeException e) { + Throwable c = e.getCause() != null ? e.getCause() : e; + throw new IOException("From option " + key + ' ' + c, c); + } + } + + public static COSCredentialsProvider createCOSCredentialProvider( + Configuration conf, + Class credClass) throws IOException { + COSCredentialsProvider credentialsProvider; + if (!COSCredentialsProvider.class.isAssignableFrom(credClass)) { + throw new IllegalArgumentException( + "class " + credClass + " " + NOT_COS_CREDENTIAL_PROVIDER); + } + if (Modifier.isAbstract(credClass.getModifiers())) { + throw new IllegalArgumentException( + "class " + credClass + " " + ABSTRACT_CREDENTIAL_PROVIDER); + } + LOG.debug("Credential Provider class: " + credClass.getName()); + + try { + // new credClass() + Constructor constructor = getConstructor(credClass); + if (constructor != null) { + credentialsProvider = + (COSCredentialsProvider) constructor.newInstance(); + return credentialsProvider; + } + // new credClass(conf) + constructor = getConstructor(credClass, Configuration.class); + if (null != constructor) { + credentialsProvider = + (COSCredentialsProvider) constructor.newInstance(conf); + return credentialsProvider; + } + + Method factory = getFactoryMethod( + credClass, COSCredentialsProvider.class, "getInstance"); + if (null != factory) { + credentialsProvider = (COSCredentialsProvider) factory.invoke(null); + return credentialsProvider; + } + + throw new IllegalArgumentException( + "Not supported constructor or factory method found" + ); + + } catch (IllegalAccessException e) { + throw new IOException( + credClass.getName() + " " + INSTANTIATION_EXCEPTION + ": " + e, e); + } catch (InstantiationException e) { + throw new IOException( + credClass.getName() + " " + INSTANTIATION_EXCEPTION + ": " + e, e); + } catch (InvocationTargetException e) { + Throwable targetException = e.getTargetException(); + if (targetException == null) { + targetException = e; + } + throw new IOException( + credClass.getName() + " " + INSTANTIATION_EXCEPTION + ": " + + targetException, targetException); + } + } + + private static Constructor getConstructor(Class cl, Class... args) { + try { + Constructor constructor = cl.getDeclaredConstructor(args); + return Modifier.isPublic(constructor.getModifiers()) ? constructor : null; + } catch (NoSuchMethodException e) { + return null; + } + } + + private static Method getFactoryMethod( + Class cl, Class returnType, String methodName) { + try { + Method m = cl.getDeclaredMethod(methodName); + if (Modifier.isPublic(m.getModifiers()) + && Modifier.isStatic(m.getModifiers()) + && returnType.isAssignableFrom(m.getReturnType())) { + return m; + } else { + return null; + } + } catch (NoSuchMethodException e) { + return null; + } + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNativeFileSystemStore.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNativeFileSystemStore.java new file mode 100644 index 0000000000000..833f42d7be6e7 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNativeFileSystemStore.java @@ -0,0 +1,768 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.BufferedInputStream; +import java.io.ByteArrayInputStream; +import java.io.File; +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.net.URI; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Comparator; +import java.util.List; +import java.util.concurrent.ThreadLocalRandom; + +import com.qcloud.cos.COSClient; +import com.qcloud.cos.ClientConfig; +import com.qcloud.cos.auth.BasicCOSCredentials; +import com.qcloud.cos.auth.COSCredentials; +import com.qcloud.cos.exception.CosClientException; +import com.qcloud.cos.exception.CosServiceException; +import com.qcloud.cos.http.HttpProtocol; +import com.qcloud.cos.model.AbortMultipartUploadRequest; +import com.qcloud.cos.model.COSObject; +import com.qcloud.cos.model.COSObjectSummary; +import com.qcloud.cos.model.CompleteMultipartUploadRequest; +import com.qcloud.cos.model.CompleteMultipartUploadResult; +import com.qcloud.cos.model.CopyObjectRequest; +import com.qcloud.cos.model.DeleteObjectRequest; +import com.qcloud.cos.model.GetObjectMetadataRequest; +import com.qcloud.cos.model.GetObjectRequest; +import com.qcloud.cos.model.InitiateMultipartUploadRequest; +import com.qcloud.cos.model.InitiateMultipartUploadResult; +import com.qcloud.cos.model.ListObjectsRequest; +import com.qcloud.cos.model.ObjectListing; +import com.qcloud.cos.model.ObjectMetadata; +import com.qcloud.cos.model.PartETag; +import com.qcloud.cos.model.PutObjectRequest; +import com.qcloud.cos.model.PutObjectResult; +import com.qcloud.cos.model.UploadPartRequest; +import com.qcloud.cos.model.UploadPartResult; +import com.qcloud.cos.region.Region; +import com.qcloud.cos.utils.Base64; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.cosn.auth.COSCredentialProviderList; +import org.apache.hadoop.util.VersionInfo; +import org.apache.http.HttpStatus; + +/** + * The class actually performs access operation to the COS blob store. + * It provides the bridging logic for the Hadoop's abstract filesystem and COS. + */ +@InterfaceAudience.Private +@InterfaceStability.Unstable +class CosNativeFileSystemStore implements NativeFileSystemStore { + private COSClient cosClient; + private String bucketName; + private int maxRetryTimes; + + public static final Logger LOG = + LoggerFactory.getLogger(CosNativeFileSystemStore.class); + + /** + * Initialize the client to access COS blob storage. + * + * @param conf Hadoop configuration with COS configuration options. + * @throws IOException Initialize the COS client failed, + * caused by incorrect options. + */ + private void initCOSClient(Configuration conf) throws IOException { + COSCredentialProviderList credentialProviderList = + CosNUtils.createCosCredentialsProviderSet(conf); + String region = conf.get(CosNConfigKeys.COSN_REGION_KEY); + String endpointSuffix = conf.get( + CosNConfigKeys.COSN_ENDPOINT_SUFFIX_KEY); + if (null == region && null == endpointSuffix) { + String exceptionMsg = String.format("config %s and %s at least one", + CosNConfigKeys.COSN_REGION_KEY, + CosNConfigKeys.COSN_ENDPOINT_SUFFIX_KEY); + throw new IOException(exceptionMsg); + } + + COSCredentials cosCred; + cosCred = new BasicCOSCredentials( + credentialProviderList.getCredentials().getCOSAccessKeyId(), + credentialProviderList.getCredentials().getCOSSecretKey()); + + boolean useHttps = conf.getBoolean(CosNConfigKeys.COSN_USE_HTTPS_KEY, + CosNConfigKeys.DEFAULT_USE_HTTPS); + + ClientConfig config; + if (null == region) { + config = new ClientConfig(new Region("")); + config.setEndPointSuffix(endpointSuffix); + } else { + config = new ClientConfig(new Region(region)); + } + if (useHttps) { + config.setHttpProtocol(HttpProtocol.https); + } + + config.setUserAgent(conf.get(CosNConfigKeys.USER_AGENT, + CosNConfigKeys.DEFAULT_USER_AGENT) + " For " + " Hadoop " + + VersionInfo.getVersion()); + + this.maxRetryTimes = conf.getInt(CosNConfigKeys.COSN_MAX_RETRIES_KEY, + CosNConfigKeys.DEFAULT_MAX_RETRIES); + + config.setMaxConnectionsCount( + conf.getInt(CosNConfigKeys.MAX_CONNECTION_NUM, + CosNConfigKeys.DEFAULT_MAX_CONNECTION_NUM)); + + this.cosClient = new COSClient(cosCred, config); + } + + /** + * Initialize the CosNativeFileSystemStore object, including + * its COS client and default COS bucket. + * + * @param uri The URI of the COS bucket accessed by default. + * @param conf Hadoop configuration with COS configuration options. + * @throws IOException Initialize the COS client failed. + */ + @Override + public void initialize(URI uri, Configuration conf) throws IOException { + try { + initCOSClient(conf); + this.bucketName = uri.getHost(); + } catch (Exception e) { + handleException(e, ""); + } + } + + /** + * Store a file into COS from the specified input stream, which would be + * retried until the success or maximum number. + * + * @param key COS object key. + * @param inputStream Input stream to be uploaded into COS. + * @param md5Hash MD5 value of the content to be uploaded. + * @param length Length of uploaded content. + * @throws IOException Upload the file failed. + */ + private void storeFileWithRetry(String key, InputStream inputStream, + byte[] md5Hash, long length) throws IOException { + try { + ObjectMetadata objectMetadata = new ObjectMetadata(); + objectMetadata.setContentMD5(Base64.encodeAsString(md5Hash)); + objectMetadata.setContentLength(length); + PutObjectRequest putObjectRequest = + new PutObjectRequest(bucketName, key, inputStream, objectMetadata); + + PutObjectResult putObjectResult = + (PutObjectResult) callCOSClientWithRetry(putObjectRequest); + LOG.debug("Store file successfully. COS key: [{}], ETag: [{}], " + + "MD5: [{}].", key, putObjectResult.getETag(), new String(md5Hash)); + } catch (Exception e) { + String errMsg = String.format("Store file failed. COS key: [%s], " + + "exception: [%s]", key, e.toString()); + LOG.error(errMsg); + handleException(new Exception(errMsg), key); + } + } + + /** + * Store a local file into COS. + * + * @param key COS object key. + * @param file The local file to be uploaded. + * @param md5Hash The MD5 value of the file to be uploaded. + * @throws IOException Upload the file failed. + */ + @Override + public void storeFile(String key, File file, byte[] md5Hash) + throws IOException { + LOG.info("Store file from local path: [{}]. file length: [{}] COS key: " + + "[{}] MD5: [{}].", file.getCanonicalPath(), file.length(), key, + new String(md5Hash)); + storeFileWithRetry(key, new BufferedInputStream(new FileInputStream(file)), + md5Hash, file.length()); + } + + /** + * Store a file into COS from the specified input stream. + * + * @param key COS object key. + * @param inputStream The Input stream to be uploaded. + * @param md5Hash The MD5 value of the content to be uploaded. + * @param contentLength Length of uploaded content. + * @throws IOException Upload the file failed. + */ + @Override + public void storeFile( + String key, + InputStream inputStream, + byte[] md5Hash, + long contentLength) throws IOException { + LOG.info("Store file from input stream. COS key: [{}], " + + "length: [{}], MD5: [{}].", key, contentLength, md5Hash); + storeFileWithRetry(key, inputStream, md5Hash, contentLength); + } + + // For cos, storeEmptyFile means creating a directory + @Override + public void storeEmptyFile(String key) throws IOException { + if (!key.endsWith(CosNFileSystem.PATH_DELIMITER)) { + key = key + CosNFileSystem.PATH_DELIMITER; + } + + ObjectMetadata objectMetadata = new ObjectMetadata(); + objectMetadata.setContentLength(0); + InputStream input = new ByteArrayInputStream(new byte[0]); + PutObjectRequest putObjectRequest = + new PutObjectRequest(bucketName, key, input, objectMetadata); + try { + PutObjectResult putObjectResult = + (PutObjectResult) callCOSClientWithRetry(putObjectRequest); + LOG.debug("Store empty file successfully. COS key: [{}], ETag: [{}].", + key, putObjectResult.getETag()); + } catch (Exception e) { + String errMsg = String.format("Store empty file failed. " + + "COS key: [%s], exception: [%s]", key, e.toString()); + LOG.error(errMsg); + handleException(new Exception(errMsg), key); + } + } + + public PartETag uploadPart(File file, String key, String uploadId, + int partNum) throws IOException { + InputStream inputStream = new FileInputStream(file); + return uploadPart(inputStream, key, uploadId, partNum, file.length()); + } + + @Override + public PartETag uploadPart(InputStream inputStream, String key, + String uploadId, int partNum, long partSize) throws IOException { + UploadPartRequest uploadPartRequest = new UploadPartRequest(); + uploadPartRequest.setBucketName(this.bucketName); + uploadPartRequest.setUploadId(uploadId); + uploadPartRequest.setInputStream(inputStream); + uploadPartRequest.setPartNumber(partNum); + uploadPartRequest.setPartSize(partSize); + uploadPartRequest.setKey(key); + + try { + UploadPartResult uploadPartResult = + (UploadPartResult) callCOSClientWithRetry(uploadPartRequest); + return uploadPartResult.getPartETag(); + } catch (Exception e) { + String errMsg = String.format("Current thread: [%d], COS key: [%s], " + + "upload id: [%s], part num: [%d], exception: [%s]", + Thread.currentThread().getId(), key, uploadId, partNum, e.toString()); + handleException(new Exception(errMsg), key); + } + + return null; + } + + public void abortMultipartUpload(String key, String uploadId) { + LOG.info("Abort the multipart upload. COS key: [{}], upload id: [{}].", + key, uploadId); + AbortMultipartUploadRequest abortMultipartUploadRequest = + new AbortMultipartUploadRequest(bucketName, key, uploadId); + cosClient.abortMultipartUpload(abortMultipartUploadRequest); + } + + /** + * Initialize a multipart upload and return the upload id. + * + * @param key The COS object key initialized to multipart upload. + * @return The multipart upload id. + */ + public String getUploadId(String key) { + if (null == key || key.length() == 0) { + return ""; + } + + LOG.info("Initiate a multipart upload. bucket: [{}], COS key: [{}].", + bucketName, key); + InitiateMultipartUploadRequest initiateMultipartUploadRequest = + new InitiateMultipartUploadRequest(bucketName, key); + InitiateMultipartUploadResult initiateMultipartUploadResult = + cosClient.initiateMultipartUpload(initiateMultipartUploadRequest); + return initiateMultipartUploadResult.getUploadId(); + } + + /** + * Finish a multipart upload process, which will merge all parts uploaded. + * + * @param key The COS object key to be finished. + * @param uploadId The upload id of the multipart upload to be finished. + * @param partETagList The etag list of the part that has been uploaded. + * @return The result object of completing the multipart upload process. + */ + public CompleteMultipartUploadResult completeMultipartUpload( + String key, String uploadId, List partETagList) { + Collections.sort(partETagList, new Comparator() { + @Override + public int compare(PartETag o1, PartETag o2) { + return o1.getPartNumber() - o2.getPartNumber(); + } + }); + LOG.info("Complete the multipart upload. bucket: [{}], COS key: [{}], " + + "upload id: [{}].", bucketName, key, uploadId); + CompleteMultipartUploadRequest completeMultipartUploadRequest = + new CompleteMultipartUploadRequest( + bucketName, key, uploadId, partETagList); + return cosClient.completeMultipartUpload(completeMultipartUploadRequest); + } + + private FileMetadata queryObjectMetadata(String key) throws IOException { + GetObjectMetadataRequest getObjectMetadataRequest = + new GetObjectMetadataRequest(bucketName, key); + try { + ObjectMetadata objectMetadata = + (ObjectMetadata) callCOSClientWithRetry(getObjectMetadataRequest); + long mtime = 0; + if (objectMetadata.getLastModified() != null) { + mtime = objectMetadata.getLastModified().getTime(); + } + long fileSize = objectMetadata.getContentLength(); + FileMetadata fileMetadata = new FileMetadata(key, fileSize, mtime, + !key.endsWith(CosNFileSystem.PATH_DELIMITER)); + LOG.debug("Retrieve file metadata. COS key: [{}], ETag: [{}], " + + "length: [{}].", key, objectMetadata.getETag(), + objectMetadata.getContentLength()); + return fileMetadata; + } catch (CosServiceException e) { + if (e.getStatusCode() != HttpStatus.SC_NOT_FOUND) { + String errorMsg = String.format("Retrieve file metadata file failed. " + + "COS key: [%s], CosServiceException: [%s].", key, e.toString()); + LOG.error(errorMsg); + handleException(new Exception(errorMsg), key); + } + } + return null; + } + + @Override + public FileMetadata retrieveMetadata(String key) throws IOException { + if (key.endsWith(CosNFileSystem.PATH_DELIMITER)) { + key = key.substring(0, key.length() - 1); + } + + if (!key.isEmpty()) { + FileMetadata fileMetadata = queryObjectMetadata(key); + if (fileMetadata != null) { + return fileMetadata; + } + } + + // If the key is a directory. + key = key + CosNFileSystem.PATH_DELIMITER; + return queryObjectMetadata(key); + } + + /** + * Download a COS object and return the input stream associated with it. + * + * @param key The object key that is being retrieved from the COS bucket + * @return This method returns null if the key is not found + * @throws IOException if failed to download. + */ + @Override + public InputStream retrieve(String key) throws IOException { + LOG.debug("Retrieve object key: [{}].", key); + GetObjectRequest getObjectRequest = + new GetObjectRequest(this.bucketName, key); + try { + COSObject cosObject = + (COSObject) callCOSClientWithRetry(getObjectRequest); + return cosObject.getObjectContent(); + } catch (Exception e) { + String errMsg = String.format("Retrieving key: [%s] occurs " + + "an exception: [%s].", key, e.toString()); + LOG.error("Retrieving COS key: [{}] occurs an exception: [{}].", key, e); + handleException(new Exception(errMsg), key); + } + // never will get here + return null; + } + + /** + * Retrieved a part of a COS object, which is specified the start position. + * + * @param key The object key that is being retrieved from + * the COS bucket. + * @param byteRangeStart The start position of the part to be retrieved in + * the object. + * @return The input stream associated with the retrieved object. + * @throws IOException if failed to retrieve. + */ + @Override + public InputStream retrieve(String key, long byteRangeStart) + throws IOException { + try { + LOG.debug("Retrieve COS key:[{}]. range start:[{}].", + key, byteRangeStart); + long fileSize = getFileLength(key); + long byteRangeEnd = fileSize - 1; + GetObjectRequest getObjectRequest = + new GetObjectRequest(this.bucketName, key); + if (byteRangeEnd >= byteRangeStart) { + getObjectRequest.setRange(byteRangeStart, fileSize - 1); + } + COSObject cosObject = + (COSObject) callCOSClientWithRetry(getObjectRequest); + return cosObject.getObjectContent(); + } catch (Exception e) { + String errMsg = + String.format("Retrieving COS key: [%s] occurs an exception. " + + "byte range start: [%s], exception: [%s].", + key, byteRangeStart, e.toString()); + LOG.error(errMsg); + handleException(new Exception(errMsg), key); + } + + // never will get here + return null; + } + + /** + * Download a part of a COS object, which is specified the start and + * end position. + * + * @param key The object key that is being downloaded + * @param byteRangeStart The start position of the part to be retrieved in + * the object. + * @param byteRangeEnd The end position of the part to be retrieved in + * the object. + * @return The input stream associated with the retrieved objects. + * @throws IOException If failed to retrieve. + */ + @Override + public InputStream retrieveBlock(String key, long byteRangeStart, + long byteRangeEnd) throws IOException { + try { + GetObjectRequest request = new GetObjectRequest(this.bucketName, key); + request.setRange(byteRangeStart, byteRangeEnd); + COSObject cosObject = (COSObject) this.callCOSClientWithRetry(request); + return cosObject.getObjectContent(); + } catch (CosServiceException e) { + String errMsg = + String.format("Retrieving key [%s] with byteRangeStart [%d] occurs " + + "an CosServiceException: [%s].", + key, byteRangeStart, e.toString()); + LOG.error(errMsg); + handleException(new Exception(errMsg), key); + return null; + } catch (CosClientException e) { + String errMsg = + String.format("Retrieving key [%s] with byteRangeStart [%d] " + + "occurs an exception: [%s].", + key, byteRangeStart, e.toString()); + LOG.error("Retrieving COS key: [{}] with byteRangeStart: [{}] " + + "occurs an exception: [{}].", key, byteRangeStart, e); + handleException(new Exception(errMsg), key); + } + + return null; + } + + @Override + public PartialListing list(String prefix, int maxListingLength) + throws IOException { + return list(prefix, maxListingLength, null, false); + } + + @Override + public PartialListing list(String prefix, int maxListingLength, + String priorLastKey, boolean recurse) throws IOException { + return list(prefix, recurse ? null : CosNFileSystem.PATH_DELIMITER, + maxListingLength, priorLastKey); + } + + /** + * List the metadata for all objects that + * the object key has the specified prefix. + * + * @param prefix The prefix to be listed. + * @param delimiter The delimiter is a sign, the same paths between + * are listed. + * @param maxListingLength The maximum number of listed entries. + * @param priorLastKey The last key in any previous search. + * @return A metadata list on the match. + * @throws IOException If list objects failed. + */ + private PartialListing list(String prefix, String delimiter, + int maxListingLength, String priorLastKey) throws IOException { + LOG.debug("List objects. prefix: [{}], delimiter: [{}], " + + "maxListLength: [{}], priorLastKey: [{}].", + prefix, delimiter, maxListingLength, priorLastKey); + + if (!prefix.startsWith(CosNFileSystem.PATH_DELIMITER)) { + prefix += CosNFileSystem.PATH_DELIMITER; + } + ListObjectsRequest listObjectsRequest = new ListObjectsRequest(); + listObjectsRequest.setBucketName(bucketName); + listObjectsRequest.setPrefix(prefix); + listObjectsRequest.setDelimiter(delimiter); + listObjectsRequest.setMarker(priorLastKey); + listObjectsRequest.setMaxKeys(maxListingLength); + ObjectListing objectListing = null; + try { + objectListing = + (ObjectListing) callCOSClientWithRetry(listObjectsRequest); + } catch (Exception e) { + String errMsg = String.format("prefix: [%s], delimiter: [%s], " + + "maxListingLength: [%d], priorLastKey: [%s]. " + + "List objects occur an exception: [%s].", prefix, + (delimiter == null) ? "" : delimiter, maxListingLength, priorLastKey, + e.toString()); + LOG.error(errMsg); + handleException(new Exception(errMsg), prefix); + } + ArrayList fileMetadataArray = new ArrayList<>(); + ArrayList commonPrefixArray = new ArrayList<>(); + + if (null == objectListing) { + String errMsg = String.format("List the prefix: [%s] failed. " + + "delimiter: [%s], max listing length:" + + " [%s], prior last key: [%s]", + prefix, delimiter, maxListingLength, priorLastKey); + handleException(new Exception(errMsg), prefix); + } + + List summaries = objectListing.getObjectSummaries(); + for (COSObjectSummary cosObjectSummary : summaries) { + String filePath = cosObjectSummary.getKey(); + if (!filePath.startsWith(CosNFileSystem.PATH_DELIMITER)) { + filePath = CosNFileSystem.PATH_DELIMITER + filePath; + } + if (filePath.equals(prefix)) { + continue; + } + long mtime = 0; + if (cosObjectSummary.getLastModified() != null) { + mtime = cosObjectSummary.getLastModified().getTime(); + } + long fileLen = cosObjectSummary.getSize(); + fileMetadataArray.add( + new FileMetadata(filePath, fileLen, mtime, true)); + } + List commonPrefixes = objectListing.getCommonPrefixes(); + for (String commonPrefix : commonPrefixes) { + if (!commonPrefix.startsWith(CosNFileSystem.PATH_DELIMITER)) { + commonPrefix = CosNFileSystem.PATH_DELIMITER + commonPrefix; + } + commonPrefixArray.add( + new FileMetadata(commonPrefix, 0, 0, false)); + } + + FileMetadata[] fileMetadata = new FileMetadata[fileMetadataArray.size()]; + for (int i = 0; i < fileMetadataArray.size(); ++i) { + fileMetadata[i] = fileMetadataArray.get(i); + } + FileMetadata[] commonPrefixMetaData = + new FileMetadata[commonPrefixArray.size()]; + for (int i = 0; i < commonPrefixArray.size(); ++i) { + commonPrefixMetaData[i] = commonPrefixArray.get(i); + } + // when truncated is false, it means that listing is finished. + if (!objectListing.isTruncated()) { + return new PartialListing( + null, fileMetadata, commonPrefixMetaData); + } else { + return new PartialListing( + objectListing.getNextMarker(), fileMetadata, commonPrefixMetaData); + } + } + + @Override + public void delete(String key) throws IOException { + LOG.debug("Delete object key: [{}] from bucket: {}.", key, this.bucketName); + try { + DeleteObjectRequest deleteObjectRequest = + new DeleteObjectRequest(bucketName, key); + callCOSClientWithRetry(deleteObjectRequest); + } catch (Exception e) { + String errMsg = + String.format("Delete key: [%s] occurs an exception: [%s].", + key, e.toString()); + LOG.error(errMsg); + handleException(new Exception(errMsg), key); + } + } + + public void rename(String srcKey, String dstKey) throws IOException { + LOG.debug("Rename source key: [{}] to dest key: [{}].", srcKey, dstKey); + try { + CopyObjectRequest copyObjectRequest = + new CopyObjectRequest(bucketName, srcKey, bucketName, dstKey); + callCOSClientWithRetry(copyObjectRequest); + DeleteObjectRequest deleteObjectRequest = + new DeleteObjectRequest(bucketName, srcKey); + callCOSClientWithRetry(deleteObjectRequest); + } catch (Exception e) { + String errMsg = String.format("Rename object unsuccessfully. " + + "source cos key: [%s], dest COS " + + "key: [%s], exception: [%s]", + srcKey, + dstKey, e.toString()); + LOG.error(errMsg); + handleException(new Exception(errMsg), srcKey); + } + } + + @Override + public void copy(String srcKey, String dstKey) throws IOException { + LOG.debug("Copy source key: [{}] to dest key: [{}].", srcKey, dstKey); + try { + CopyObjectRequest copyObjectRequest = + new CopyObjectRequest(bucketName, srcKey, bucketName, dstKey); + callCOSClientWithRetry(copyObjectRequest); + } catch (Exception e) { + String errMsg = String.format("Copy object unsuccessfully. " + + "source COS key: %s, dest COS key: " + + "%s, exception: %s", + srcKey, + dstKey, e.toString()); + LOG.error(errMsg); + handleException(new Exception(errMsg), srcKey); + } + } + + @Override + public void purge(String prefix) throws IOException { + throw new IOException("purge not supported"); + } + + @Override + public void dump() throws IOException { + throw new IOException("dump not supported"); + } + + // process Exception and print detail + private void handleException(Exception e, String key) throws IOException { + String cosPath = CosNFileSystem.SCHEME + "://" + bucketName + key; + String exceptInfo = String.format("%s : %s", cosPath, e.toString()); + throw new IOException(exceptInfo); + } + + @Override + public long getFileLength(String key) throws IOException { + LOG.debug("Get file length. COS key: {}", key); + GetObjectMetadataRequest getObjectMetadataRequest = + new GetObjectMetadataRequest(bucketName, key); + try { + ObjectMetadata objectMetadata = + (ObjectMetadata) callCOSClientWithRetry(getObjectMetadataRequest); + return objectMetadata.getContentLength(); + } catch (Exception e) { + String errMsg = String.format("Getting file length occurs an exception." + + "COS key: %s, exception: %s", key, + e.toString()); + LOG.error(errMsg); + handleException(new Exception(errMsg), key); + return 0; // never will get here + } + } + + private Object callCOSClientWithRetry(X request) + throws CosServiceException, IOException { + String sdkMethod = ""; + int retryIndex = 1; + while (true) { + try { + if (request instanceof PutObjectRequest) { + sdkMethod = "putObject"; + return this.cosClient.putObject((PutObjectRequest) request); + } else if (request instanceof UploadPartRequest) { + sdkMethod = "uploadPart"; + if (((UploadPartRequest) request).getInputStream() + instanceof ByteBufferInputStream) { + ((UploadPartRequest) request).getInputStream() + .mark((int) ((UploadPartRequest) request).getPartSize()); + } + return this.cosClient.uploadPart((UploadPartRequest) request); + } else if (request instanceof GetObjectMetadataRequest) { + sdkMethod = "queryObjectMeta"; + return this.cosClient.getObjectMetadata( + (GetObjectMetadataRequest) request); + } else if (request instanceof DeleteObjectRequest) { + sdkMethod = "deleteObject"; + this.cosClient.deleteObject((DeleteObjectRequest) request); + return new Object(); + } else if (request instanceof CopyObjectRequest) { + sdkMethod = "copyFile"; + return this.cosClient.copyObject((CopyObjectRequest) request); + } else if (request instanceof GetObjectRequest) { + sdkMethod = "getObject"; + return this.cosClient.getObject((GetObjectRequest) request); + } else if (request instanceof ListObjectsRequest) { + sdkMethod = "listObjects"; + return this.cosClient.listObjects((ListObjectsRequest) request); + } else { + throw new IOException("no such method"); + } + } catch (CosServiceException cse) { + String errMsg = String.format("Call cos sdk failed, " + + "retryIndex: [%d / %d], " + + "call method: %s, exception: %s", + retryIndex, this.maxRetryTimes, sdkMethod, cse.toString()); + int statusCode = cse.getStatusCode(); + // Retry all server errors + if (statusCode / 100 == 5) { + if (retryIndex <= this.maxRetryTimes) { + LOG.info(errMsg); + long sleepLeast = retryIndex * 300L; + long sleepBound = retryIndex * 500L; + try { + if (request instanceof UploadPartRequest) { + if (((UploadPartRequest) request).getInputStream() + instanceof ByteBufferInputStream) { + ((UploadPartRequest) request).getInputStream().reset(); + } + } + Thread.sleep( + ThreadLocalRandom.current().nextLong(sleepLeast, sleepBound)); + ++retryIndex; + } catch (InterruptedException e) { + throw new IOException(e.toString()); + } + } else { + LOG.error(errMsg); + throw new IOException(errMsg); + } + } else { + throw cse; + } + } catch (Exception e) { + String errMsg = String.format("Call cos sdk failed, " + + "call method: %s, exception: %s", sdkMethod, e.toString()); + LOG.error(errMsg); + throw new IOException(errMsg); + } + } + } + + @Override + public void close() { + if (null != this.cosClient) { + this.cosClient.shutdown(); + } + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/FileMetadata.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/FileMetadata.java new file mode 100644 index 0000000000000..c11c88765762c --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/FileMetadata.java @@ -0,0 +1,68 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; + +/** + *

+ * Holds basic metadata for a file stored in a {@link NativeFileSystemStore}. + *

+ */ +@InterfaceAudience.Private +@InterfaceStability.Unstable +class FileMetadata { + private final String key; + private final long length; + private final long lastModified; + private final boolean isFile; + + FileMetadata(String key, long length, long lastModified) { + this(key, length, lastModified, true); + } + + FileMetadata(String key, long length, long lastModified, boolean isFile) { + this.key = key; + this.length = length; + this.lastModified = lastModified; + this.isFile = isFile; + } + + public String getKey() { + return key; + } + + public long getLength() { + return length; + } + + public long getLastModified() { + return lastModified; + } + + @Override + public String toString() { + return "FileMetadata[" + key + ", " + length + ", " + lastModified + ", " + + "file?" + isFile + "]"; + } + + public boolean isFile() { + return isFile; + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/NativeFileSystemStore.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/NativeFileSystemStore.java new file mode 100644 index 0000000000000..536c6de6367cb --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/NativeFileSystemStore.java @@ -0,0 +1,99 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.net.URI; +import java.util.List; + +import com.qcloud.cos.model.CompleteMultipartUploadResult; +import com.qcloud.cos.model.PartETag; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.conf.Configuration; + +/** + *

+ * An abstraction for a key-based {@link File} store. + *

+ */ +@InterfaceAudience.Private +@InterfaceStability.Stable +interface NativeFileSystemStore { + + void initialize(URI uri, Configuration conf) throws IOException; + + void storeFile(String key, File file, byte[] md5Hash) throws IOException; + + void storeFile(String key, InputStream inputStream, byte[] md5Hash, + long contentLength) throws IOException; + + void storeEmptyFile(String key) throws IOException; + + CompleteMultipartUploadResult completeMultipartUpload( + String key, String uploadId, List partETagList); + + void abortMultipartUpload(String key, String uploadId); + + String getUploadId(String key); + + PartETag uploadPart(File file, String key, String uploadId, int partNum) + throws IOException; + + PartETag uploadPart(InputStream inputStream, String key, String uploadId, + int partNum, long partSize) throws IOException; + + FileMetadata retrieveMetadata(String key) throws IOException; + + InputStream retrieve(String key) throws IOException; + + InputStream retrieve(String key, long byteRangeStart) throws IOException; + + InputStream retrieveBlock(String key, long byteRangeStart, long byteRangeEnd) + throws IOException; + + long getFileLength(String key) throws IOException; + + PartialListing list(String prefix, int maxListingLength) throws IOException; + + PartialListing list(String prefix, int maxListingLength, + String priorLastKey, boolean recursive) throws IOException; + + void delete(String key) throws IOException; + + void copy(String srcKey, String dstKey) throws IOException; + + /** + * Delete all keys with the given prefix. Used for testing. + * + * @throws IOException if purge is not supported + */ + void purge(String prefix) throws IOException; + + /** + * Diagnostic method to dump state to the console. + * + * @throws IOException if dump is not supported + */ + void dump() throws IOException; + + void close(); +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/PartialListing.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/PartialListing.java new file mode 100644 index 0000000000000..78cba38a83d60 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/PartialListing.java @@ -0,0 +1,64 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; + +/** + *

+ * Holds information on a directory listing for a + * {@link NativeFileSystemStore}. + * This includes the {@link FileMetadata files} and directories + * (their names) contained in a directory. + *

+ *

+ * This listing may be returned in chunks, so a priorLastKey + * is provided so that the next chunk may be requested. + *

+ * + * @see NativeFileSystemStore#list(String, int, String, boolean) + */ +@InterfaceAudience.Private +@InterfaceStability.Unstable +class PartialListing { + + private final String priorLastKey; + private final FileMetadata[] files; + private final FileMetadata[] commonPrefixes; + + PartialListing(String priorLastKey, FileMetadata[] files, + FileMetadata[] commonPrefixes) { + this.priorLastKey = priorLastKey; + this.files = files; + this.commonPrefixes = commonPrefixes; + } + + public FileMetadata[] getFiles() { + return files; + } + + public FileMetadata[] getCommonPrefixes() { + return commonPrefixes; + } + + public String getPriorLastKey() { + return priorLastKey; + } + +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/Unit.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/Unit.java new file mode 100644 index 0000000000000..5950ba7301423 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/Unit.java @@ -0,0 +1,32 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +/** + * Constant definition of storage unit. + */ +public final class Unit { + private Unit() { + } + + public static final int KB = 1024; + public static final int MB = 1024 * KB; + public static final int GB = 1024 * MB; + public static final long TB = (long) 1024 * GB; + public static final long PB = (long) 1024 * TB; +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/COSCredentialProviderList.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/COSCredentialProviderList.java new file mode 100644 index 0000000000000..e900b997e4858 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/COSCredentialProviderList.java @@ -0,0 +1,139 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.auth; + +import java.io.Closeable; +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; + +import com.google.common.base.Preconditions; +import com.qcloud.cos.auth.AnonymousCOSCredentials; +import com.qcloud.cos.auth.COSCredentials; +import com.qcloud.cos.auth.COSCredentialsProvider; +import com.qcloud.cos.exception.CosClientException; +import com.qcloud.cos.utils.StringUtils; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * a list of cos credentials provider. + */ +public class COSCredentialProviderList implements + COSCredentialsProvider, AutoCloseable { + private static final Logger LOG = + LoggerFactory.getLogger(COSCredentialProviderList.class); + + private static final String NO_COS_CREDENTIAL_PROVIDERS = + "No COS Credential Providers"; + private static final String CREDENTIALS_REQUESTED_WHEN_CLOSED = + "Credentials requested after provider list was closed"; + + private final List providers = + new ArrayList<>(1); + private boolean reuseLastProvider = true; + private COSCredentialsProvider lastProvider; + + private final AtomicInteger refCount = new AtomicInteger(1); + private final AtomicBoolean isClosed = new AtomicBoolean(false); + + public COSCredentialProviderList() { + } + + public COSCredentialProviderList( + Collection providers) { + this.providers.addAll(providers); + } + + public void add(COSCredentialsProvider provider) { + this.providers.add(provider); + } + + public int getRefCount() { + return this.refCount.get(); + } + + public void checkNotEmpty() { + if (this.providers.isEmpty()) { + throw new NoAuthWithCOSException(NO_COS_CREDENTIAL_PROVIDERS); + } + } + + public COSCredentialProviderList share() { + Preconditions.checkState(!this.closed(), "Provider list is closed"); + this.refCount.incrementAndGet(); + return this; + } + + public boolean closed() { + return this.isClosed.get(); + } + + @Override + public COSCredentials getCredentials() { + if (this.closed()) { + throw new NoAuthWithCOSException(CREDENTIALS_REQUESTED_WHEN_CLOSED); + } + + this.checkNotEmpty(); + + if (this.reuseLastProvider && this.lastProvider != null) { + return this.lastProvider.getCredentials(); + } + + for (COSCredentialsProvider provider : this.providers) { + try { + COSCredentials credentials = provider.getCredentials(); + if (!StringUtils.isNullOrEmpty(credentials.getCOSAccessKeyId()) + && !StringUtils.isNullOrEmpty(credentials.getCOSSecretKey()) + || credentials instanceof AnonymousCOSCredentials) { + this.lastProvider = provider; + return credentials; + } + } catch (CosClientException e) { + LOG.warn("No credentials provided by {}: {}", provider, e.toString()); + } + } + + throw new NoAuthWithCOSException( + "No COS Credentials provided by " + this.providers.toString()); + } + + @Override + public void close() throws Exception { + if (this.closed()) { + return; + } + + int remainder = this.refCount.decrementAndGet(); + if (remainder != 0) { + return; + } + this.isClosed.set(true); + + for (COSCredentialsProvider provider : this.providers) { + if (provider instanceof Closeable) { + ((Closeable) provider).close(); + } + } + } + +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/EnvironmentVariableCredentialProvider.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/EnvironmentVariableCredentialProvider.java new file mode 100644 index 0000000000000..0a7786b882f8b --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/EnvironmentVariableCredentialProvider.java @@ -0,0 +1,55 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.auth; + +import com.qcloud.cos.auth.BasicCOSCredentials; +import com.qcloud.cos.auth.COSCredentials; +import com.qcloud.cos.auth.COSCredentialsProvider; +import com.qcloud.cos.exception.CosClientException; +import com.qcloud.cos.utils.StringUtils; + +import org.apache.hadoop.fs.cosn.Constants; + +/** + * the provider obtaining the cos credentials from the environment variables. + */ +public class EnvironmentVariableCredentialProvider + implements COSCredentialsProvider { + @Override + public COSCredentials getCredentials() { + String secretId = System.getenv(Constants.COSN_SECRET_ID_ENV); + String secretKey = System.getenv(Constants.COSN_SECRET_KEY_ENV); + + secretId = StringUtils.trim(secretId); + secretKey = StringUtils.trim(secretKey); + + if (!StringUtils.isNullOrEmpty(secretId) + && !StringUtils.isNullOrEmpty(secretKey)) { + return new BasicCOSCredentials(secretId, secretKey); + } else { + throw new CosClientException( + "Unable to load COS credentials from environment variables" + + "(COS_SECRET_ID or COS_SECRET_KEY)"); + } + } + + @Override + public String toString() { + return "EnvironmentVariableCredentialProvider{}"; + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/NoAuthWithCOSException.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/NoAuthWithCOSException.java new file mode 100644 index 0000000000000..fa188bfb3ffea --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/NoAuthWithCOSException.java @@ -0,0 +1,37 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.auth; + +import com.qcloud.cos.exception.CosClientException; + +/** + * Exception thrown when no credentials can be obtained. + */ +public class NoAuthWithCOSException extends CosClientException { + public NoAuthWithCOSException(String message, Throwable t) { + super(message, t); + } + + public NoAuthWithCOSException(String message) { + super(message); + } + + public NoAuthWithCOSException(Throwable t) { + super(t); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/SimpleCredentialProvider.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/SimpleCredentialProvider.java new file mode 100644 index 0000000000000..f0635fc0d00cf --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/SimpleCredentialProvider.java @@ -0,0 +1,54 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.auth; + +import com.qcloud.cos.auth.BasicCOSCredentials; +import com.qcloud.cos.auth.COSCredentials; +import com.qcloud.cos.auth.COSCredentialsProvider; +import com.qcloud.cos.exception.CosClientException; + +import org.apache.commons.lang3.StringUtils; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.cosn.CosNConfigKeys; + +/** + * Get the credentials from the hadoop configuration. + */ +public class SimpleCredentialProvider implements COSCredentialsProvider { + private String secretId; + private String secretKey; + + public SimpleCredentialProvider(Configuration conf) { + this.secretId = conf.get( + CosNConfigKeys.COSN_SECRET_ID_KEY + ); + this.secretKey = conf.get( + CosNConfigKeys.COSN_SECRET_KEY_KEY + ); + } + + @Override + public COSCredentials getCredentials() { + if (!StringUtils.isEmpty(this.secretId) + && !StringUtils.isEmpty(this.secretKey)) { + return new BasicCOSCredentials(this.secretId, this.secretKey); + } + throw new CosClientException("secret id or secret key is unset"); + } + +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/package-info.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/package-info.java new file mode 100644 index 0000000000000..4b6f8cf48b9b2 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/package-info.java @@ -0,0 +1,18 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.auth; diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/package-info.java b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/package-info.java new file mode 100644 index 0000000000000..b46608239ab5f --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/package-info.java @@ -0,0 +1,18 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/CosNTestConfigKey.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/CosNTestConfigKey.java new file mode 100644 index 0000000000000..4d5ee4814c3b9 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/CosNTestConfigKey.java @@ -0,0 +1,33 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +/** + * Configuration options for the CosN file system for testing. + */ +public final class CosNTestConfigKey { + private CosNTestConfigKey() { + } + + public static final String TEST_COS_FILESYSTEM_CONF_KEY = + "test.fs.cosn.name"; + public static final String DEFAULT_TEST_COS_FILESYSTEM_CONF_VALUE = + ""; + public static final String TEST_UNIQUE_FORK_ID_KEY = + "test.unique.fork.id"; +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/CosNTestUtils.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/CosNTestUtils.java new file mode 100644 index 0000000000000..8afce51b1717c --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/CosNTestUtils.java @@ -0,0 +1,78 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import java.io.IOException; +import java.net.URI; + +import org.apache.commons.lang3.StringUtils; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.junit.internal.AssumptionViolatedException; + +/** + * Utilities for the CosN tests. + */ +public final class CosNTestUtils { + + private CosNTestUtils() { + } + + /** + * Create the file system for test. + * + * @param configuration hadoop's configuration + * @return The file system for test + * @throws IOException If fail to create or initialize the file system. + */ + public static CosNFileSystem createTestFileSystem( + Configuration configuration) throws IOException { + String fsName = configuration.getTrimmed( + CosNTestConfigKey.TEST_COS_FILESYSTEM_CONF_KEY, + CosNTestConfigKey.DEFAULT_TEST_COS_FILESYSTEM_CONF_VALUE); + + boolean liveTest = StringUtils.isNotEmpty(fsName); + URI testUri; + if (liveTest) { + testUri = URI.create(fsName); + liveTest = testUri.getScheme().equals(CosNFileSystem.SCHEME); + } else { + throw new AssumptionViolatedException("no test file system in " + + fsName); + } + + CosNFileSystem cosFs = new CosNFileSystem(); + cosFs.initialize(testUri, configuration); + return cosFs; + } + + /** + * Create a dir path for test. + * The value of {@link CosNTestConfigKey#TEST_UNIQUE_FORK_ID_KEY} + * will be used if it is set. + * + * @param defVal default value + * @return The test path + */ + public static Path createTestPath(Path defVal) { + String testUniqueForkId = System.getProperty( + CosNTestConfigKey.TEST_UNIQUE_FORK_ID_KEY); + return testUniqueForkId == + null ? defVal : new Path("/" + testUniqueForkId, "test"); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/TestCosNInputStream.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/TestCosNInputStream.java new file mode 100644 index 0000000000000..79884bad070c2 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/TestCosNInputStream.java @@ -0,0 +1,167 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FSDataInputStream; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.contract.ContractTestUtils; +import org.apache.hadoop.io.IOUtils; +import org.junit.*; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.Random; + +/** + * CosNInputStream Tester. + */ +public class TestCosNInputStream { + private static final Logger LOG = + LoggerFactory.getLogger(TestCosNInputStream.class); + + private FileSystem fs; + + private Path testRootDir; + + @Before + public void setUp() throws IOException { + Configuration configuration = new Configuration(); + this.fs = CosNTestUtils.createTestFileSystem(configuration); + this.testRootDir = CosNTestUtils.createTestPath(new Path("/test")); + LOG.info("test root dir: " + this.testRootDir); + } + + @After + public void tearDown() throws IOException { + if (null != this.fs) { + this.fs.delete(this.testRootDir, true); + } + } + + /** + * Method: seek(long pos). + */ + @Test + public void testSeek() throws Exception { + Path seekTestFilePath = new Path(this.testRootDir + "/" + + "seekTestFile"); + long fileSize = 5 * Unit.MB; + + ContractTestUtils.generateTestFile( + this.fs, seekTestFilePath, fileSize, 256, 255); + LOG.info("5MB file for seek test has created."); + + FSDataInputStream inputStream = this.fs.open(seekTestFilePath); + int seekTimes = 5; + for (int i = 0; i != seekTimes; i++) { + long pos = fileSize / (seekTimes - i) - 1; + inputStream.seek(pos); + assertTrue("expected position at: " + + pos + ", but got: " + inputStream.getPos(), + inputStream.getPos() == pos); + LOG.info("completed seeking at pos: " + inputStream.getPos()); + } + LOG.info("begin to random position seeking test..."); + Random random = new Random(); + for (int i = 0; i < seekTimes; i++) { + long pos = Math.abs(random.nextLong()) % fileSize; + LOG.info("seeking for pos: " + pos); + inputStream.seek(pos); + assertTrue("expected position at: " + + pos + ", but got: " + inputStream.getPos(), + inputStream.getPos() == pos); + LOG.info("completed seeking at pos: " + inputStream.getPos()); + } + } + + /** + * Method: getPos(). + */ + @Test + public void testGetPos() throws Exception { + Path seekTestFilePath = new Path(this.testRootDir + "/" + + "seekTestFile"); + long fileSize = 5 * Unit.MB; + ContractTestUtils.generateTestFile( + this.fs, seekTestFilePath, fileSize, 256, 255); + LOG.info("5MB file for getPos test has created."); + + FSDataInputStream inputStream = this.fs.open(seekTestFilePath); + Random random = new Random(); + long pos = Math.abs(random.nextLong()) % fileSize; + inputStream.seek(pos); + assertTrue("expected position at: " + + pos + ", but got: " + inputStream.getPos(), + inputStream.getPos() == pos); + LOG.info("completed get pos tests."); + } + + /** + * Method: seekToNewSource(long targetPos). + */ + @Ignore("Not ready yet") + public void testSeekToNewSource() throws Exception { + LOG.info("Currently it is not supported to " + + "seek the offset in a new source."); + } + + /** + * Method: read(). + */ + @Test + public void testRead() throws Exception { + final int bufLen = 256; + Path readTestFilePath = new Path(this.testRootDir + "/" + + "testReadSmallFile.txt"); + long fileSize = 5 * Unit.MB; + + ContractTestUtils.generateTestFile( + this.fs, readTestFilePath, fileSize, 256, 255); + LOG.info("read test file: " + readTestFilePath + " has created."); + + FSDataInputStream inputStream = this.fs.open(readTestFilePath); + byte[] buf = new byte[bufLen]; + long bytesRead = 0; + while (bytesRead < fileSize) { + int bytes = 0; + if (fileSize - bytesRead < bufLen) { + int remaining = (int) (fileSize - bytesRead); + bytes = inputStream.read(buf, 0, remaining); + } else { + bytes = inputStream.read(buf, 0, bufLen); + } + bytesRead += bytes; + + if (bytesRead % (1 * Unit.MB) == 0) { + int available = inputStream.available(); + assertTrue("expected remaining: " + (fileSize - bytesRead) + + " but got: " + available, (fileSize - bytesRead) == available); + LOG.info("Bytes read: " + + Math.round((double) bytesRead / Unit.MB) + "MB"); + } + } + + assertTrue(inputStream.available() == 0); + IOUtils.closeStream(inputStream); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/TestCosNOutputStream.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/TestCosNOutputStream.java new file mode 100644 index 0000000000000..7fd88976d62ae --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/TestCosNOutputStream.java @@ -0,0 +1,87 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.contract.ContractTestUtils; +import org.junit.Rule; +import org.junit.Test; +import org.junit.Before; +import org.junit.After; +import org.junit.rules.Timeout; + +import java.io.IOException; + +/** + * CosNOutputStream Tester. + *

+ * If the test.fs.cosn.name property is not set, all test case will fail. + */ +public class TestCosNOutputStream { + private FileSystem fs; + private Path testRootDir; + + @Rule + public Timeout timeout = new Timeout(3600 * 1000); + + @Before + public void setUp() throws Exception { + Configuration configuration = new Configuration(); + configuration.setInt( + CosNConfigKeys.COSN_BLOCK_SIZE_KEY, 2 * Unit.MB); + configuration.setLong( + CosNConfigKeys.COSN_UPLOAD_BUFFER_SIZE_KEY, + CosNConfigKeys.DEFAULT_UPLOAD_BUFFER_SIZE); + this.fs = CosNTestUtils.createTestFileSystem(configuration); + this.testRootDir = new Path("/test"); + } + + @After + public void tearDown() throws Exception { + } + + @Test + public void testEmptyFileUpload() throws IOException { + ContractTestUtils.createAndVerifyFile(this.fs, this.testRootDir, 0); + } + + @Test + public void testSingleFileUpload() throws IOException { + ContractTestUtils.createAndVerifyFile( + this.fs, this.testRootDir, 1 * Unit.MB - 1); + ContractTestUtils.createAndVerifyFile( + this.fs, this.testRootDir, 1 * Unit.MB); + ContractTestUtils.createAndVerifyFile( + this.fs, this.testRootDir, 2 * Unit.MB - 1); + } + + @Test + public void testLargeFileUpload() throws IOException { + ContractTestUtils.createAndVerifyFile( + this.fs, this.testRootDir, 2 * Unit.MB); + ContractTestUtils.createAndVerifyFile( + this.fs, this.testRootDir, 2 * Unit.MB + 1); + ContractTestUtils.createAndVerifyFile( + this.fs, this.testRootDir, 100 * Unit.MB); + // In principle, a maximum boundary test (file size: 2MB * 10000 - 1) + // should be provided here, + // but it is skipped due to network bandwidth and test time constraints. + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/CosNContract.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/CosNContract.java new file mode 100644 index 0000000000000..cd4097951d33b --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/CosNContract.java @@ -0,0 +1,46 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.contract; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.contract.AbstractBondedFSContract; +import org.apache.hadoop.fs.cosn.CosNFileSystem; +import org.apache.hadoop.fs.cosn.CosNTestUtils; + +/** + * The contract of CosN: only enabled if the test bucket is provided. + */ +public class CosNContract extends AbstractBondedFSContract { + private static final String CONTRACT_XML = "contract/cosn.xml"; + + protected CosNContract(Configuration conf) { + super(conf); + addConfResource(CONTRACT_XML); + } + + @Override + public String getScheme() { + return CosNFileSystem.SCHEME; + } + + @Override + public Path getTestPath() { + return CosNTestUtils.createTestPath(super.getTestPath()); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractCreate.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractCreate.java new file mode 100644 index 0000000000000..9488bd469af99 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractCreate.java @@ -0,0 +1,32 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.contract; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.contract.AbstractContractCreateTest; +import org.apache.hadoop.fs.contract.AbstractFSContract; + +/** + * CosN contract tests for creating files. + */ +public class TestCosNContractCreate extends AbstractContractCreateTest { + @Override + protected AbstractFSContract createContract(Configuration configuration) { + return new CosNContract(configuration); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractDelete.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractDelete.java new file mode 100644 index 0000000000000..1c23ac218453f --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractDelete.java @@ -0,0 +1,32 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.contract; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.contract.AbstractContractDeleteTest; +import org.apache.hadoop.fs.contract.AbstractFSContract; + +/** + * CosN contract tests for deleting files. + */ +public class TestCosNContractDelete extends AbstractContractDeleteTest { + @Override + protected AbstractFSContract createContract(Configuration configuration) { + return new CosNContract(configuration); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractDistCp.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractDistCp.java new file mode 100644 index 0000000000000..75ac53b9de8fb --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractDistCp.java @@ -0,0 +1,54 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.contract; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.contract.AbstractFSContract; +import org.apache.hadoop.fs.cosn.CosNConfigKeys; +import org.apache.hadoop.fs.cosn.Unit; +import org.apache.hadoop.tools.contract.AbstractContractDistCpTest; + +/** + * Contract test suit covering CosN integration with DistCp. + */ +public class TestCosNContractDistCp extends AbstractContractDistCpTest { + + private static final int MULTIPART_SETTING = 2 * Unit.MB; + private static final long UPLOAD_BUFFER_POOL_SIZE = 5 * 2 * Unit.MB; + private static final int UPLOAD_THREAD_POOL_SIZE = 5; + private static final int COPY_THREAD_POOL_SIZE = 3; + + @Override + protected AbstractFSContract createContract(Configuration conf) { + return new CosNContract(conf); + } + + @Override + protected Configuration createConfiguration() { + Configuration newConf = super.createConfiguration(); + newConf.setInt(CosNConfigKeys.COSN_BLOCK_SIZE_KEY, + MULTIPART_SETTING); + newConf.setLong(CosNConfigKeys.COSN_UPLOAD_BUFFER_SIZE_KEY, + UPLOAD_BUFFER_POOL_SIZE); + newConf.setInt(CosNConfigKeys.UPLOAD_THREAD_POOL_SIZE_KEY, + UPLOAD_THREAD_POOL_SIZE); + newConf.setInt(CosNConfigKeys.COPY_THREAD_POOL_SIZE_KEY, + COPY_THREAD_POOL_SIZE); + return newConf; + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractGetFileStatus.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractGetFileStatus.java new file mode 100644 index 0000000000000..9fba6eef44f36 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractGetFileStatus.java @@ -0,0 +1,33 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.contract; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest; +import org.apache.hadoop.fs.contract.AbstractFSContract; + +/** + * CosN contract tests covering getFileStatus. + */ +public class TestCosNContractGetFileStatus + extends AbstractContractGetFileStatusTest { + @Override + protected AbstractFSContract createContract(Configuration conf) { + return new CosNContract(conf); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractMkdir.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractMkdir.java new file mode 100644 index 0000000000000..e704e13df856a --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractMkdir.java @@ -0,0 +1,32 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.contract; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.contract.AbstractContractMkdirTest; +import org.apache.hadoop.fs.contract.AbstractFSContract; + +/** + * CosN contract tests for making directories. + */ +public class TestCosNContractMkdir extends AbstractContractMkdirTest { + @Override + protected AbstractFSContract createContract(Configuration configuration) { + return new CosNContract(configuration); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractOpen.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractOpen.java new file mode 100644 index 0000000000000..1bb732b312bf6 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractOpen.java @@ -0,0 +1,32 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.contract; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.contract.AbstractContractOpenTest; +import org.apache.hadoop.fs.contract.AbstractFSContract; + +/** + * CosN contract tests for opening files. + */ +public class TestCosNContractOpen extends AbstractContractOpenTest { + @Override + protected AbstractFSContract createContract(Configuration configuration) { + return new CosNContract(configuration); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractRename.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractRename.java new file mode 100644 index 0000000000000..f82c8dfb4cb87 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractRename.java @@ -0,0 +1,32 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.contract; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.contract.AbstractContractRenameTest; +import org.apache.hadoop.fs.contract.AbstractFSContract; + +/** + * CosN contract tests for renaming a file. + */ +public class TestCosNContractRename extends AbstractContractRenameTest { + @Override + protected AbstractFSContract createContract(Configuration configuration) { + return new CosNContract(configuration); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractRootDir.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractRootDir.java new file mode 100644 index 0000000000000..145aee9ca1048 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractRootDir.java @@ -0,0 +1,33 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.contract; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest; +import org.apache.hadoop.fs.contract.AbstractFSContract; + +/** + * root dir operations against an COS bucket. + */ +public class TestCosNContractRootDir + extends AbstractContractRootDirectoryTest { + @Override + protected AbstractFSContract createContract(Configuration configuration) { + return new CosNContract(configuration); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractSeek.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractSeek.java new file mode 100644 index 0000000000000..e3915676ee3bd --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractSeek.java @@ -0,0 +1,32 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.contract; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.contract.AbstractContractSeekTest; +import org.apache.hadoop.fs.contract.AbstractFSContract; + +/** + * CosN contract tests for seeking a position in a file. + */ +public class TestCosNContractSeek extends AbstractContractSeekTest { + @Override + protected AbstractFSContract createContract(Configuration configuration) { + return new CosNContract(configuration); + } +} diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/package-info.java b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/package-info.java new file mode 100644 index 0000000000000..97598b10bc73a --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/package-info.java @@ -0,0 +1,18 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.cosn.contract; \ No newline at end of file diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/resources/contract/cosn.xml b/hadoop-cloud-storage-project/hadoop-cos/src/test/resources/contract/cosn.xml new file mode 100644 index 0000000000000..ac4f58c066be0 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/resources/contract/cosn.xml @@ -0,0 +1,120 @@ + + + + + fs.contract.test.root-tests-enabled + true + + + + fs.contract.is-blobstore + true + + + + fs.contract.create-overwrites-directory + true + + + + fs.contract.create-visibility-delayed + true + + + + fs.contract.is-case-sensitive + true + + + + fs.contract.rename-returns-false-if-source-missing + false + + + + fs.contract.rename-remove-dest-if-empty-dir + false + + + + fs.contract.supports-append + false + + + + fs.contract.supports-atomic-directory-delete + false + + + + fs.contract.supports-atomic-rename + false + + + + fs.contract.supports-block-locality + false + + + + fs.contract.supports-concat + false + + + + fs.contract.supports-seek + true + + + + fs.contract.supports-seek-on-closed-file + false + + + + fs.contract.rejects-seek-past-eof + true + + + + fs.contract.rename-overwrites-dest + false + + + + fs.contract.supports-getfilestatus + true + + + + fs.contract.supports-unix-permissions + false + + + + fs.contract.supports-strict-exceptions + true + + + + fs.contract.test.random-seek-count + 10 + + + \ No newline at end of file diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/resources/core-site.xml b/hadoop-cloud-storage-project/hadoop-cos/src/test/resources/core-site.xml new file mode 100644 index 0000000000000..fbd23bb326eb4 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/resources/core-site.xml @@ -0,0 +1,107 @@ + + + + + + + + + + fs.cosn.credentials.provider + org.apache.hadoop.fs.cosn.auth.SimpleCredentialProvider + + This option allows the user to specify how to get the credentials. + Comma-separated class names of credential provider classes which + implement + com.qcloud.cos.auth.COSCredentialsProvider: + + 1.org.apache.hadoop.fs.cosn.auth.SimpleCredentialProvider: Obtain the + secret id and secret key + from fs.cosn.userinfo.secretId and fs.cosn.userinfo.secretKey in + core-site.xml + 2.org.apache.hadoop.fs.cosn.auth.EnvironmentVariableCredentialProvider: + Obtain the secret id and secret key + from the system environment variables named COS_SECRET_ID and + COS_SECRET_KEY + + If unspecified, the default order of credential providers is: + 1. org.apache.hadoop.fs.cosn.auth.SimpleCredentialProvider + 2. org.apache.hadoop.fs.cosn.auth.EnvironmentVariableCredentialProvider + + + + fs.cosn.impl + org.apache.hadoop.fs.cosn.CosNFileSystem + + The implementation class of the CosN Filesystem. + + + + + fs.AbstractFileSystem.cosn.impl + org.apache.hadoop.fs.cosn.CosN + + The implementation class of the CosN AbstractFileSystem. + + + + + fs.cosn.tmp.dir + /tmp/hadoop_cos + + Temporary files will be placed here. + + + + + fs.cosn.block.size + 8388608 + + Block size to use cosn filesystem, which is the part size for + MultipartUpload. + Considering the COS supports up to 10000 blocks, user should + estimate the maximum size of a single file. + for example, 8MB part size can allow writing a 78GB single file. + + + + + fs.cosn.upload.buffer.size + 536870912 + The total size of the memory buffer pool. + + + + fs.cosn.read.ahead.block.size + 1048576 + + Bytes to read ahead during a seek() before closing and + re-opening the cosn HTTP connection. + + + + + fs.cosn.read.ahead.queue.size + 64 + + The length of the pre-read queue. + + + + + + + + diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/test/resources/log4j.properties b/hadoop-cloud-storage-project/hadoop-cos/src/test/resources/log4j.properties new file mode 100644 index 0000000000000..1a6baaec65ae7 --- /dev/null +++ b/hadoop-cloud-storage-project/hadoop-cos/src/test/resources/log4j.properties @@ -0,0 +1,18 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# log4j configuration used during build and unit tests + +log4j.rootLogger=info,stdout +log4j.threshhold=ALL +log4j.appender.stdout=org.apache.log4j.ConsoleAppender +log4j.appender.stdout.layout=org.apache.log4j.PatternLayout +log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n diff --git a/hadoop-cloud-storage-project/pom.xml b/hadoop-cloud-storage-project/pom.xml index 50dd70ef6169a..f39e8c3aaf9f8 100644 --- a/hadoop-cloud-storage-project/pom.xml +++ b/hadoop-cloud-storage-project/pom.xml @@ -15,7 +15,7 @@ + https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -31,6 +31,7 @@ hadoop-cloud-storage + hadoop-cos diff --git a/hadoop-common-project/hadoop-annotations/pom.xml b/hadoop-common-project/hadoop-annotations/pom.xml index d8373c801f5c6..738f0ada8f1e9 100644 --- a/hadoop-common-project/hadoop-annotations/pom.xml +++ b/hadoop-common-project/hadoop-annotations/pom.xml @@ -15,7 +15,7 @@ + https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-common-project/hadoop-auth-examples/pom.xml b/hadoop-common-project/hadoop-auth-examples/pom.xml index a105885654a1a..fb904912999b8 100644 --- a/hadoop-common-project/hadoop-auth-examples/pom.xml +++ b/hadoop-common-project/hadoop-auth-examples/pom.xml @@ -15,7 +15,7 @@ + https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-common-project/hadoop-auth/pom.xml b/hadoop-common-project/hadoop-auth/pom.xml index d9ea33dfb2189..20a3e7059b154 100644 --- a/hadoop-common-project/hadoop-auth/pom.xml +++ b/hadoop-common-project/hadoop-auth/pom.xml @@ -15,7 +15,7 @@ + https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java index 5aeddac26d4ca..94d11f48cf2a9 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java @@ -681,7 +681,7 @@ public static void createAuthCookie(HttpServletResponse resp, String token, if (expires >= 0 && isCookiePersistent) { Date date = new Date(expires); SimpleDateFormat df = new SimpleDateFormat("EEE, " + - "dd-MMM-yyyy HH:mm:ss zzz"); + "dd-MMM-yyyy HH:mm:ss zzz", Locale.US); df.setTimeZone(TimeZone.getTimeZone("GMT")); sb.append("; Expires=").append(df.format(date)); } diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java index 7a918b20bb606..50eeb2a965e27 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java @@ -13,6 +13,7 @@ */ package org.apache.hadoop.security.authentication.server; +import com.google.common.annotations.VisibleForTesting; import org.apache.hadoop.security.authentication.client.AuthenticationException; import org.apache.hadoop.security.authentication.client.KerberosAuthenticator; import org.apache.commons.codec.binary.Base64; @@ -38,6 +39,8 @@ import java.security.Principal; import java.security.PrivilegedActionException; import java.security.PrivilegedExceptionAction; +import java.util.Collection; +import java.util.HashSet; import java.util.Properties; import java.util.Set; import java.util.regex.Pattern; @@ -94,10 +97,18 @@ public class KerberosAuthenticationHandler implements AuthenticationHandler { */ public static final String RULE_MECHANISM = TYPE + ".name.rules.mechanism"; + /** + * Constant for the list of endpoints that skips Kerberos authentication. + */ + @VisibleForTesting + static final String ENDPOINT_WHITELIST = TYPE + ".endpoint.whitelist"; + private static final Pattern ENDPOINT_PATTERN = Pattern.compile("^/[\\w]+"); + private String type; private String keytab; private GSSManager gssManager; private Subject serverSubject = new Subject(); + private final Collection whitelist = new HashSet<>(); /** * Creates a Kerberos SPNEGO authentication handler with the default @@ -173,6 +184,22 @@ public void init(Properties config) throws ServletException { if (ruleMechanism != null) { KerberosName.setRuleMechanism(ruleMechanism); } + + final String whitelistStr = config.getProperty(ENDPOINT_WHITELIST, null); + if (whitelistStr != null) { + final String[] strs = whitelistStr.trim().split("\\s*[,\n]\\s*"); + for (String s: strs) { + if (s.isEmpty()) continue; + if (ENDPOINT_PATTERN.matcher(s).matches()) { + whitelist.add(s); + } else { + throw new ServletException( + "The element of the whitelist: " + s + " must start with '/'" + + " and must not contain special characters afterwards"); + } + } + } + try { gssManager = Subject.doAs(serverSubject, new PrivilegedExceptionAction() { @@ -269,6 +296,16 @@ public boolean managementOperation(AuthenticationToken token, public AuthenticationToken authenticate(HttpServletRequest request, final HttpServletResponse response) throws IOException, AuthenticationException { + + // If the request servlet path is in the whitelist, + // skip Kerberos authentication and return anonymous token. + final String path = request.getServletPath(); + for(final String endpoint: whitelist) { + if (endpoint.equals(path)) { + return AuthenticationToken.ANONYMOUS; + } + } + AuthenticationToken token = null; String authorization = request.getHeader( KerberosAuthenticator.AUTHORIZATION); diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler.java index f452317f931a1..8cc8d03447a99 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler.java @@ -96,9 +96,8 @@ public class LdapAuthenticationHandler implements AuthenticationHandler { public static final String LDAP_BIND_DOMAIN = TYPE + ".binddomain"; /** - * Constant for the configuration property that indicates the base - * distinguished name (DN) to be used with the LDAP server. This value is - * appended to the provided user id for authentication purpose. + * Constant for the configuration property that indicates whether + * the LDAP server supports 'StartTLS' extension. */ public static final String ENABLE_START_TLS = TYPE + ".enablestarttls"; @@ -336,4 +335,4 @@ private static int indexOfDomainMatch(String userName) { return endIdx; } -} \ No newline at end of file +} diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java index 684d2c8c1e8ce..67c2c10237d49 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java @@ -281,7 +281,7 @@ static String replaceParameters(String format, if (paramNum != null) { try { int num = Integer.parseInt(paramNum); - if (num < 0 || num > params.length) { + if (num < 0 || num >= params.length) { throw new BadFormatString("index " + num + " from " + format + " is outside of the valid range 0 to " + (params.length - 1)); diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java index a7fc76f6a12e4..f0c350ed9594b 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java @@ -15,7 +15,6 @@ import com.google.common.annotations.VisibleForTesting; import java.nio.ByteBuffer; -import java.nio.charset.Charset; import java.security.SecureRandom; import java.util.Collections; import java.util.HashMap; @@ -36,7 +35,7 @@ import org.apache.hadoop.classification.InterfaceStability; import org.apache.zookeeper.KeeperException; import org.apache.zookeeper.ZooDefs.Perms; -import org.apache.zookeeper.client.ZooKeeperSaslClient; +import org.apache.zookeeper.client.ZKClientConfig; import org.apache.zookeeper.data.ACL; import org.apache.zookeeper.data.Id; import org.apache.zookeeper.data.Stat; @@ -368,7 +367,7 @@ protected CuratorFramework createCuratorClient(Properties config) LOG.info("Connecting to ZooKeeper with SASL/Kerberos" + "and using 'sasl' ACLs"); String principal = setJaasConfiguration(config); - System.setProperty(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY, + System.setProperty(ZKClientConfig.LOGIN_CONTEXT_NAME_KEY, JAAS_LOGIN_ENTRY_NAME); System.setProperty("zookeeper.authProvider.1", "org.apache.zookeeper.server.auth.SASLAuthenticationProvider"); diff --git a/hadoop-common-project/hadoop-auth/src/site/markdown/Examples.md b/hadoop-common-project/hadoop-auth/src/site/markdown/Examples.md index 4dad79dd063ae..4f29c8da5cf40 100644 --- a/hadoop-common-project/hadoop-auth/src/site/markdown/Examples.md +++ b/hadoop-common-project/hadoop-auth/src/site/markdown/Examples.md @@ -36,14 +36,14 @@ Login to the KDC using **kinit** and then use `curl` to fetch protected URL: $ kinit Please enter the password for tucu@LOCALHOST: - $ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/hadoop-auth-examples/kerberos/who + $ curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt http://$(hostname -f):8080/hadoop-auth-examples/kerberos/who Enter host password for user 'tucu': Hello Hadoop Auth Examples! * The `--negotiate` option enables SPNEGO in `curl`. -* The `-u foo` option is required but the user ignored (the principal +* The `-u :` option is required but the user ignored (the principal that has been kinit-ed is used). * The `-b` and `-c` are use to store and send HTTP Cookies. @@ -88,7 +88,7 @@ Try accessing protected resources using `curl`. The protected resources are: $ curl http://localhost:8080/hadoop-auth-examples/simple/who?user.name=foo - $ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/hadoop-auth-examples/kerberos/who + $ curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt http://$(hostname -f):8080/hadoop-auth-examples/kerberos/who ### Accessing the server using the Java client example diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java index bef3018ca4a70..629b68bffbbd9 100644 --- a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java @@ -89,6 +89,9 @@ public void setup() throws Exception { // handler handler = getNewAuthenticationHandler(); Properties props = getDefaultProperties(); + // Set whitelist for testing + props.setProperty(KerberosAuthenticationHandler.ENDPOINT_WHITELIST, + "/white,/white2,/white3"); try { handler.init(props); } catch (Exception ex) { @@ -371,6 +374,27 @@ public void testRequestWithInvalidKerberosAuthorization() { } } + @Test + public void testRequestToWhitelist() throws Exception { + final String token = new Base64(0).encodeToString(new byte[]{0, 1, 2}); + final HttpServletRequest request = Mockito.mock(HttpServletRequest.class); + final HttpServletResponse response = + Mockito.mock(HttpServletResponse.class); + Mockito.when(request.getHeader(KerberosAuthenticator.AUTHORIZATION)) + .thenReturn(KerberosAuthenticator.NEGOTIATE + token); + Mockito.when(request.getServletPath()).thenReturn("/white"); + handler.authenticate(request, response); + Mockito.when(request.getServletPath()).thenReturn("/white4"); + try { + handler.authenticate(request, response); + Assert.fail(); + } catch (AuthenticationException ex) { + // Expected + } catch (Exception ex) { + Assert.fail(); + } + } + @After public void tearDown() throws Exception { if (handler != null) { diff --git a/hadoop-common-project/hadoop-common/dev-support/jdiff/Apache_Hadoop_Common_2.10.0.xml b/hadoop-common-project/hadoop-common/dev-support/jdiff/Apache_Hadoop_Common_2.10.0.xml new file mode 100644 index 0000000000000..7b63a07b309c7 --- /dev/null +++ b/hadoop-common-project/hadoop-common/dev-support/jdiff/Apache_Hadoop_Common_2.10.0.xml @@ -0,0 +1,40847 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + UnsupportedOperationException + + If a key is deprecated in favor of multiple keys, they are all treated as + aliases of each other, and setting any one of them resets all the others + to the new value. + + If you have multiple deprecation entries to add, it is more efficient to + use #addDeprecations(DeprecationDelta[] deltas) instead. + + @param key + @param newKeys + @param customMessage + @deprecated use {@link #addDeprecation(String key, String newKey, + String customMessage)} instead]]> + + + + + + + + UnsupportedOperationException + + If you have multiple deprecation entries to add, it is more efficient to + use #addDeprecations(DeprecationDelta[] deltas) instead. + + @param key + @param newKey + @param customMessage]]> + + + + + + + UnsupportedOperationException + + If a key is deprecated in favor of multiple keys, they are all treated as + aliases of each other, and setting any one of them resets all the others + to the new value. + + If you have multiple deprecation entries to add, it is more efficient to + use #addDeprecations(DeprecationDelta[] deltas) instead. + + @param key Key that is to be deprecated + @param newKeys list of keys that take up the values of deprecated key + @deprecated use {@link #addDeprecation(String key, String newKey)} instead]]> + + + + + + + UnsupportedOperationException + + If you have multiple deprecation entries to add, it is more efficient to + use #addDeprecations(DeprecationDelta[] deltas) instead. + + @param key Key that is to be deprecated + @param newKey key that takes up the value of deprecated key]]> + + + + + + key is deprecated. + + @param key the parameter which is to be checked for deprecation + @return true if the key is deprecated and + false otherwise.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + final. + + @param name resource to be added, the classpath is examined for a file + with that name.]]> + + + + + + + + + + final. + + @param url url of the resource to be added, the local filesystem is + examined directly to find the resource, without referring to + the classpath.]]> + + + + + + + + + + final. + + @param file file-path of resource to be added, the local filesystem is + examined directly to find the resource, without referring to + the classpath.]]> + + + + + + + + + + final. + + WARNING: The contents of the InputStream will be cached, by this method. + So use this sparingly because it does increase the memory consumption. + + @param in InputStream to deserialize the object from. In will be read from + when a get or set is called next. After it is read the stream will be + closed.]]> + + + + + + + + + + + final. + + @param in InputStream to deserialize the object from. + @param name the name of the resource because InputStream.toString is not + very descriptive some times.]]> + + + + + + + + + + + final. + + @param conf Configuration object from which to load properties]]> + + + + + + + + + + + name property, null if + no such property exists. If the key is deprecated, it returns the value of + the first key which replaces the deprecated key and is not null. + + Values are processed for variable expansion + before being returned. + + @param name the property name, will be trimmed before get value. + @return the value of the name or its replacing property, + or null if no such property exists.]]> + + + + + + + + + + + + + + + name property, but only for + names which have no valid value, usually non-existent or commented + out in XML. + + @param name the property name + @return true if the property name exists without value]]> + + + + + + name property as a trimmed String, + null if no such property exists. + If the key is deprecated, it returns the value of + the first key which replaces the deprecated key and is not null + + Values are processed for variable expansion + before being returned. + + @param name the property name. + @return the value of the name or its replacing property, + or null if no such property exists.]]> + + + + + + + name property as a trimmed String, + defaultValue if no such property exists. + See @{Configuration#getTrimmed} for more details. + + @param name the property name. + @param defaultValue the property default value. + @return the value of the name or defaultValue + if it is not set.]]> + + + + + + name property, without doing + variable expansion.If the key is + deprecated, it returns the value of the first key which replaces + the deprecated key and is not null. + + @param name the property name. + @return the value of the name property or + its replacing property and null if no such property exists.]]> + + + + + + + value of the name property. If + name is deprecated or there is a deprecated name associated to it, + it sets the value to both names. Name will be trimmed before put into + configuration. + + @param name property name. + @param value property value.]]> + + + + + + + + value of the name property. If + name is deprecated, it also sets the value to + the keys that replace the deprecated key. Name will be trimmed before put + into configuration. + + @param name property name. + @param value property value. + @param source the place that this configuration value came from + (For debugging). + @throws IllegalArgumentException when the value or name is null.]]> + + + + + + + + + + + + + + + + + + + + name. If the key is deprecated, + it returns the value of the first key which replaces the deprecated key + and is not null. + If no such property exists, + then defaultValue is returned. + + @param name property name, will be trimmed before get value. + @param defaultValue default value. + @return property value, or defaultValue if the property + doesn't exist.]]> + + + + + + + name property as an int. + + If no such property exists, the provided default value is returned, + or if the specified value is not a valid int, + then an error is thrown. + + @param name property name. + @param defaultValue default value. + @throws NumberFormatException when the value is invalid + @return property value as an int, + or defaultValue.]]> + + + + + + name property as a set of comma-delimited + int values. + + If no such property exists, an empty array is returned. + + @param name property name + @return property value interpreted as an array of comma-delimited + int values]]> + + + + + + + name property to an int. + + @param name property name. + @param value int value of the property.]]> + + + + + + + name property as a long. + If no such property exists, the provided default value is returned, + or if the specified value is not a valid long, + then an error is thrown. + + @param name property name. + @param defaultValue default value. + @throws NumberFormatException when the value is invalid + @return property value as a long, + or defaultValue.]]> + + + + + + + name property as a long or + human readable format. If no such property exists, the provided default + value is returned, or if the specified value is not a valid + long or human readable format, then an error is thrown. You + can use the following suffix (case insensitive): k(kilo), m(mega), g(giga), + t(tera), p(peta), e(exa) + + @param name property name. + @param defaultValue default value. + @throws NumberFormatException when the value is invalid + @return property value as a long, + or defaultValue.]]> + + + + + + + name property to a long. + + @param name property name. + @param value long value of the property.]]> + + + + + + + name property as a float. + If no such property exists, the provided default value is returned, + or if the specified value is not a valid float, + then an error is thrown. + + @param name property name. + @param defaultValue default value. + @throws NumberFormatException when the value is invalid + @return property value as a float, + or defaultValue.]]> + + + + + + + name property to a float. + + @param name property name. + @param value property value.]]> + + + + + + + name property as a double. + If no such property exists, the provided default value is returned, + or if the specified value is not a valid double, + then an error is thrown. + + @param name property name. + @param defaultValue default value. + @throws NumberFormatException when the value is invalid + @return property value as a double, + or defaultValue.]]> + + + + + + + name property to a double. + + @param name property name. + @param value property value.]]> + + + + + + + name property as a boolean. + If no such property is specified, or if the specified value is not a valid + boolean, then defaultValue is returned. + + @param name property name. + @param defaultValue default value. + @return property value as a boolean, + or defaultValue.]]> + + + + + + + name property to a boolean. + + @param name property name. + @param value boolean value of the property.]]> + + + + + + + + + + + + + + name property to the given type. This + is equivalent to set(<name>, value.toString()). + @param name property name + @param value new value]]> + + + + + + + + + + + + + + + name to the given time duration. This + is equivalent to set(<name>, value + <time suffix>). + @param name Property name + @param value Time duration + @param unit Unit of time]]> + + + + + + + + + + + + + + + + + + + name property as a Pattern. + If no such property is specified, or if the specified value is not a valid + Pattern, then DefaultValue is returned. + Note that the returned value is NOT trimmed by this method. + + @param name property name + @param defaultValue default value + @return property value as a compiled Pattern, or defaultValue]]> + + + + + + + Pattern. + If the pattern is passed as null, sets the empty pattern which results in + further calls to getPattern(...) returning the default value. + + @param name property name + @param pattern new value]]> + + + + + + + + + + + + + + + + + + + name property as + a collection of Strings. + If no such property is specified then empty collection is returned. +

+ This is an optimized version of {@link #getStrings(String)} + + @param name property name. + @return property value as a collection of Strings.]]> + + + + + + name property as + an array of Strings. + If no such property is specified then null is returned. + + @param name property name. + @return property value as an array of Strings, + or null.]]> + + + + + + + name property as + an array of Strings. + If no such property is specified then default value is returned. + + @param name property name. + @param defaultValue The default value + @return property value as an array of Strings, + or default value.]]> + + + + + + name property as + a collection of Strings, trimmed of the leading and trailing whitespace. + If no such property is specified then empty Collection is returned. + + @param name property name. + @return property value as a collection of Strings, or empty Collection]]> + + + + + + name property as + an array of Strings, trimmed of the leading and trailing whitespace. + If no such property is specified then an empty array is returned. + + @param name property name. + @return property value as an array of trimmed Strings, + or empty array.]]> + + + + + + + name property as + an array of Strings, trimmed of the leading and trailing whitespace. + If no such property is specified then default value is returned. + + @param name property name. + @param defaultValue The default value + @return property value as an array of trimmed Strings, + or default value.]]> + + + + + + + name property as + as comma delimited values. + + @param name property name. + @param values The values]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + hostProperty as a + InetSocketAddress. If hostProperty is + null, addressProperty will be used. This + is useful for cases where we want to differentiate between host + bind address and address clients should use to establish connection. + + @param hostProperty bind host property name. + @param addressProperty address property name. + @param defaultAddressValue the default value + @param defaultPort the default port + @return InetSocketAddress]]> + + + + + + + + name property as a + InetSocketAddress. + @param name property name. + @param defaultAddress the default value + @param defaultPort the default port + @return InetSocketAddress]]> + + + + + + + name property as + a host:port.]]> + + + + + + + + + name property as a host:port. The wildcard + address is replaced with the local host's address. If the host and address + properties are configured the host component of the address will be combined + with the port component of the addr to generate the address. This is to allow + optional control over which host name is used in multi-home bind-host + cases where a host can have multiple names + @param hostProperty the bind-host configuration name + @param addressProperty the service address configuration name + @param defaultAddressValue the service default address configuration value + @param addr InetSocketAddress of the service listener + @return InetSocketAddress for clients to connect]]> + + + + + + + name property as a host:port. The wildcard + address is replaced with the local host's address. + @param name property name. + @param addr InetSocketAddress of a listener to store in the given property + @return InetSocketAddress for clients to connect]]> + + + + + + + + + + + + + + + + + + + + name property + as an array of Class. + The value of the property specifies a list of comma separated class names. + If no such property is specified, then defaultValue is + returned. + + @param name the property name. + @param defaultValue default value. + @return property value as a Class[], + or defaultValue.]]> + + + + + + + name property as a Class. + If no such property is specified, then defaultValue is + returned. + + @param name the class name. + @param defaultValue default value. + @return property value as a Class, + or defaultValue.]]> + + + + + + + + name property as a Class + implementing the interface specified by xface. + + If no such property is specified, then defaultValue is + returned. + + An exception is thrown if the returned class does not implement the named + interface. + + @param name the class name. + @param defaultValue default value. + @param xface the interface implemented by the named class. + @return property value as a Class, + or defaultValue.]]> + + + + + + + name property as a List + of objects implementing the interface specified by xface. + + An exception is thrown if any of the classes does not exist, or if it does + not implement the named interface. + + @param name the property name. + @param xface the interface implemented by the classes named by + name. + @return a List of objects implementing xface.]]> + + + + + + + + name property to the name of a + theClass implementing the given interface xface. + + An exception is thrown if theClass does not implement the + interface xface. + + @param name property name. + @param theClass property value. + @param xface the interface implemented by the named class.]]> + + + + + + + + dirsProp with + the given path. If dirsProp contains multiple directories, + then one is chosen based on path's hash code. If the selected + directory does not exist, an attempt is made to create it. + + @param dirsProp directory in which to locate the file. + @param path file-path. + @return local file under the directory with the given path.]]> + + + + + + + + dirsProp with + the given path. If dirsProp contains multiple directories, + then one is chosen based on path's hash code. If the selected + directory does not exist, an attempt is made to create it. + + @param dirsProp directory in which to locate the file. + @param path file-path. + @return local file under the directory with the given path.]]> + + + + + + + + + + + + name. + + @param name configuration resource name. + @return an input stream attached to the resource.]]> + + + + + + name. + + @param name configuration resource name. + @return a reader attached to the resource.]]> + + + + + + + + + + + + + + + + + + + + + + String + key-value pairs in the configuration. + + @return an iterator over the entries.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + When property name is not empty and the property exists in the + configuration, this method writes the property and its attributes + to the {@link Writer}. + +

+ +

  • + When property name is null or empty, this method writes all the + configuration properties and their attributes to the {@link Writer}. +
    • +

    + +

  • + When property name is not empty but the property doesn't exist in + the configuration, this method throws an {@link IllegalArgumentException}. +
    • +

    + @param out the writer to write to.]]> + + + + + + + + + + When propertyName is not empty, and the property exists + in the configuration, the format of the output would be, + 

    +  {
+    "property": {
+      "key" : "key1",
+      "value" : "value1",
+      "isFinal" : "key1.isFinal",
+      "resource" : "key1.resource"
+    }
+  }
+
    + + +
  • + When propertyName is null or empty, it behaves same as + {@link #dumpConfiguration(Configuration, Writer)}, the + output would be, + 
    +  { "properties" :
+      [ { key : "key1",
+          value : "value1",
+          isFinal : "key1.isFinal",
+          resource : "key1.resource" },
+        { key : "key2",
+          value : "value2",
+          isFinal : "ke2.isFinal",
+          resource : "key2.resource" }
+       ]
+   }
+
    +
    • + +
  • + When propertyName is not empty, and the property is not + found in the configuration, this method will throw an + {@link IllegalArgumentException}. +
    • +

    + @param config the configuration + @param propertyName property name + @param out the Writer to write to + @throws IOException + @throws IllegalArgumentException when property name is not + empty and the property is not found in configuration]]> + + + + + + + + + { "properties" : + [ { key : "key1", + value : "value1", + isFinal : "key1.isFinal", + resource : "key1.resource" }, + { key : "key2", + value : "value2", + isFinal : "ke2.isFinal", + resource : "key2.resource" } + ] + } + + + It does not output the properties of the configuration object which + is loaded from an input stream. +

    + + @param config the configuration + @param out the Writer to write to + @throws IOException]]> + + + + + + + + + + + + + + + + + + + true to set quiet-mode on, false + to turn it off.]]> + + + + + + + + + + + + + + + + + + + + + with matching keys]]> + + + + + + + + + + + + Resources + +

    Configurations are specified by resources. A resource contains a set of + name/value pairs as XML data. Each resource is named by either a + String or by a {@link Path}. If named by a String, + then the classpath is examined for a file with that name. If named by a + Path, then the local filesystem is examined directly, without + referring to the classpath. + +

    Unless explicitly turned off, Hadoop by default specifies two + resources, loaded in-order from the classpath:

      +
    1. + + core-default.xml: Read-only defaults for hadoop.
      2. +
    2. core-site.xml: Site-specific configuration for a given hadoop + installation.
      3. +
    + Applications may add additional resources, which are loaded + subsequent to these resources in the order they are added. + +

    Final Parameters

    + +

    Configuration parameters may be declared final. + Once a resource declares a value final, no subsequently-loaded + resource can alter that value. + For example, one might define a final parameter with: + 

    +  <property>
+    <name>dfs.hosts.include</name>
+    <value>/etc/hadoop/conf/hosts.include</value>
+    <final>true</final>
+  </property>
    + + Administrators typically define parameters as final in + core-site.xml for values that user applications may not alter. + +

    Variable Expansion

    + +

    Value strings are first processed for variable expansion. The + available properties are:

      +
    1. Other properties defined in this Configuration; and, if a name is + undefined here,
      2. +
    2. Properties in {@link System#getProperties()}.
      3. +
    + +

    For example, if a configuration resource contains the following property + definitions: + 

    +  <property>
+    <name>basedir</name>
+    <value>/user/${user.name}</value>
+  </property>
+  
+  <property>
+    <name>tempdir</name>
+    <value>${basedir}/tmp</value>
+  </property>
    + + When conf.get("tempdir") is called, then ${basedir} + will be resolved to another property in this Configuration, while + ${user.name} would then ordinarily be resolved to the value + of the System property with that name. +

    When conf.get("otherdir") is called, then ${env.BASE_DIR} + will be resolved to the value of the ${BASE_DIR} environment variable. + It supports ${env.NAME:-default} and ${env.NAME-default} notations. + The former is resolved to "default" if ${NAME} environment variable is undefined + or its value is empty. + The latter behaves the same way only if ${NAME} is undefined. +

    By default, warnings will be given to any deprecated configuration + parameters and these are suppressible by configuring + log4j.logger.org.apache.hadoop.conf.Configuration.deprecation in + log4j.properties file.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This implementation generates the key material and calls the + {@link #createKey(String, byte[], Options)} method. + + @param name the base name of the key + @param options the options for the new key. + @return the version name of the first version of the key. + @throws IOException + @throws NoSuchAlgorithmException]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This implementation generates the key material and calls the + {@link #rollNewVersion(String, byte[])} method. + + @param name the basename of the key + @return the name of the new version of the key + @throws IOException]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + KeyProvider implementations must be thread safe.]]> + + + + + + + + + + + + + + + + + + + + + + NULL if + a provider for the specified URI scheme could not be found. + @throws IOException thrown if the provider failed to initialize.]]> + + + + + + + + + + + + + + + + + + + + + uri has syntax error]]> + + + + + + + + + + + + + + + + uri is + not found]]> + + + + + + + + + + + + + + + + + + + + + + + uri + determines a configuration property name, + fs.AbstractFileSystem.scheme.impl whose value names the + AbstractFileSystem class. + + The entire URI and conf is passed to the AbstractFileSystem factory method. + + @param uri for the file system to be created. + @param conf which is passed to the file system impl. + + @return file system for the given URI. + + @throws UnsupportedFileSystemException if the file system for + uri is not supported.]]> + + + + + + + + + + + + default port;]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + describing modifications + @throws IOException if an ACL could not be modified]]> + + + + + + + + describing entries to remove + @throws IOException if an ACL could not be modified]]> + + + + + + + + + + + + + + + + + + + + + + describing modifications, must include entries + for user, group, and others for compatibility with permission bits. + @throws IOException if an ACL could not be modified]]> + + + + + + + which returns each AclStatus + @throws IOException if an ACL could not be read]]> + + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to modify + @param name xattr name. + @param value xattr value. + @throws IOException]]> + + + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to modify + @param name xattr name. + @param value xattr value. + @param flag xattr set flag + @throws IOException]]> + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attribute + @param name xattr name. + @return byte[] xattr value. + @throws IOException]]> + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attributes + @return Map describing the XAttrs of the file or directory + @throws IOException]]> + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attributes + @param names XAttr names. + @return Map describing the XAttrs of the file or directory + @throws IOException]]> + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attributes + @return Map describing the XAttrs of the file or directory + @throws IOException]]> + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to remove extended attribute + @param name xattr name + @throws IOException]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + After a successful call, {@code buf.position()} will be advanced by the + number of bytes read and {@code buf.limit()} will be unchanged. +

    + In the case of an exception, the state of the buffer (the contents of the + buffer, the {@code buf.position()}, the {@code buf.limit()}, etc.) is + undefined, and callers should be prepared to recover from this eventuality. +

    + Callers should use {@link StreamCapabilities#hasCapability(String)} with + {@link StreamCapabilities#PREADBYTEBUFFER} to check if the underlying + stream supports this interface, otherwise they might get a + {@link UnsupportedOperationException}. +

    + Implementations should treat 0-length requests as legitimate, and must not + signal an error upon their receipt. + + @param position position within file + @param buf the ByteBuffer to receive the results of the read operation. + @return the number of bytes read, possibly zero, or -1 if reached + end-of-stream + @throws IOException if there is some error performing the read]]> + + + + + + + + + + + + + + + After a successful call, buf.position() will be advanced by the number + of bytes read and buf.limit() should be unchanged. +

    + In the case of an exception, the values of buf.position() and buf.limit() + are undefined, and callers should be prepared to recover from this + eventuality. +

    + Many implementations will throw {@link UnsupportedOperationException}, so + callers that are not confident in support for this method from the + underlying filesystem should be prepared to handle that exception. +

    + Implementations should treat 0-length requests as legitimate, and must not + signal an error upon their receipt. + + @param buf + the ByteBuffer to receive the results of the read operation. + @return the number of bytes read, possibly zero, or -1 if + reach end-of-stream + @throws IOException + if there is some error performing the read]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + setReplication of FileSystem + @param src file name + @param replication new replication + @throws IOException + @return true if successful; + false if file does not exist or is a directory]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + core-default.xml]]> + + + + + + + + + + + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + + + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + + + + + + + + core-default.xml]]> + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + + + + + + + core-default.xml]]> + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + core-default.xml]]> + + + + + + + + core-default.xml]]> + + + + + + + + core-default.xml]]> + + + + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + core-default.xml]]> + + + + + + core-default.xml]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND) + +

    + + Use the CreateFlag as follows: +

      +
    1. CREATE - to create a file if it does not exist, + else throw FileAlreadyExists.
      2. +
    2. APPEND - to append to a file if it exists, + else throw FileNotFoundException.
      3. +
    3. OVERWRITE - to truncate a file if it exists, + else throw FileNotFoundException.
      4. +
    4. CREATE|APPEND - to create a file if it does not exist, + else append to an existing file.
      5. +
    5. CREATE|OVERWRITE - to create a file if it does not exist, + else overwrite an existing file.
      6. +
    6. SYNC_BLOCK - to force closed blocks to the disk device. + In addition {@link Syncable#hsync()} should be called after each write, + if true synchronous behavior is required.
      7. +
    7. LAZY_PERSIST - Create the block on transient storage (RAM) if + available.
      8. +
    8. APPEND_NEWBLOCK - Append data to a new block instead of end of the last + partial block.
      9. +
    + + Following combinations are not valid and will result in + {@link HadoopIllegalArgumentException}: +
      +
    1. APPEND|OVERWRITE
      2. +
    2. CREATE|APPEND|OVERWRITE
      3. +
    ]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + absOrFqPath     is not supported. + @throws IOException If the file system for absOrFqPath could + not be instantiated.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + defaultFsUri     is not supported]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + NewWdir can be one of: +
      +
    • relative path: "foo/bar";
      • +
    • absolute without scheme: "/foo/bar"
      • +
    • fully qualified with scheme: "xx://auth/foo/bar"
      • +
    +
    + Illegal WDs: +
      +
    • relative with scheme: "xx:foo/bar"
      • +
    • non existent directory
      • +
    ]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + f does not exist + @throws AccessControlException if access denied + @throws IOException If an IO Error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server + + RuntimeExceptions: + @throws InvalidPathException If path f is not valid]]> + + + + + + + + + + + + + + + + + + + + +
  • Progress - to report progress on the operation - default null +
  • Permission - umask is applied against permisssion: default is + FsPermissions:getDefault() + +
  • CreateParent - create missing parent path; default is to not + to create parents +
  • The defaults for the following are SS defaults of the file + server implementing the target path. Not all parameters make sense + for all kinds of file system - eg. localFS ignores Blocksize, + replication, checksum +
      +
    • BufferSize - buffersize used in FSDataOutputStream +
    • Blocksize - block size for file blocks +
    • ReplicationFactor - replication for blocks +
    • ChecksumParam - Checksum parameters. server default is used + if not specified. +
    + + + @return {@link FSDataOutputStream} for created file + + @throws AccessControlException If access is denied + @throws FileAlreadyExistsException If file f already exists + @throws FileNotFoundException If parent of f does not exist + and createParent is false + @throws ParentNotDirectoryException If parent of f is not a + directory. + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server + + RuntimeExceptions: + @throws InvalidPathException If path f is not valid]]> + + + + + + + + + + + + + + dir already + exists + @throws FileNotFoundException If parent of dir does not exist + and createParent is false + @throws ParentNotDirectoryException If parent of dir is not a + directory + @throws UnsupportedFileSystemException If file system for dir + is not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server + + RuntimeExceptions: + @throws InvalidPathException If path dir is not valid]]> + + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server + + RuntimeExceptions: + @throws InvalidPathException If path f is invalid]]> + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f + is not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + + +
  • Fails if path is a directory. +
  • Fails if path does not exist. +
  • Fails if path is not closed. +
  • Fails if new size is greater than current size. + + @param f The path to the file to be truncated + @param newLength The size the file is to be truncated to + + @return true if the file has been truncated to the desired + newLength and is immediately available to be reused for + write operations such as append, or + false if a background process of adjusting the length of + the last block has been started, and clients should wait for it to + complete before proceeding with further file updates. + + @throws AccessControlException If access is denied + @throws FileNotFoundException If file f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + f does not exist + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + + + + + +
  • Fails if src is a file and dst is a directory. +
  • Fails if src is a directory and dst is a file. +
  • Fails if the parent of dst does not exist or is a file. + +

    + If OVERWRITE option is not passed as an argument, rename fails if the dst + already exists. +

    + If OVERWRITE option is passed as an argument, rename overwrites the dst if + it is a file or an empty directory. Rename fails if dst is a non-empty + directory. +

    + Note that atomicity of rename is dependent on the file system + implementation. Please refer to the file system documentation for details +

    + + @param src path to be renamed + @param dst new path after rename + + @throws AccessControlException If access is denied + @throws FileAlreadyExistsException If dst already exists and + options has {@link Options.Rename#OVERWRITE} + option false. + @throws FileNotFoundException If src does not exist + @throws ParentNotDirectoryException If parent of dst is not a + directory + @throws UnsupportedFileSystemException If file system for src + and dst is not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f + is not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server + + RuntimeExceptions: + @throws HadoopIllegalArgumentException If username or + groupname is invalid.]]> + + + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + f does not exist + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred]]> + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If the given path does not refer to a symlink + or an I/O error occurred]]> + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + + + + + + Given a path referring to a symlink of form: + + <---X---> + fs://host/A/B/link + <-----Y-----> + + In this path X is the scheme and authority that identify the file system, + and Y is the path leading up to the final path component "link". If Y is + a symlink itself then let Y' be the target of Y and X' be the scheme and + authority of Y'. Symlink targets may: + + 1. Fully qualified URIs + + fs://hostX/A/B/file Resolved according to the target file system. + + 2. Partially qualified URIs (eg scheme but no host) + + fs:///A/B/file Resolved according to the target file system. Eg resolving + a symlink to hdfs:///A results in an exception because + HDFS URIs must be fully qualified, while a symlink to + file:///A will not since Hadoop's local file systems + require partially qualified URIs. + + 3. Relative paths + + path Resolves to [Y'][path]. Eg if Y resolves to hdfs://host/A and path + is "../B/file" then [Y'][path] is hdfs://host/B/file + + 4. Absolute paths + + path Resolves to [X'][path]. Eg if Y resolves hdfs://host/A/B and path + is "/file" then [X][path] is hdfs://host/file + + + @param target the target of the symbolic link + @param link the path to be created that points to target + @param createParent if true then missing parent dirs are created if + false then parent must exist + + + @throws AccessControlException If access is denied + @throws FileAlreadyExistsException If file linkcode> already exists + @throws FileNotFoundException If target does not exist + @throws ParentNotDirectoryException If parent of link is not a + directory. + @throws UnsupportedFileSystemException If file system for + target or link is not supported + @throws IOException If an I/O error occurred]]> + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + + + + + + + + f does not exist + @throws UnsupportedFileSystemException If file system for f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + f is + not supported + @throws IOException If an I/O error occurred + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + describing modifications + @throws IOException if an ACL could not be modified]]> + + + + + + + + describing entries to remove + @throws IOException if an ACL could not be modified]]> + + + + + + + + + + + + + + + + + + + + + + describing modifications, must include entries + for user, group, and others for compatibility with permission bits. + @throws IOException if an ACL could not be modified]]> + + + + + + + which returns each AclStatus + @throws IOException if an ACL could not be read]]> + + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to modify + @param name xattr name. + @param value xattr value. + @throws IOException]]> + + + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to modify + @param name xattr name. + @param value xattr value. + @param flag xattr set flag + @throws IOException]]> + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attribute + @param name xattr name. + @return byte[] xattr value. + @throws IOException]]> + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attributes + @return Map describing the XAttrs of the file or directory + @throws IOException]]> + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attributes + @param names XAttr names. + @return Map describing the XAttrs of the file or directory + @throws IOException]]> + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to remove extended attribute + @param name xattr name + @throws IOException]]> + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attributes + @return List of the XAttr names of the file or directory + @throws IOException]]> + + + + + + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + Exceptions applicable to file systems accessed over RPC: + @throws RpcClientException If an exception occurred in the RPC client + @throws RpcServerException If an exception occurred in the RPC server + @throws UnexpectedServerException If server implementation throws + undeclared exception to RPC server]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + *** Path Names *** +

    + + The Hadoop file system supports a URI name space and URI names. + It offers a forest of file systems that can be referenced using fully + qualified URIs. + Two common Hadoop file systems implementations are +

      +
    • the local file system: file:///path +
    • the hdfs file system hdfs://nnAddress:nnPort/path +
    + + While URI names are very flexible, it requires knowing the name or address + of the server. For convenience one often wants to access the default system + in one's environment without knowing its name/address. This has an + additional benefit that it allows one to change one's default fs + (e.g. admin moves application from cluster1 to cluster2). +

    + + To facilitate this, Hadoop supports a notion of a default file system. + The user can set his default file system, although this is + typically set up for you in your environment via your default config. + A default file system implies a default scheme and authority; slash-relative + names (such as /for/bar) are resolved relative to that default FS. + Similarly a user can also have working-directory-relative names (i.e. names + not starting with a slash). While the working directory is generally in the + same default FS, the wd can be in a different FS. +

    + Hence Hadoop path names can be one of: +

      +
    • fully qualified URI: scheme://authority/path +
    • slash relative names: /path relative to the default file system +
    • wd-relative names: path relative to the working dir +
    + Relative paths with scheme (scheme:foo/bar) are illegal. + +

    + ****The Role of the FileContext and configuration defaults**** +

    + The FileContext provides file namespace context for resolving file names; + it also contains the umask for permissions, In that sense it is like the + per-process file-related state in Unix system. + These two properties +

      +
    • default file system i.e your slash) +
    • umask +
    + in general, are obtained from the default configuration file + in your environment, (@see {@link Configuration}). + + No other configuration parameters are obtained from the default config as + far as the file context layer is concerned. All file system instances + (i.e. deployments of file systems) have default properties; we call these + server side (SS) defaults. Operation like create allow one to select many + properties: either pass them in as explicit parameters or use + the SS properties. +

    + The file system related SS defaults are +

      +
    • the home directory (default is "/user/userName") +
    • the initial wd (only for local fs) +
    • replication factor +
    • block size +
    • buffer size +
    • encryptDataTransfer +
    • checksum option. (checksumType and bytesPerChecksum) +
    + +

    + *** Usage Model for the FileContext class *** +

    + Example 1: use the default config read from the $HADOOP_CONFIG/core.xml. + Unspecified values come from core-defaults.xml in the release jar. +

      +
    • myFContext = FileContext.getFileContext(); // uses the default config + // which has your default FS +
    • myFContext.create(path, ...); +
    • myFContext.setWorkingDir(path) +
    • myFContext.open (path, ...); +
    + Example 2: Get a FileContext with a specific URI as the default FS +
      +
    • myFContext = FileContext.getFileContext(URI) +
    • myFContext.create(path, ...); + ... +
    + Example 3: FileContext with local file system as the default +
      +
    • myFContext = FileContext.getLocalFSFileContext() +
    • myFContext.create(path, ...); +
    • ... +
    + Example 4: Use a specific config, ignoring $HADOOP_CONFIG + Generally you should not need use a config unless you are doing +
      +
    • configX = someConfigSomeOnePassedToYou. +
    • myFContext = getFileContext(configX); // configX is not changed, + // is passed down +
    • myFContext.create(path, ...); +
    • ... +
    ]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This implementation throws an UnsupportedOperationException. + + @return the protocol scheme for this FileSystem. + @throws UnsupportedOperationException if the operation is unsupported + (default).]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
  • + If the configuration has the property + {@code "fs.$SCHEME.impl.disable.cache"} set to true, + a new instance will be created, initialized with the supplied URI and + configuration, then returned without being cached. +
    • +
  • + If the there is a cached FS instance matching the same URI, it will + be returned. +
    • +
  • + Otherwise: a new FS instance will be created, initialized with the + configuration and URI, cached and returned to the caller. +
    • + + @throws IOException if the FileSystem cannot be instantiated.]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + if f == null : + result = null + elif f.getLen() <= start: + result = [] + else result = [ locations(FS, b) for b in blocks(FS, p, s, s+l)] + + This call is most helpful with and distributed filesystem + where the hostnames of machines that contain blocks of the given file + can be determined. + + The default implementation returns an array containing one element: + 
    + BlockLocation( { "localhost:50010" },  { "localhost" }, 0, file.getLen())
+
    > + + @param file FilesStatus to get data from + @param start offset into the given file + @param len length for which to get locations for + @throws IOException IO failure]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Important: the default implementation is not atomic + @param f path to use for create + @throws IOException IO failure]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
  • Fails if src is a file and dst is a directory.
    • +
  • Fails if src is a directory and dst is a file.
    • +
  • Fails if the parent of dst does not exist or is a file.
    • + +

    + If OVERWRITE option is not passed as an argument, rename fails + if the dst already exists. +

    + If OVERWRITE option is passed as an argument, rename overwrites + the dst if it is a file or an empty directory. Rename fails if dst is + a non-empty directory. +

    + Note that atomicity of rename is dependent on the file system + implementation. Please refer to the file system documentation for + details. This default implementation is non atomic. +

    + This method is deprecated since it is a temporary method added to + support the transition from FileSystem to FileContext for user + applications. + + @param src path to be renamed + @param dst new path after rename + @throws FileNotFoundException src path does not exist, or the parent + path of dst does not exist. + @throws FileAlreadyExistsException dest path exists and is a file + @throws ParentNotDirectoryException if the parent path of dest is not + a directory + @throws IOException on failure]]> + + + + + + + + +

  • Fails if path is a directory.
    • +
  • Fails if path does not exist.
    • +
  • Fails if path is not closed.
    • +
  • Fails if new size is greater than current size.
    • + + @param f The path to the file to be truncated + @param newLength The size the file is to be truncated to + + @return true if the file has been truncated to the desired + newLength and is immediately available to be reused for + write operations such as append, or + false if a background process of adjusting the length of + the last block has been started, and clients should wait for it to + complete before proceeding with further file updates. + @throws IOException IO failure + @throws UnsupportedOperationException if the operation is unsupported + (default).]]> +     +     + + + + + + + + + + + + + + + + + + + + +
  • Clean shutdown of the JVM cannot be guaranteed.
    • +
  • The time to shut down a FileSystem will depends on the number of + files to delete. For filesystems where the cost of checking + for the existence of a file/directory and the actual delete operation + (for example: object stores) is high, the time to shutdown the JVM can be + significantly extended by over-use of this feature.
    • +
  • Connectivity problems with a remote filesystem may delay shutdown + further, and may cause the files to not be deleted.
    • + + @param f the path to delete. + @return true if deleteOnExit is successful, otherwise false. + @throws IOException IO failure]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Does not guarantee to return the List of files/directories status in a + sorted order. + @param f given path + @return the statuses of the files/directories in the given patch + @throws FileNotFoundException when the path does not exist + @throws IOException see specific implementation]]> + + + + + + + + + + + + + + + + + Does not guarantee to return the List of files/directories status in a + sorted order. + + @param f + a path name + @param filter + the user-supplied path filter + @return an array of FileStatus objects for the files under the given path + after applying the filter + @throws FileNotFoundException when the path does not exist + @throws IOException see specific implementation]]> + + + + + + + + + Does not guarantee to return the List of files/directories status in a + sorted order. + + @param files + a list of paths + @return a list of statuses for the files under the given paths after + applying the filter default Path filter + @throws FileNotFoundException when the path does not exist + @throws IOException see specific implementation]]> + + + + + + + + + + Does not guarantee to return the List of files/directories status in a + sorted order. + + @param files + a list of paths + @param filter + the user-supplied path filter + @return a list of statuses for the files under the given paths after + applying the filter + @throws FileNotFoundException when the path does not exist + @throws IOException see specific implementation]]> + + + + + + + Return all the files that match filePattern and are not checksum + files. Results are sorted by their names. + +

    + A filename pattern is composed of regular characters and + special pattern matching characters, which are: + +

    +
    +
    +

    +

    ? +
    Matches any single character. + +

    +

    * +
    Matches zero or more characters. + +

    +

    [abc] +
    Matches a single character from character set + {a,b,c}. + +

    +

    [a-b] +
    Matches a single character from the character range + {a...b}. Note that character a must be + lexicographically less than or equal to character b. + +

    +

    [^a] +
    Matches a single character that is not from character set or range + {a}. Note that the ^ character must occur + immediately to the right of the opening bracket. + +

    +

    \c +
    Removes (escapes) any special meaning of character c. + +

    +

    {ab,cd} +
    Matches a string from the string set {ab, cd} + +

    +

    {ab,c{de,fh}} +
    Matches a string from the string set {ab, cde, cfh} + +
    +
    +
    + + @param pathPattern a regular expression specifying a pth pattern + + @return an array of paths that match the path pattern + @throws IOException IO failure]]> +     +     + + + + + + + + + + + + + + f does not exist + @throws IOException If an I/O error occurred]]> + + + + + + + + + f does not exist + @throws IOException if any I/O error occurred]]> + + + + + + + + p does not exist + @throws IOException if any I/O error occurred]]> + + + + + + + + + + If the path is a directory, + if recursive is false, returns files in the directory; + if recursive is true, return files in the subtree rooted at the path. + If the path is a file, return the file's status and block locations. + + @param f is the path + @param recursive if the subdirectories need to be traversed recursively + + @return an iterator that traverses statuses of the files + + @throws FileNotFoundException when the path does not exist; + @throws IOException see specific implementation]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + undefined. + @throws IOException IO failure]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + describing modifications + @throws IOException if an ACL could not be modified + @throws UnsupportedOperationException if the operation is unsupported + (default outcome).]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to modify + @param name xattr name. + @param value xattr value. + @throws IOException IO failure + @throws UnsupportedOperationException if the operation is unsupported + (default outcome).]]> + + + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to modify + @param name xattr name. + @param value xattr value. + @param flag xattr set flag + @throws IOException IO failure + @throws UnsupportedOperationException if the operation is unsupported + (default outcome).]]> + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attribute + @param name xattr name. + @return byte[] xattr value. + @throws IOException IO failure + @throws UnsupportedOperationException if the operation is unsupported + (default outcome).]]> + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attributes + @return Map describing the XAttrs of the file or directory + @throws IOException IO failure + @throws UnsupportedOperationException if the operation is unsupported + (default outcome).]]> + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attributes + @param names XAttr names. + @return Map describing the XAttrs of the file or directory + @throws IOException IO failure + @throws UnsupportedOperationException if the operation is unsupported + (default outcome).]]> + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to get extended attributes + @return List of the XAttr names of the file or directory + @throws IOException IO failure + @throws UnsupportedOperationException if the operation is unsupported + (default outcome).]]> + + + + + + + + + Refer to the HDFS extended attributes user documentation for details. + + @param path Path to remove extended attribute + @param name xattr name + @throws IOException IO failure + @throws UnsupportedOperationException if the operation is unsupported + (default outcome).]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This is a default method which is intended to be overridden by + subclasses. The default implementation returns an empty storage statistics + object.

    + + @return The StorageStatistics for this FileSystem instance. + Will never be null.]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + All user code that may potentially use the Hadoop Distributed + File System should be written to use a FileSystem object or its + successor, {@link FileContext}. + +

    + The local implementation is {@link LocalFileSystem} and distributed + implementation is DistributedFileSystem. There are other implementations + for object stores and (outside the Apache Hadoop codebase), + third party filesystems. +

    + Notes +

      +
    1. The behaviour of the filesystem is + + specified in the Hadoop documentation. + However, the normative specification of the behavior of this class is + actually HDFS: if HDFS does not behave the way these Javadocs or + the specification in the Hadoop documentations define, assume that + the documentation is incorrect. +
      2. +
    2. The term {@code FileSystem} refers to an instance of this class.
      3. +
    3. The acronym "FS" is used as an abbreviation of FileSystem.
      4. +
    4. The term {@code filesystem} refers to the distributed/local filesystem + itself, rather than the class used to interact with it.
      5. +
    5. The term "file" refers to a file in the remote filesystem, + rather than instances of {@code java.io.File}.
      6. +
    ]]> +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + caller's environment variables to use + for expansion + @return String[] with absolute path to new jar in position 0 and + unexpanded wild card entry path in position 1 + @throws IOException if there is an I/O error while writing the jar file]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + FilterFileSystem contains + some other file system, which it uses as + its basic file system, possibly transforming + the data along the way or providing additional + functionality. The class FilterFileSystem + itself simply overrides all methods of + FileSystem with versions that + pass all requests to the contained file + system. Subclasses of FilterFileSystem + may further override some of these methods + and may also provide additional methods + and fields.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + -1 + if there is no more data because the end of the stream has been + reached]]> + + + + + + + + + + length bytes have been read. + + @param position position in the input stream to seek + @param buffer buffer into which data is read + @param offset offset into the buffer in which data is written + @param length the number of bytes to read + @throws IOException IO problems + @throws EOFException If the end of stream is reached while reading. + If an exception is thrown an undetermined number + of bytes in the buffer may have been written.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + path is invalid]]> + + + + + + + + + + + + + + + + + + + + + + + @return file]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + and the scheme is null, and the authority + is null. + + @return whether the path is absolute and the URI has no scheme nor + authority parts]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + true if and only if pathname + should be included]]> + + + + + + + + + + + + + + Warning: Not all filesystems satisfy the thread-safety requirement. + @param position position within file + @param buffer destination buffer + @param offset offset in the buffer + @param length number of bytes to read + @return actual number of bytes read; -1 means "none" + @throws IOException IO problems.]]> + + + + + + + + + + Warning: Not all filesystems satisfy the thread-safety requirement. + @param position position within file + @param buffer destination buffer + @param offset offset in the buffer + @param length number of bytes to read + @throws IOException IO problems. + @throws EOFException the end of the data was reached before + the read operation completed]]> + + + + + + + + Warning: Not all filesystems satisfy the thread-safety requirement. + @param position position within file + @param buffer destination buffer + @throws IOException IO problems. + @throws EOFException the end of the data was reached before + the read operation completed]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <----15----> <----15----> <----15----> <-------18-------> + QUOTA REMAINING_QUATA SPACE_QUOTA SPACE_QUOTA_REM FILE_NAME]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + XAttr is byte[], this class is to + covert byte[] to some kind of string representation or convert back. + String representation is convenient for display and input. For example + display in screen as shell response and json response, input as http + or shell parameter.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @return ftp]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + A {@link FileSystem} backed by an FTP client provided by Apache Commons Net. +

    ]]> +     +     + +     + + + + + + + + + + + + + + + + + + (cause==null ? null : cause.toString()) (which + typically contains the class and detail message of cause). + @param cause the cause (which is saved for later retrieval by the + {@link #getCause()} method). (A null value is + permitted, and indicates that the cause is nonexistent or + unknown.)]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + But for removeAcl operation it will be false. i.e. AclSpec should + not contain permissions.
    + Example: "user:foo,group:bar" + @return Returns list of {@link AclEntry} parsed]]> +     +     + + + + + + The expected format of ACL entries in the string parameter is the same + format produced by the {@link #toStringStable()} method. + + @param aclStr + String representation of an ACL.
    + Example: "user:foo:rw-" + @param includePermission + for setAcl operations this will be true. i.e. Acl should include + permissions.
    + But for removeAcl operation it will be false. i.e. Acl should not + contain permissions.
    + Example: "user:foo,group:bar,mask::" + @return Returns an {@link AclEntry} object]]> +     +     + + + + + + + + + +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + unmodifiable ordered list of all ACL entries]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + Recommended to use this API ONLY if client communicates with the old + NameNode, needs to pass the Permission for the path to get effective + permission, else use {@link AclStatus#getEffectivePermission(AclEntry)}. + @param entry AclEntry to get the effective action + @param permArg Permission for the path. However if the client is NOT + communicating with old namenode, then this argument will not have + any preference. + @return Returns the effective permission for the entry. + @throws IllegalArgumentException If the client communicating with old + namenode and permission is not passed as an argument.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + mode is invalid]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @return viewfs]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
  • /user -> hdfs://nnContainingUserDir/user +
  • /project/foo -> hdfs://nnProject1/projects/foo +
  • /project/bar -> hdfs://nnProject2/projects/bar +
  • /tmp -> hdfs://nnTmp/privateTmpForUserXXX + + + ViewFs is specified with the following URI: viewfs:/// +

    + To use viewfs one would typically set the default file system in the + config (i.e. fs.defaultFS < = viewfs:///) along with the + mount table config variables as described below. + +

    + ** Config variables to specify the mount table entries ** +

    + + The file system is initialized from the standard Hadoop config through + config variables. + See {@link FsConstants} for URI and Scheme constants; + See {@link Constants} for config var constants; + see {@link ConfigUtil} for convenient lib. + +

    + All the mount table config entries for view fs are prefixed by + fs.viewfs.mounttable. + For example the above example can be specified with the following + config variables: +

      +
    • fs.viewfs.mounttable.default.link./user= + hdfs://nnContainingUserDir/user +
    • fs.viewfs.mounttable.default.link./project/foo= + hdfs://nnProject1/projects/foo +
    • fs.viewfs.mounttable.default.link./project/bar= + hdfs://nnProject2/projects/bar +
    • fs.viewfs.mounttable.default.link./tmp= + hdfs://nnTmp/privateTmpForUserXXX +
    + + The default mount table (when no authority is specified) is + from config variables prefixed by fs.viewFs.mounttable.default + The authority component of a URI can be used to specify a different mount + table. For example, +
      +
    • viewfs://sanjayMountable/ +
    + is initialized from fs.viewFs.mounttable.sanjayMountable.* config variables. + +

    + **** Merge Mounts **** (NOTE: merge mounts are not implemented yet.) +

    + + One can also use "MergeMounts" to merge several directories (this is + sometimes called union-mounts or junction-mounts in the literature. + For example of the home directories are stored on say two file systems + (because they do not fit on one) then one could specify a mount + entry such as following merges two dirs: +

      +
    • /user -> hdfs://nnUser1/user,hdfs://nnUser2/user +
    + Such a mergeLink can be specified with the following config var where "," + is used as the separator for each of links to be merged: +
      +
    • fs.viewfs.mounttable.default.linkMerge./user= + hdfs://nnUser1/user,hdfs://nnUser1/user +
    + A special case of the merge mount is where mount table's root is merged + with the root (slash) of another file system: +
      +
    • fs.viewfs.mounttable.default.linkMergeSlash=hdfs://nn99/ +
    + In this cases the root of the mount table is merged with the root of + hdfs://nn99/ ]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Since these methods are often vendor- or device-specific, operators + may implement this interface in order to achieve fencing. +

    + Fencing is configured by the operator as an ordered list of methods to + attempt. Each method will be tried in turn, and the next in the list + will only be attempted if the previous one fails. See {@link NodeFencer} + for more information. +

    + If an implementation also implements {@link Configurable} then its + setConf method will be called upon instantiation.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + state (e.g ACTIVE/STANDBY) as well as + some additional information. + + @throws AccessControlException + if access is denied. + @throws IOException + if other errors happen + @see HAServiceStatus]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + hadoop.http.filter.initializers. + +

      +
    • StaticUserWebFilter - An authorization plugin that makes all +users a static configured user. +
    ]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + public class IntArrayWritable extends ArrayWritable { + public IntArrayWritable() { + super(IntWritable.class); + } + } + ]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + o is a ByteWritable with the same value.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + the class of the item + @param conf the configuration to store + @param item the object to be stored + @param keyName the name of the key to use + @throws IOException : forwards Exceptions from the underlying + {@link Serialization} classes.]]> + + + + + + + + + the class of the item + @param conf the configuration to use + @param keyName the name of the key to use + @param itemClass the class of the item + @return restored object + @throws IOException : forwards Exceptions from the underlying + {@link Serialization} classes.]]> + + + + + + + + + the class of the item + @param conf the configuration to use + @param items the objects to be stored + @param keyName the name of the key to use + @throws IndexOutOfBoundsException if the items array is empty + @throws IOException : forwards Exceptions from the underlying + {@link Serialization} classes.]]> + + + + + + + + + the class of the item + @param conf the configuration to use + @param keyName the name of the key to use + @param itemClass the class of the item + @return restored object + @throws IOException : forwards Exceptions from the underlying + {@link Serialization} classes.]]> + + + + + DefaultStringifier offers convenience methods to store/load objects to/from + the configuration. + + @param the class of the objects to stringify]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + o is a DoubleWritable with the same value.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + value argument is null or + its size is zero, the elementType argument must not be null. If + the argument value's size is bigger than zero, the argument + elementType is not be used. + + @param value + @param elementType]]> + + + + + value should not be null + or empty. + + @param value]]> + + + + + + + + + + + + + + value and elementType. If the value argument + is null or its size is zero, the elementType argument must not be + null. If the argument value's size is bigger than zero, the + argument elementType is not be used. + + @param value + @param elementType]]> + + + + + + + + + + + + + + + + + + + o is an EnumSetWritable with the same value, + or both are null.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + o is a FloatWritable with the same value.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + When two sequence files, which have same Key type but different Value + types, are mapped out to reduce, multiple Value types is not allowed. + In this case, this class can help you wrap instances with different types. +

    + +

    + Compared with ObjectWritable, this class is much more effective, + because ObjectWritable will append the class declaration as a String + into the output file in every Key-Value pair. +

    + +

    + Generic Writable implements {@link Configurable} interface, so that it will be + configured by the framework. The configuration is passed to the wrapped objects + implementing {@link Configurable} interface before deserialization. +

    + + how to use it:
    + 1. Write your own class, such as GenericObject, which extends GenericWritable.
    + 2. Implements the abstract method getTypes(), defines + the classes which will be wrapped in GenericObject in application. + Attention: this classes defined in getTypes() method, must + implement Writable interface. +

    + + The code looks like this: + 
    + public class GenericObject extends GenericWritable {
+ 
+   private static Class[] CLASSES = {
+               ClassType1.class, 
+               ClassType2.class,
+               ClassType3.class,
+               };
+
+   protected Class[] getTypes() {
+       return CLASSES;
+   }
+
+ }
+
    + + @since Nov 8, 2006]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + o is a IntWritable with the same value.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + closes the input and output streams + at the end. + + @param in InputStrem to read from + @param out OutputStream to write to + @param conf the Configuration object]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ignore any {@link Throwable} or + null pointers. Must only be used for cleanup in exception handlers. + + @param log the log to record problems to at debug level. Can be null. + @param closeables the objects to close + @deprecated use {@link #cleanupWithLogger(Logger, java.io.Closeable...)} + instead]]> + + + + + + + ignore any {@link Throwable} or + null pointers. Must only be used for cleanup in exception handlers. + + @param logger the log to record problems to at debug level. Can be null. + @param closeables the objects to close]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This is better than File#listDir because it does not ignore IOExceptions. + + @param dir The directory to list. + @param filter If non-null, the filter to use when listing + this directory. + @return The list of files in the directory. + + @throws IOException On I/O error]]> + + + + + + + + Borrowed from Uwe Schindler in LUCENE-5588 + @param fileToSync the file to fsync]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + o is a LongWritable with the same value.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + A map is a directory containing two files, the data file, + containing all keys and values in the map, and a smaller index + file, containing a fraction of the keys. The fraction is determined by + {@link Writer#getIndexInterval()}. + +

    The index file is read entirely into memory. Thus key implementations + should try to keep themselves small. + +

    Map files are created by adding entries in-order. To maintain a large + database, perform updates by copying the previous version of a database and + merging in a sorted change list, to create a new version of the database in + a new file. Sorting large change lists can be done with {@link + SequenceFile.Sorter}.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + o is an MD5Hash whose digest contains the + same values.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + className by first finding + it in the specified conf. If the specified conf is null, + try load it directly.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + A {@link Comparator} that operates directly on byte representations of + objects. +

    + @param + @see DeserializerComparator]]> +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + SequenceFiles are flat files consisting of binary key/value + pairs. + +

    SequenceFile provides {@link SequenceFile.Writer}, + {@link SequenceFile.Reader} and {@link Sorter} classes for writing, + reading and sorting respectively.

    + + There are three SequenceFile Writers based on the + {@link CompressionType} used to compress key/value pairs: +
      +
    1. + Writer : Uncompressed records. +
      2. +
    2. + RecordCompressWriter : Record-compressed files, only compress + values. +
      3. +
    3. + BlockCompressWriter : Block-compressed files, both keys & + values are collected in 'blocks' + separately and compressed. The size of + the 'block' is configurable. +
    + +

    The actual compression algorithm used to compress key and/or values can be + specified by using the appropriate {@link CompressionCodec}.

    + +

    The recommended way is to use the static createWriter methods + provided by the SequenceFile to chose the preferred format.

    + +

    The {@link SequenceFile.Reader} acts as the bridge and can read any of the + above SequenceFile formats.

    + +

    SequenceFile Formats

    + +

    Essentially there are 3 different formats for SequenceFiles + depending on the CompressionType specified. All of them share a + common header described below. + +

    +
      +
    • + version - 3 bytes of magic header SEQ, followed by 1 byte of actual + version number (e.g. SEQ4 or SEQ6) +
      • +
    • + keyClassName -key class +
      • +
    • + valueClassName - value class +
      • +
    • + compression - A boolean which specifies if compression is turned on for + keys/values in this file. +
      • +
    • + blockCompression - A boolean which specifies if block-compression is + turned on for keys/values in this file. +
      • +
    • + compression codec - CompressionCodec class which is used for + compression of keys and/or values (if compression is + enabled). +
      • +
    • + metadata - {@link Metadata} for this file. +
      • +
    • + sync - A sync marker to denote end of the header. +
      • +
    + +
    Uncompressed SequenceFile Format
    +
      +
    • + Header +
      • +
    • + Record +
        +
      • Record length
        • +
      • Key length
        • +
      • Key
        • +
      • Value
        • +
      +
      • +
    • + A sync-marker every few 100 bytes or so. +
      • +
    + +
    Record-Compressed SequenceFile Format
    +
      +
    • + Header +
      • +
    • + Record +
        +
      • Record length
        • +
      • Key length
        • +
      • Key
        • +
      • Compressed Value
        • +
      +
      • +
    • + A sync-marker every few 100 bytes or so. +
      • +
    + +
    Block-Compressed SequenceFile Format
    +
      +
    • + Header +
      • +
    • + Record Block +
        +
      • Uncompressed number of records in the block
        • +
      • Compressed key-lengths block-size
        • +
      • Compressed key-lengths block
        • +
      • Compressed keys block-size
        • +
      • Compressed keys block
        • +
      • Compressed value-lengths block-size
        • +
      • Compressed value-lengths block
        • +
      • Compressed values block-size
        • +
      • Compressed values block
        • +
      +
      • +
    • + A sync-marker every block. +
      • +
    + +

    The compressed blocks of key lengths and value lengths consist of the + actual lengths of individual keys/values encoded in ZeroCompressedInteger + format.

    + + @see CompressionCodec]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + o is a ShortWritable with the same value.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + the class of the objects to stringify]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + position. Note that this + method avoids using the converter or doing String instantiation + @return the Unicode scalar value at position or -1 + if the position is invalid or points to a + trailing byte]]> + + + + + + + + + + what in the backing + buffer, starting as position start. The starting + position is measured in bytes and the return value is in + terms of byte position in the buffer. The backing buffer is + not converted to a string for this operation. + @return byte position of the first occurence of the search + string in the UTF-8 buffer or -1 if not found]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Note: For performance reasons, this call does not clear the + underlying byte array that is retrievable via {@link #getBytes()}. + In order to free the byte-array memory, call {@link #set(byte[])} + with an empty byte array (For example, new byte[0]).]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + o is a Text with the same contents.]]> + + + + + + + + + + + + + + + + + + + + + + + + + replace is true, then + malformed input is replaced with the + substitution character, which is U+FFFD. Otherwise the + method throws a MalformedInputException.]]> + + + + + + + + + + + + + + + replace is true, then + malformed input is replaced with the + substitution character, which is U+FFFD. Otherwise the + method throws a MalformedInputException. + @return ByteBuffer: bytes stores at ByteBuffer.array() + and length is ByteBuffer.limit()]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + In + addition, it provides methods for string traversal without converting the + byte array to a string.

    Also includes utilities for + serializing/deserialing a string, coding/decoding a string, checking if a + byte array contains valid UTF8 code, calculating the length of an encoded + string.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This is useful when a class may evolve, so that instances written by the + old version of the class may still be processed by the new version. To + handle this situation, {@link #readFields(DataInput)} + implementations should catch {@link VersionMismatchException}.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + o is a VIntWritable with the same value.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + o is a VLongWritable with the same value.]]> + + + + + + + + + + + + + + + + + + + + + + + + out. + + @param out DataOuput to serialize this object into. + @throws IOException]]> + + + + + + + in. + +

    For efficiency, implementations should attempt to re-use storage in the + existing object where possible.

    + + @param in DataInput to deseriablize this object from. + @throws IOException]]> +     + + + Any key or value type in the Hadoop Map-Reduce + framework implements this interface.

    + +

    Implementations typically implement a static read(DataInput) + method which constructs a new instance, calls {@link #readFields(DataInput)} + and returns the instance.

    + +

    Example:

    + 

    +     public class MyWritable implements Writable {
+       // Some data     
+       private int counter;
+       private long timestamp;
+       
+       public void write(DataOutput out) throws IOException {
+         out.writeInt(counter);
+         out.writeLong(timestamp);
+       }
+       
+       public void readFields(DataInput in) throws IOException {
+         counter = in.readInt();
+         timestamp = in.readLong();
+       }
+       
+       public static MyWritable read(DataInput in) throws IOException {
+         MyWritable w = new MyWritable();
+         w.readFields(in);
+         return w;
+       }
+     }
+

    ]]> +     + + + + + + + + WritableComparables can be compared to each other, typically + via Comparators. Any type which is to be used as a + key in the Hadoop Map-Reduce framework should implement this + interface.

    + +

    Note that hashCode() is frequently used in Hadoop to partition + keys. It's important that your implementation of hashCode() returns the same + result across different instances of the JVM. Note also that the default + hashCode() implementation in Object does not + satisfy this property.

    + +

    Example:

    + 

    +     public class MyWritableComparable implements WritableComparable {
+       // Some data
+       private int counter;
+       private long timestamp;
+       
+       public void write(DataOutput out) throws IOException {
+         out.writeInt(counter);
+         out.writeLong(timestamp);
+       }
+       
+       public void readFields(DataInput in) throws IOException {
+         counter = in.readInt();
+         timestamp = in.readLong();
+       }
+       
+       public int compareTo(MyWritableComparable o) {
+         int thisValue = this.value;
+         int thatValue = o.value;
+         return (thisValue < thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
+       }
+
+       public int hashCode() {
+         final int prime = 31;
+         int result = 1;
+         result = prime * result + counter;
+         result = prime * result + (int) (timestamp ^ (timestamp >>> 32));
+         return result
+       }
+     }
+

    ]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + The default implementation reads the data into two {@link + WritableComparable}s (using {@link + Writable#readFields(DataInput)}, then calls {@link + #compare(WritableComparable,WritableComparable)}.]]> + + + + + + + The default implementation uses the natural ordering, calling {@link + Comparable#compareTo(Object)}.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This base implemenation uses the natural ordering. To define alternate + orderings, override {@link #compare(WritableComparable,WritableComparable)}. + +

    One may optimize compare-intensive operations by overriding + {@link #compare(byte[],int,int,byte[],int,int)}. Static utility methods are + provided to assist in optimized implementations of this method.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Enum type + @param in DataInput to read from + @param enumType Class type of Enum + @return Enum represented by String read from DataInput + @throws IOException]]> + + + + + + + + + + + + + + + + len number of bytes in input streamin + @param in input stream + @param len number of bytes to skip + @throws IOException when skipped less number of bytes]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CompressionCodec for which to get the + Compressor + @param conf the Configuration object which contains confs for creating or reinit the compressor + @return Compressor for the given + CompressionCodec from the pool or a new one]]> + + + + + + + + + CompressionCodec for which to get the + Decompressor + @return Decompressor for the given + CompressionCodec the pool or a new one]]> + + + + + + Compressor to be returned to the pool]]> + + + + + + Decompressor to be returned to the + pool]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Codec aliases are case insensitive. +

    + The code alias is the short class name (without the package name). + If the short class name ends with 'Codec', then there are two aliases for + the codec, the complete short class name and the short class name without + the 'Codec' ending. For example for the 'GzipCodec' codec class name the + alias are 'gzip' and 'gzipcodec'. + + @param codecName the canonical class name of the codec + @return the codec object]]> + + + + + + + Codec aliases are case insensitive. +

    + The code alias is the short class name (without the package name). + If the short class name ends with 'Codec', then there are two aliases for + the codec, the complete short class name and the short class name without + the 'Codec' ending. For example for the 'GzipCodec' codec class name the + alias are 'gzip' and 'gzipcodec'. + + @param codecName the canonical class name of the codec + @return the codec class]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Implementations are assumed to be buffered. This permits clients to + reposition the underlying input stream then call {@link #resetState()}, + without having to also synchronize client buffers.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + true indicating that more input data is required. + + @param b Input data + @param off Start offset + @param len Length]]> + + + + + true if the input data buffer is empty and + #setInput() should be called in order to provide more input.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + true if the end of the compressed + data output stream has been reached.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + true indicating that more input data is required. + (Both native and non-native versions of various Decompressors require + that the data passed in via b[] remain unmodified until + the caller is explicitly notified--via {@link #needsInput()}--that the + buffer may be safely modified. With this requirement, an extra + buffer-copy can be avoided.) + + @param b Input data + @param off Start offset + @param len Length]]> + + + + + true if the input data buffer is empty and + {@link #setInput(byte[], int, int)} should be called to + provide more input. + + @return true if the input data buffer is empty and + {@link #setInput(byte[], int, int)} should be called in + order to provide more input.]]> + + + + + + + + + + + + + true if a preset dictionary is needed for decompression. + @return true if a preset dictionary is needed for decompression]]> + + + + + true if the end of the decompressed + data output stream has been reached. Indicates a concatenated data stream + when finished() returns true and {@link #getRemaining()} + returns a positive value. finished() will be reset with the + {@link #reset()} method. + @return true if the end of the decompressed + data output stream has been reached.]]> + + + + + + + + + + + + + + true and getRemaining() returns a positive value. If + {@link #finished()} returns true and getRemaining() returns + a zero value, indicates that the end of data stream has been reached and + is not a concatenated data stream. + @return The number of bytes remaining in the compressed data buffer.]]> + + + + + true and {@link #getRemaining()} returns a positive value, + reset() is called before processing of the next data stream in the + concatenated data stream. {@link #finished()} will be reset and will + return false when reset() is called.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

  • "none" - No compression. +
  • "lzo" - LZO compression. +
  • "gz" - GZIP compression. + ]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
  • Block Compression. +
  • Named meta data blocks. +
  • Sorted or unsorted keys. +
  • Seek by key or by file offset. + + The memory footprint of a TFile includes the following: +
      +
    • Some constant overhead of reading or writing a compressed block. +
        +
      • Each compressed block requires one compression/decompression codec for + I/O. +
      • Temporary space to buffer the key. +
      • Temporary space to buffer the value (for TFile.Writer only). Values are + chunk encoded, so that we buffer at most one chunk of user data. By default, + the chunk buffer is 1MB. Reading chunked value does not require additional + memory. +
      +
    • TFile index, which is proportional to the total number of Data Blocks. + The total amount of memory needed to hold the index can be estimated as + (56+AvgKeySize)*NumBlocks. +
    • MetaBlock index, which is proportional to the total number of Meta + Blocks.The total amount of memory needed to hold the index for Meta Blocks + can be estimated as (40+AvgMetaBlockName)*NumMetaBlock. +
    +

    + The behavior of TFile can be customized by the following variables through + Configuration: +

      +
    • tfile.io.chunk.size: Value chunk size. Integer (in bytes). Default + to 1MB. Values of the length less than the chunk size is guaranteed to have + known value length in read time (See + {@link TFile.Reader.Scanner.Entry#isValueLengthKnown()}). +
    • tfile.fs.output.buffer.size: Buffer size used for + FSDataOutputStream. Integer (in bytes). Default to 256KB. +
    • tfile.fs.input.buffer.size: Buffer size used for + FSDataInputStream. Integer (in bytes). Default to 256KB. +
    +

    + Suggestions on performance optimization. +

      +
    • Minimum block size. We recommend a setting of minimum block size between + 256KB to 1MB for general usage. Larger block size is preferred if files are + primarily for sequential access. However, it would lead to inefficient random + access (because there are more data to decompress). Smaller blocks are good + for random access, but require more memory to hold the block index, and may + be slower to create (because we must flush the compressor stream at the + conclusion of each data block, which leads to an FS I/O flush). Further, due + to the internal caching in Compression codec, the smallest possible block + size would be around 20KB-30KB. +
    • The current implementation does not offer true multi-threading for + reading. The implementation uses FSDataInputStream seek()+read(), which is + shown to be much faster than positioned-read call in single thread mode. + However, it also means that if multiple threads attempt to access the same + TFile (using multiple scanners) simultaneously, the actual I/O is carried out + sequentially even if they access different DFS blocks. +
    • Compression codec. Use "none" if the data is not very compressable (by + compressable, I mean a compression ratio at least 2:1). Generally, use "lzo" + as the starting point for experimenting. "gz" overs slightly better + compression ratio over "lzo" but requires 4x CPU to compress and 2x CPU to + decompress, comparing to "lzo". +
    • File system buffering, if the underlying FSDataInputStream and + FSDataOutputStream is already adequately buffered; or if applications + reads/writes keys and values in large buffers, we can reduce the sizes of + input/output buffering in TFile layer by setting the configuration parameters + "tfile.fs.input.buffer.size" and "tfile.fs.output.buffer.size". +
    + + Some design rationale behind TFile can be found at Hadoop-3315.]]> + + + + + + + + + + + Utils#writeVLong(out, n). + + @param out + output stream + @param n + The integer to be encoded + @throws IOException + @see Utils#writeVLong(DataOutput, long)]]> + + + + + + + + +
  • if n in [-32, 127): encode in one byte with the actual value. + Otherwise, +
  • if n in [-20*2^8, 20*2^8): encode in two bytes: byte[0] = n/256 - 52; + byte[1]=n&0xff. Otherwise, +
  • if n IN [-16*2^16, 16*2^16): encode in three bytes: byte[0]=n/2^16 - + 88; byte[1]=(n>>8)&0xff; byte[2]=n&0xff. Otherwise, +
  • if n in [-8*2^24, 8*2^24): encode in four bytes: byte[0]=n/2^24 - 112; + byte[1] = (n>>16)&0xff; byte[2] = (n>>8)&0xff; byte[3]=n&0xff. Otherwise: +
  • if n in [-2^31, 2^31): encode in five bytes: byte[0]=-125; byte[1] = + (n>>24)&0xff; byte[2]=(n>>16)&0xff; byte[3]=(n>>8)&0xff; byte[4]=n&0xff; +
  • if n in [-2^39, 2^39): encode in six bytes: byte[0]=-124; byte[1] = + (n>>32)&0xff; byte[2]=(n>>24)&0xff; byte[3]=(n>>16)&0xff; + byte[4]=(n>>8)&0xff; byte[5]=n&0xff +
  • if n in [-2^47, 2^47): encode in seven bytes: byte[0]=-123; byte[1] = + (n>>40)&0xff; byte[2]=(n>>32)&0xff; byte[3]=(n>>24)&0xff; + byte[4]=(n>>16)&0xff; byte[5]=(n>>8)&0xff; byte[6]=n&0xff; +
  • if n in [-2^55, 2^55): encode in eight bytes: byte[0]=-122; byte[1] = + (n>>48)&0xff; byte[2] = (n>>40)&0xff; byte[3]=(n>>32)&0xff; + byte[4]=(n>>24)&0xff; byte[5]=(n>>16)&0xff; byte[6]=(n>>8)&0xff; + byte[7]=n&0xff; +
  • if n in [-2^63, 2^63): encode in nine bytes: byte[0]=-121; byte[1] = + (n>>54)&0xff; byte[2] = (n>>48)&0xff; byte[3] = (n>>40)&0xff; + byte[4]=(n>>32)&0xff; byte[5]=(n>>24)&0xff; byte[6]=(n>>16)&0xff; + byte[7]=(n>>8)&0xff; byte[8]=n&0xff; + + + @param out + output stream + @param n + the integer number + @throws IOException]]> + + + + + + + (int)Utils#readVLong(in). + + @param in + input stream + @return the decoded integer + @throws IOException + + @see Utils#readVLong(DataInput)]]> + + + + + + + +
  • if (FB >= -32), return (long)FB; +
  • if (FB in [-72, -33]), return (FB+52)<<8 + NB[0]&0xff; +
  • if (FB in [-104, -73]), return (FB+88)<<16 + (NB[0]&0xff)<<8 + + NB[1]&0xff; +
  • if (FB in [-120, -105]), return (FB+112)<<24 + (NB[0]&0xff)<<16 + + (NB[1]&0xff)<<8 + NB[2]&0xff; +
  • if (FB in [-128, -121]), return interpret NB[FB+129] as a signed + big-endian integer. + + @param in + input stream + @return the decoded long integer. + @throws IOException]]> + + + + + + + + + + + + + + + + + + + + + + + + Type of the input key. + @param list + The list + @param key + The input key. + @param cmp + Comparator for the key. + @return The index to the desired element if it exists; or list.size() + otherwise.]]> + + + + + + + + + Type of the input key. + @param list + The list + @param key + The input key. + @param cmp + Comparator for the key. + @return The index to the desired element if it exists; or list.size() + otherwise.]]> + + + + + + + + Type of the input key. + @param list + The list + @param key + The input key. + @return The index to the desired element if it exists; or list.size() + otherwise.]]> + + + + + + + + Type of the input key. + @param list + The list + @param key + The input key. + @return The index to the desired element if it exists; or list.size() + otherwise.]]> + + + + + + + + + + + + + + + + + An experimental {@link Serialization} for Java {@link Serializable} classes. +

    + @see JavaSerializationComparator]]> +     +     + + + + + + + + + A {@link RawComparator} that uses a {@link JavaSerialization} + {@link Deserializer} to deserialize objects that are then compared via + their {@link Comparable} interfaces. +

    + @param + @see JavaSerialization]]> +     +     + + + + + + + + + + + + + +This package provides a mechanism for using different serialization frameworks +in Hadoop. The property "io.serializations" defines a list of +{@link org.apache.hadoop.io.serializer.Serialization}s that know how to create +{@link org.apache.hadoop.io.serializer.Serializer}s and +{@link org.apache.hadoop.io.serializer.Deserializer}s. +

    + +

    +To add a new serialization framework write an implementation of +{@link org.apache.hadoop.io.serializer.Serialization} and add its name to the +"io.serializations" property. +

    ]]> +     +     + + + + + + + + + + + + + + + + + + + avro.reflect.pkgs or implement + {@link AvroReflectSerializable} interface.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + +This package provides Avro serialization in Hadoop. This can be used to +serialize/deserialize Avro types in Hadoop. +

    + +

    +Use {@link org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization} for +serialization of classes generated by Avro's 'specific' compiler. +

    + +

    +Use {@link org.apache.hadoop.io.serializer.avro.AvroReflectSerialization} for +other classes. +{@link org.apache.hadoop.io.serializer.avro.AvroReflectSerialization} work for +any class which is either in the package list configured via +{@link org.apache.hadoop.io.serializer.avro.AvroReflectSerialization#AVRO_REFLECT_PACKAGES} +or implement {@link org.apache.hadoop.io.serializer.avro.AvroReflectSerializable} +interface. +

    ]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + +The API is abstract so that it can be implemented on top of +a variety of metrics client libraries. The choice of +client library is a configuration option, and different +modules within the same application can use +different metrics implementation libraries. +

    +Sub-packages: +

    +
    org.apache.hadoop.metrics.spi
    +
    The abstract Server Provider Interface package. Those wishing to + integrate the metrics API with a particular metrics client library should + extend this package.
    + +
    org.apache.hadoop.metrics.file
    +
    An implementation package which writes the metric data to + a file, or sends it to the standard output stream.
    + +
    org.apache.hadoop.metrics.ganglia
    +
    An implementation package which sends metric data to + Ganglia.
    +
    + +

    Introduction to the Metrics API

    + +Here is a simple example of how to use this package to report a single +metric value: +
    +    private ContextFactory contextFactory = ContextFactory.getFactory();
+    
+    void reportMyMetric(float myMetric) {
+        MetricsContext myContext = contextFactory.getContext("myContext");
+        MetricsRecord myRecord = myContext.getRecord("myRecord");
+        myRecord.setMetric("myMetric", myMetric);
+        myRecord.update();
+    }
+
    + +In this example there are three names: +
    +
    myContext
    +
    The context name will typically identify either the application, or else a + module within an application or library.
    + +
    myRecord
    +
    The record name generally identifies some entity for which a set of + metrics are to be reported. For example, you could have a record named + "cacheStats" for reporting a number of statistics relating to the usage of + some cache in your application.
    + +
    myMetric
    +
    This identifies a particular metric. For example, you might have metrics + named "cache_hits" and "cache_misses". +
    +
    + +

    Tags

    + +In some cases it is useful to have multiple records with the same name. For +example, suppose that you want to report statistics about each disk on a computer. +In this case, the record name would be something like "diskStats", but you also +need to identify the disk which is done by adding a tag to the record. +The code could look something like this: +
    +    private MetricsRecord diskStats =
+            contextFactory.getContext("myContext").getRecord("diskStats");
+            
+    void reportDiskMetrics(String diskName, float diskBusy, float diskUsed) {
+        diskStats.setTag("diskName", diskName);
+        diskStats.setMetric("diskBusy", diskBusy);
+        diskStats.setMetric("diskUsed", diskUsed);
+        diskStats.update();
+    }
+
    + +

    Buffering and Callbacks

    + +Data is not sent immediately to the metrics system when +MetricsRecord.update() is called. Instead it is stored in an +internal table, and the contents of the table are sent periodically. +This can be important for two reasons: +
      +
    1. It means that a programmer is free to put calls to this API in an + inner loop, since updates can be very frequent without slowing down + the application significantly.
      2. +
    2. Some implementations can gain efficiency by combining many metrics + into a single UDP message.
      3. +
    + +The API provides a timer-based callback via the +registerUpdater() method. The benefit of this +versus using java.util.Timer is that the callbacks will be done +immediately before sending the data, making the data as current as possible. + +

    Configuration

    + +It is possible to programmatically examine and modify configuration data +before creating a context, like this: +
    +    ContextFactory factory = ContextFactory.getFactory();
+    ... examine and/or modify factory attributes ...
+    MetricsContext context = factory.getContext("myContext");
+
    +The factory attributes can be examined and modified using the following +ContextFactorymethods: +
      +
    • Object getAttribute(String attributeName)
      • +
    • String[] getAttributeNames()
      • +
    • void setAttribute(String name, Object value)
      • +
    • void removeAttribute(attributeName)
      • +
    + +

    +ContextFactory.getFactory() initializes the factory attributes by +reading the properties file hadoop-metrics.properties if it exists +on the class path. + +

    +A factory attribute named: +

    +contextName.class
+
    +should have as its value the fully qualified name of the class to be +instantiated by a call of the CodeFactory method +getContext(contextName). If this factory attribute is not +specified, the default is to instantiate +org.apache.hadoop.metrics.file.FileContext. + +

    +Other factory attributes are specific to a particular implementation of this +API and are documented elsewhere. For example, configuration attributes for +the file and Ganglia implementations can be found in the javadoc for +their respective packages.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Implementation of the metrics package that sends metric data to +Ganglia. +Programmers should not normally need to use this package directly. Instead +they should use org.hadoop.metrics. + +

    +These are the implementation specific factory attributes +(See ContextFactory.getFactory()): + +

    +
    contextName.servers
    +
    Space and/or comma separated sequence of servers to which UDP + messages should be sent.
    + +
    contextName.period
    +
    The period in seconds on which the metric data is sent to the + server(s).
    + +
    contextName.multicast
    +
    Enable multicast for Ganglia
    + +
    contextName.multicast.ttl
    +
    TTL for multicast packets
    + +
    contextName.units.recordName.metricName
    +
    The units for the specified metric in the specified record.
    + +
    contextName.slope.recordName.metricName
    +
    The slope for the specified metric in the specified record.
    + +
    contextName.tmax.recordName.metricName
    +
    The tmax for the specified metric in the specified record.
    + +
    contextName.dmax.recordName.metricName
    +
    The dmax for the specified metric in the specified record.
    + +
    ]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + contextName.tableName. The returned map consists of + those attributes with the contextName and tableName stripped off.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + recordName. + Throws an exception if the metrics implementation is configured with a fixed + set of record names and recordName is not in that set. + + @param recordName the name of the record + @throws MetricsException if recordName conflicts with configuration data]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This class implements the internal table of metric data, and the timer + on which data is to be sent to the metrics system. Subclasses must + override the abstract emitRecord method in order to transmit + the data.

    + + @deprecated Use org.apache.hadoop.metrics2 package instead.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + update + and remove(). + + @deprecated Use {@link org.apache.hadoop.metrics2.impl.MetricsRecordImpl} + instead.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + hostname or hostname:port. If + the specs string is null, defaults to localhost:defaultPort. + + @return a list of InetSocketAddress objects.]]> + + + + + + + + + org.apache.hadoop.metrics.file and +org.apache.hadoop.metrics.ganglia.

    + +Plugging in an implementation involves writing a concrete subclass of +AbstractMetricsContext. The subclass should get its + configuration information using the getAttribute(attributeName) + method.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Implementations of this interface consume the {@link MetricsRecord} generated + from {@link MetricsSource}. It registers with {@link MetricsSystem} which + periodically pushes the {@link MetricsRecord} to the sink using + {@link #putMetrics(MetricsRecord)} method. If the implementing class also + implements {@link Closeable}, then the MetricsSystem will close the sink when + it is stopped.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + the actual type of the source object + @param source object to register + @return the source object + @exception MetricsException]]> + + + + + + + + the actual type of the source object + @param source object to register + @param name of the source. Must be unique or null (then extracted from + the annotations of the source object.) + @param desc the description of the source (or null. See above.) + @return the source object + @exception MetricsException]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + (aggregate). + Filter out entries that don't have at least minSamples. + + @return a map of peer DataNode Id to the average latency to that + node seen over the measurement period.]]> + + + + + This class maintains a group of rolling average metrics. It implements the + algorithm of rolling average, i.e. a number of sliding windows are kept to + roll over and evict old subsets of samples. Each window has a subset of + samples in a stream, where sub-sum and sub-total are collected. All sub-sums + and sub-totals in all windows will be aggregated to final-sum and final-total + used to compute final average, which is called rolling average. +

    ]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This class is a metrics sink that uses + {@link org.apache.hadoop.fs.FileSystem} to write the metrics logs. Every + roll interval a new directory will be created under the path specified by the + basepath property. All metrics will be logged to a file in the + current interval's directory in a file named <hostname>.log, where + <hostname> is the name of the host on which the metrics logging + process is running. The base path is set by the + <prefix>.sink.<instance>.basepath property. The + time zone used to create the current interval's directory name is GMT. If + the basepath property isn't specified, it will default to + "/tmp", which is the temp directory on whatever default file + system is configured for the cluster.

    + +

    The <prefix>.sink.<instance>.ignore-error + property controls whether an exception is thrown when an error is encountered + writing a log file. The default value is true. When set to + false, file errors are quietly swallowed.

    + +

    The roll-interval property sets the amount of time before + rolling the directory. The default value is 1 hour. The roll interval may + not be less than 1 minute. The property's value should be given as + number unit, where number is an integer value, and + unit is a valid unit. Valid units are minute, hour, + and day. The units are case insensitive and may be abbreviated or + plural. If no units are specified, hours are assumed. For example, + "2", "2h", "2 hour", and + "2 hours" are all valid ways to specify two hours.

    + +

    The roll-offset-interval-millis property sets the upper + bound on a random time interval (in milliseconds) that is used to delay + before the initial roll. All subsequent rolls will happen an integer + number of roll intervals after the initial roll, hence retaining the original + offset. The purpose of this property is to insert some variance in the roll + times so that large clusters using this sink on every node don't cause a + performance impact on HDFS by rolling simultaneously. The default value is + 30000 (30s). When writing to HDFS, as a rule of thumb, the roll offset in + millis should be no less than the number of sink instances times 5. + +

    The primary use of this class is for logging to HDFS. As it uses + {@link org.apache.hadoop.fs.FileSystem} to access the target file system, + however, it can be used to write to the local file system, Amazon S3, or any + other supported file system. The base path for the sink will determine the + file system used. An unqualified path will write to the default file system + set by the configuration.

    + +

    Not all file systems support the ability to append to files. In file + systems without the ability to append to files, only one writer can write to + a file at a time. To allow for concurrent writes from multiple daemons on a + single host, the source property is used to set unique headers + for the log files. The property should be set to the name of + the source daemon, e.g. namenode. The value of the + source property should typically be the same as the property's + prefix. If this property is not set, the source is taken to be + unknown.

    + +

    Instead of appending to an existing file, by default the sink + will create a new file with a suffix of ".<n>&quet;, where + n is the next lowest integer that isn't already used in a file name, + similar to the Hadoop daemon logs. NOTE: the file with the highest + sequence number is the newest file, unlike the Hadoop daemon logs.

    + +

    For file systems that allow append, the sink supports appending to the + existing file instead. If the allow-append property is set to + true, the sink will instead append to the existing file on file systems that + support appends. By default, the allow-append property is + false.

    + +

    Note that when writing to HDFS with allow-append set to true, + there is a minimum acceptable number of data nodes. If the number of data + nodes drops below that minimum, the append will succeed, but reading the + data will fail with an IOException in the DataStreamer class. The minimum + number of data nodes required for a successful append is generally 2 or + 3.

    + +

    Note also that when writing to HDFS, the file size information is not + updated until the file is closed (at the end of the interval) even though + the data is being written successfully. This is a known HDFS limitation that + exists because of the performance cost of updating the metadata. See + HDFS-5478.

    + +

    When using this sink in a secure (Kerberos) environment, two additional + properties must be set: keytab-key and + principal-key. keytab-key should contain the key by + which the keytab file can be found in the configuration, for example, + yarn.nodemanager.keytab. principal-key should + contain the key by which the principal can be found in the configuration, + for example, yarn.nodemanager.principal.]]> + + + + + + + + + + + + + + + + + + + + + + + + + CollectD StatsD plugin). +
    + To configure this plugin, you will need to add the following + entries to your hadoop-metrics2.properties file: +
    + 

    + *.sink.statsd.class=org.apache.hadoop.metrics2.sink.StatsDSink
+ [prefix].sink.statsd.server.host=
+ [prefix].sink.statsd.server.port=
+ [prefix].sink.statsd.skip.hostname=true|false (optional)
+ [prefix].sink.statsd.service.name=NameNode (name you want for service)
+
    ]]> +     +     + +     + + + + + + + + + + + + + + + ,name=" + Where the and are the supplied parameters + + @param serviceName + @param nameName + @param theMbean - the MBean to register + @return the named used to register the MBean]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + hostname or hostname:port. If + the specs string is null, defaults to localhost:defaultPort. + + @param specs server specs (see description) + @param defaultPort the default port if not specified + @return a list of InetSocketAddress objects.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This method is used when parts of Hadoop need know whether to apply + single rack vs multi-rack policies, such as during block placement. + Such algorithms behave differently if they are on multi-switch systems. +

    + + @return true if the mapping thinks that it is on a single switch]]> +     +     + + + + + + + + + + + + + + + + + This predicate simply assumes that all mappings not derived from + this class are multi-switch. + @param mapping the mapping to query + @return true if the base class says it is single switch, or the mapping + is not derived from this class.]]> + + + + It is not mandatory to + derive {@link DNSToSwitchMapping} implementations from it, but it is strongly + recommended, as it makes it easy for the Hadoop developers to add new methods + to this base class that are automatically picked up by all implementations. +

    + + This class does not extend the Configured + base class, and should not be changed to do so, as it causes problems + for subclasses. The constructor of the Configured calls + the {@link #setConf(Configuration)} method, which will call into the + subclasses before they have been fully constructed.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + If a name cannot be resolved to a rack, the implementation + should return {@link NetworkTopology#DEFAULT_RACK}. This + is what the bundled implementations do, though it is not a formal requirement + + @param names the list of hosts to resolve (can be empty) + @return list of resolved network paths. + If names is empty, the returned list is also empty]]> + + + + + + + + + + + + + + + + + + + + + + + + Calling {@link #setConf(Configuration)} will trigger a + re-evaluation of the configuration settings and so be used to + set up the mapping script.]]> + + + + + + + + + + + + + + + + + + + + + This will get called in the superclass constructor, so a check is needed + to ensure that the raw mapping is defined before trying to relaying a null + configuration. + @param conf]]> + + + + + + + + + + It contains a static class RawScriptBasedMapping that performs + the work: reading the configuration parameters, executing any defined + script, handling errors and such like. The outer + class extends {@link CachedDNSToSwitchMapping} to cache the delegated + queries. +

    + This DNS mapper's {@link #isSingleSwitch()} predicate returns + true if and only if a script is defined.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Simple {@link DNSToSwitchMapping} implementation that reads a 2 column text + file. The columns are separated by whitespace. The first column is a DNS or + IP address and the second column specifies the rack where the address maps. +

    +

    + This class uses the configuration parameter {@code + net.topology.table.file.name} to locate the mapping file. +

    +

    + Calls to {@link #resolve(List)} will look up the address as defined in the + mapping file. If no entry corresponding to the address is found, the value + {@code /default-rack} is returned. +

    ]]> +     +     + + + + +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + =} getCount(). + @param newCapacity The new capacity in bytes.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + Index idx = startVector(...); + while (!idx.done()) { + .... // read element of a vector + idx.incr(); + } + + + @deprecated Replaced by Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + (DEPRECATED) Hadoop record I/O contains classes and a record description language + translator for simplifying serialization and deserialization of records in a + language-neutral manner. +

    + +

    + DEPRECATED: Replaced by Avro. +

    + +

    Introduction

    + + Software systems of any significant complexity require mechanisms for data +interchange with the outside world. These interchanges typically involve the +marshaling and unmarshaling of logical units of data to and from data streams +(files, network connections, memory buffers etc.). Applications usually have +some code for serializing and deserializing the data types that they manipulate +embedded in them. The work of serialization has several features that make +automatic code generation for it worthwhile. Given a particular output encoding +(binary, XML, etc.), serialization of primitive types and simple compositions +of primitives (structs, vectors etc.) is a very mechanical task. Manually +written serialization code can be susceptible to bugs especially when records +have a large number of fields or a record definition changes between software +versions. Lastly, it can be very useful for applications written in different +programming languages to be able to share and interchange data. This can be +made a lot easier by describing the data records manipulated by these +applications in a language agnostic manner and using the descriptions to derive +implementations of serialization in multiple target languages. + +This document describes Hadoop Record I/O, a mechanism that is aimed +at +
      +
    • enabling the specification of simple serializable data types (records) +
    • enabling the generation of code in multiple target languages for +marshaling and unmarshaling such types +
    • providing target language specific support that will enable application +programmers to incorporate generated code into their applications +
    + +The goals of Hadoop Record I/O are similar to those of mechanisms such as XDR, +ASN.1, PADS and ICE. While these systems all include a DDL that enables +the specification of most record types, they differ widely in what else they +focus on. The focus in Hadoop Record I/O is on data marshaling and +multi-lingual support. We take a translator-based approach to serialization. +Hadoop users have to describe their data in a simple data description +language. The Hadoop DDL translator rcc generates code that users +can invoke in order to read/write their data from/to simple stream +abstractions. Next we list explicitly some of the goals and non-goals of +Hadoop Record I/O. + + +

    Goals

    + +
      +
    • Support for commonly used primitive types. Hadoop should include as +primitives commonly used builtin types from programming languages we intend to +support. + +
    • Support for common data compositions (including recursive compositions). +Hadoop should support widely used composite types such as structs and +vectors. + +
    • Code generation in multiple target languages. Hadoop should be capable of +generating serialization code in multiple target languages and should be +easily extensible to new target languages. The initial target languages are +C++ and Java. + +
    • Support for generated target languages. Hadooop should include support +in the form of headers, libraries, packages for supported target languages +that enable easy inclusion and use of generated code in applications. + +
    • Support for multiple output encodings. Candidates include +packed binary, comma-separated text, XML etc. + +
    • Support for specifying record types in a backwards/forwards compatible +manner. This will probably be in the form of support for optional fields in +records. This version of the document does not include a description of the +planned mechanism, we intend to include it in the next iteration. + +
    + +

    Non-Goals

    + +
      +
    • Serializing existing arbitrary C++ classes. +
    • Serializing complex data structures such as trees, linked lists etc. +
    • Built-in indexing schemes, compression, or check-sums. +
    • Dynamic construction of objects from an XML schema. +
    + +The remainder of this document describes the features of Hadoop record I/O +in more detail. Section 2 describes the data types supported by the system. +Section 3 lays out the DDL syntax with some examples of simple records. +Section 4 describes the process of code generation with rcc. Section 5 +describes target language mappings and support for Hadoop types. We include a +fairly complete description of C++ mappings with intent to include Java and +others in upcoming iterations of this document. The last section talks about +supported output encodings. + + +

    Data Types and Streams

    + +This section describes the primitive and composite types supported by Hadoop. +We aim to support a set of types that can be used to simply and efficiently +express a wide range of record types in different programming languages. + +

    Primitive Types

    + +For the most part, the primitive types of Hadoop map directly to primitive +types in high level programming languages. Special cases are the +ustring (a Unicode string) and buffer types, which we believe +find wide use and which are usually implemented in library code and not +available as language built-ins. Hadoop also supplies these via library code +when a target language built-in is not present and there is no widely +adopted "standard" implementation. The complete list of primitive types is: + +
      +
    • byte: An 8-bit unsigned integer. +
    • boolean: A boolean value. +
    • int: A 32-bit signed integer. +
    • long: A 64-bit signed integer. +
    • float: A single precision floating point number as described by + IEEE-754. +
    • double: A double precision floating point number as described by + IEEE-754. +
    • ustring: A string consisting of Unicode characters. +
    • buffer: An arbitrary sequence of bytes. +
    + + +

    Composite Types

    +Hadoop supports a small set of composite types that enable the description +of simple aggregate types and containers. A composite type is serialized +by sequentially serializing it constituent elements. The supported +composite types are: + +
      + +
    • record: An aggregate type like a C-struct. This is a list of +typed fields that are together considered a single unit of data. A record +is serialized by sequentially serializing its constituent fields. In addition +to serialization a record has comparison operations (equality and less-than) +implemented for it, these are defined as memberwise comparisons. + +
    • vector: A sequence of entries of the same data type, primitive +or composite. + +
    • map: An associative container mapping instances of a key type to +instances of a value type. The key and value types may themselves be primitive +or composite types. + +
    + +

    Streams

    + +Hadoop generates code for serializing and deserializing record types to +abstract streams. For each target language Hadoop defines very simple input +and output stream interfaces. Application writers can usually develop +concrete implementations of these by putting a one method wrapper around +an existing stream implementation. + + +

    DDL Syntax and Examples

    + +We now describe the syntax of the Hadoop data description language. This is +followed by a few examples of DDL usage. + +

    Hadoop DDL Syntax

    + +
    
+recfile = *include module *record
+include = "include" path
+path = (relative-path / absolute-path)
+module = "module" module-name
+module-name = name *("." name)
+record := "class" name "{" 1*(field) "}"
+field := type name ";"
+name :=  ALPHA (ALPHA / DIGIT / "_" )*
+type := (ptype / ctype)
+ptype := ("byte" / "boolean" / "int" |
+          "long" / "float" / "double"
+          "ustring" / "buffer")
+ctype := (("vector" "<" type ">") /
+          ("map" "<" type "," type ">" ) ) / name)
+
    + +A DDL file describes one or more record types. It begins with zero or +more include declarations, a single mandatory module declaration +followed by zero or more class declarations. The semantics of each of +these declarations are described below: + +
      + +
    • include: An include declaration specifies a DDL file to be +referenced when generating code for types in the current DDL file. Record types +in the current compilation unit may refer to types in all included files. +File inclusion is recursive. An include does not trigger code +generation for the referenced file. + +
    • module: Every Hadoop DDL file must have a single module +declaration that follows the list of includes and precedes all record +declarations. A module declaration identifies a scope within which +the names of all types in the current file are visible. Module names are +mapped to C++ namespaces, Java packages etc. in generated code. + +
    • class: Records types are specified through class +declarations. A class declaration is like a Java class declaration. +It specifies a named record type and a list of fields that constitute records +of the type. Usage is illustrated in the following examples. + +
    + +

    Examples

    + +
      +
    • A simple DDL file links.jr with just one record declaration. +
      
+module links {
+    class Link {
+        ustring URL;
+        boolean isRelative;
+        ustring anchorText;
+    };
+}
+
      + +
    • A DDL file outlinks.jr which includes another +
      
+include "links.jr"
+
+module outlinks {
+    class OutLinks {
+        ustring baseURL;
+        vector outLinks;
+    };
+}
+
      +
    + +

    Code Generation

    + +The Hadoop translator is written in Java. Invocation is done by executing a +wrapper shell script named named rcc. It takes a list of +record description files as a mandatory argument and an +optional language argument (the default is Java) --language or +-l. Thus a typical invocation would look like: +
    
+$ rcc -l C++  ...
+
    + + +

    Target Language Mappings and Support

    + +For all target languages, the unit of code generation is a record type. +For each record type, Hadoop generates code for serialization and +deserialization, record comparison and access to record members. + +

    C++

    + +Support for including Hadoop generated C++ code in applications comes in the +form of a header file recordio.hh which needs to be included in source +that uses Hadoop types and a library librecordio.a which applications need +to be linked with. The header declares the Hadoop C++ namespace which defines +appropriate types for the various primitives, the basic interfaces for +records and streams and enumerates the supported serialization encodings. +Declarations of these interfaces and a description of their semantics follow: + +
    
+namespace hadoop {
+
+  enum RecFormat { kBinary, kXML, kCSV };
+
+  class InStream {
+  public:
+    virtual ssize_t read(void *buf, size_t n) = 0;
+  };
+
+  class OutStream {
+  public:
+    virtual ssize_t write(const void *buf, size_t n) = 0;
+  };
+
+  class IOError : public runtime_error {
+  public:
+    explicit IOError(const std::string& msg);
+  };
+
+  class IArchive;
+  class OArchive;
+
+  class RecordReader {
+  public:
+    RecordReader(InStream& in, RecFormat fmt);
+    virtual ~RecordReader(void);
+
+    virtual void read(Record& rec);
+  };
+
+  class RecordWriter {
+  public:
+    RecordWriter(OutStream& out, RecFormat fmt);
+    virtual ~RecordWriter(void);
+
+    virtual void write(Record& rec);
+  };
+
+
+  class Record {
+  public:
+    virtual std::string type(void) const = 0;
+    virtual std::string signature(void) const = 0;
+  protected:
+    virtual bool validate(void) const = 0;
+
+    virtual void
+    serialize(OArchive& oa, const std::string& tag) const = 0;
+
+    virtual void
+    deserialize(IArchive& ia, const std::string& tag) = 0;
+  };
+}
+
    + +
      + +
    • RecFormat: An enumeration of the serialization encodings supported +by this implementation of Hadoop. + +
    • InStream: A simple abstraction for an input stream. This has a +single public read method that reads n bytes from the stream into +the buffer buf. Has the same semantics as a blocking read system +call. Returns the number of bytes read or -1 if an error occurs. + +
    • OutStream: A simple abstraction for an output stream. This has a +single write method that writes n bytes to the stream from the +buffer buf. Has the same semantics as a blocking write system +call. Returns the number of bytes written or -1 if an error occurs. + +
    • RecordReader: A RecordReader reads records one at a time from +an underlying stream in a specified record format. The reader is instantiated +with a stream and a serialization format. It has a read method that +takes an instance of a record and deserializes the record from the stream. + +
    • RecordWriter: A RecordWriter writes records one at a +time to an underlying stream in a specified record format. The writer is +instantiated with a stream and a serialization format. It has a +write method that takes an instance of a record and serializes the +record to the stream. + +
    • Record: The base class for all generated record types. This has two +public methods type and signature that return the typename and the +type signature of the record. + +
    + +Two files are generated for each record file (note: not for each record). If a +record file is named "name.jr", the generated files are +"name.jr.cc" and "name.jr.hh" containing serialization +implementations and record type declarations respectively. + +For each record in the DDL file, the generated header file will contain a +class definition corresponding to the record type, method definitions for the +generated type will be present in the '.cc' file. The generated class will +inherit from the abstract class hadoop::Record. The DDL files +module declaration determines the namespace the record belongs to. +Each '.' delimited token in the module declaration results in the +creation of a namespace. For instance, the declaration module docs.links +results in the creation of a docs namespace and a nested +docs::links namespace. In the preceding examples, the Link class +is placed in the links namespace. The header file corresponding to +the links.jr file will contain: + +
    
+namespace links {
+  class Link : public hadoop::Record {
+    // ....
+  };
+};
+
    + +Each field within the record will cause the generation of a private member +declaration of the appropriate type in the class declaration, and one or more +acccessor methods. The generated class will implement the serialize and +deserialize methods defined in hadoop::Record+. It will also +implement the inspection methods type and signature from +hadoop::Record. A default constructor and virtual destructor will also +be generated. Serialization code will read/write records into streams that +implement the hadoop::InStream and the hadoop::OutStream interfaces. + +For each member of a record an accessor method is generated that returns +either the member or a reference to the member. For members that are returned +by value, a setter method is also generated. This is true for primitive +data members of the types byte, int, long, boolean, float and +double. For example, for a int field called MyField the folowing +code is generated. + +
    
+...
+private:
+  int32_t mMyField;
+  ...
+public:
+  int32_t getMyField(void) const {
+    return mMyField;
+  };
+
+  void setMyField(int32_t m) {
+    mMyField = m;
+  };
+  ...
+
    + +For a ustring or buffer or composite field. The generated code +only contains accessors that return a reference to the field. A const +and a non-const accessor are generated. For example: + +
    
+...
+private:
+  std::string mMyBuf;
+  ...
+public:
+
+  std::string& getMyBuf() {
+    return mMyBuf;
+  };
+
+  const std::string& getMyBuf() const {
+    return mMyBuf;
+  };
+  ...
+
    + +

    Examples

    + +Suppose the inclrec.jr file contains: +
    
+module inclrec {
+    class RI {
+        int      I32;
+        double   D;
+        ustring  S;
+    };
+}
+
    + +and the testrec.jr file contains: + +
    
+include "inclrec.jr"
+module testrec {
+    class R {
+        vector VF;
+        RI            Rec;
+        buffer        Buf;
+    };
+}
+
    + +Then the invocation of rcc such as: +
    
+$ rcc -l c++ inclrec.jr testrec.jr
+
    +will result in generation of four files: +inclrec.jr.{cc,hh} and testrec.jr.{cc,hh}. + +The inclrec.jr.hh will contain: + +
    
+#ifndef _INCLREC_JR_HH_
+#define _INCLREC_JR_HH_
+
+#include "recordio.hh"
+
+namespace inclrec {
+  
+  class RI : public hadoop::Record {
+
+  private:
+
+    int32_t      I32;
+    double       D;
+    std::string  S;
+
+  public:
+
+    RI(void);
+    virtual ~RI(void);
+
+    virtual bool operator==(const RI& peer) const;
+    virtual bool operator<(const RI& peer) const;
+
+    virtual int32_t getI32(void) const { return I32; }
+    virtual void setI32(int32_t v) { I32 = v; }
+
+    virtual double getD(void) const { return D; }
+    virtual void setD(double v) { D = v; }
+
+    virtual std::string& getS(void) const { return S; }
+    virtual const std::string& getS(void) const { return S; }
+
+    virtual std::string type(void) const;
+    virtual std::string signature(void) const;
+
+  protected:
+
+    virtual void serialize(hadoop::OArchive& a) const;
+    virtual void deserialize(hadoop::IArchive& a);
+  };
+} // end namespace inclrec
+
+#endif /* _INCLREC_JR_HH_ */
+
+
    + +The testrec.jr.hh file will contain: + + +
    
+
+#ifndef _TESTREC_JR_HH_
+#define _TESTREC_JR_HH_
+
+#include "inclrec.jr.hh"
+
+namespace testrec {
+  class R : public hadoop::Record {
+
+  private:
+
+    std::vector VF;
+    inclrec::RI        Rec;
+    std::string        Buf;
+
+  public:
+
+    R(void);
+    virtual ~R(void);
+
+    virtual bool operator==(const R& peer) const;
+    virtual bool operator<(const R& peer) const;
+
+    virtual std::vector& getVF(void) const;
+    virtual const std::vector& getVF(void) const;
+
+    virtual std::string& getBuf(void) const ;
+    virtual const std::string& getBuf(void) const;
+
+    virtual inclrec::RI& getRec(void) const;
+    virtual const inclrec::RI& getRec(void) const;
+    
+    virtual bool serialize(hadoop::OutArchive& a) const;
+    virtual bool deserialize(hadoop::InArchive& a);
+    
+    virtual std::string type(void) const;
+    virtual std::string signature(void) const;
+  };
+}; // end namespace testrec
+#endif /* _TESTREC_JR_HH_ */
+
+
    + +

    Java

    + +Code generation for Java is similar to that for C++. A Java class is generated +for each record type with private members corresponding to the fields. Getters +and setters for fields are also generated. Some differences arise in the +way comparison is expressed and in the mapping of modules to packages and +classes to files. For equality testing, an equals method is generated +for each record type. As per Java requirements a hashCode method is also +generated. For comparison a compareTo method is generated for each +record type. This has the semantics as defined by the Java Comparable +interface, that is, the method returns a negative integer, zero, or a positive +integer as the invoked object is less than, equal to, or greater than the +comparison parameter. + +A .java file is generated per record type as opposed to per DDL +file as in C++. The module declaration translates to a Java +package declaration. The module name maps to an identical Java package +name. In addition to this mapping, the DDL compiler creates the appropriate +directory hierarchy for the package and places the generated .java +files in the correct directories. + +

    Mapping Summary

    + +
    
+DDL Type        C++ Type            Java Type 
+
+boolean         bool                boolean
+byte            int8_t              byte
+int             int32_t             int
+long            int64_t             long
+float           float               float
+double          double              double
+ustring         std::string         java.lang.String
+buffer          std::string         org.apache.hadoop.record.Buffer
+class type      class type          class type
+vector    std::vector   java.util.ArrayList
+map  std::map java.util.TreeMap
+
    + +

    Data encodings

    + +This section describes the format of the data encodings supported by Hadoop. +Currently, three data encodings are supported, namely binary, CSV and XML. + +

    Binary Serialization Format

    + +The binary data encoding format is fairly dense. Serialization of composite +types is simply defined as a concatenation of serializations of the constituent +elements (lengths are included in vectors and maps). + +Composite types are serialized as follows: +
      +
    • class: Sequence of serialized members. +
    • vector: The number of elements serialized as an int. Followed by a +sequence of serialized elements. +
    • map: The number of key value pairs serialized as an int. Followed +by a sequence of serialized (key,value) pairs. +
    + +Serialization of primitives is more interesting, with a zero compression +optimization for integral types and normalization to UTF-8 for strings. +Primitive types are serialized as follows: + +
      +
    • byte: Represented by 1 byte, as is. +
    • boolean: Represented by 1-byte (0 or 1) +
    • int/long: Integers and longs are serialized zero compressed. +Represented as 1-byte if -120 <= value < 128. Otherwise, serialized as a +sequence of 2-5 bytes for ints, 2-9 bytes for longs. The first byte represents +the number of trailing bytes, N, as the negative number (-120-N). For example, +the number 1024 (0x400) is represented by the byte sequence 'x86 x04 x00'. +This doesn't help much for 4-byte integers but does a reasonably good job with +longs without bit twiddling. +
    • float/double: Serialized in IEEE 754 single and double precision +format in network byte order. This is the format used by Java. +
    • ustring: Serialized as 4-byte zero compressed length followed by +data encoded as UTF-8. Strings are normalized to UTF-8 regardless of native +language representation. +
    • buffer: Serialized as a 4-byte zero compressed length followed by the +raw bytes in the buffer. +
    + + +

    CSV Serialization Format

    + +The CSV serialization format has a lot more structure than the "standard" +Excel CSV format, but we believe the additional structure is useful because + +
      +
    • it makes parsing a lot easier without detracting too much from legibility +
    • the delimiters around composites make it obvious when one is reading a +sequence of Hadoop records +
    + +Serialization formats for the various types are detailed in the grammar that +follows. The notable feature of the formats is the use of delimiters for +indicating the certain field types. + +
      +
    • A string field begins with a single quote ('). +
    • A buffer field begins with a sharp (#). +
    • A class, vector or map begins with 's{', 'v{' or 'm{' respectively and +ends with '}'. +
    + +The CSV format can be described by the following grammar: + +
    
+record = primitive / struct / vector / map
+primitive = boolean / int / long / float / double / ustring / buffer
+
+boolean = "T" / "F"
+int = ["-"] 1*DIGIT
+long = ";" ["-"] 1*DIGIT
+float = ["-"] 1*DIGIT "." 1*DIGIT ["E" / "e" ["-"] 1*DIGIT]
+double = ";" ["-"] 1*DIGIT "." 1*DIGIT ["E" / "e" ["-"] 1*DIGIT]
+
+ustring = "'" *(UTF8 char except NULL, LF, % and , / "%00" / "%0a" / "%25" / "%2c" )
+
+buffer = "#" *(BYTE except NULL, LF, % and , / "%00" / "%0a" / "%25" / "%2c" )
+
+struct = "s{" record *("," record) "}"
+vector = "v{" [record *("," record)] "}"
+map = "m{" [*(record "," record)] "}"
+
    + +

    XML Serialization Format

    + +The XML serialization format is the same used by Apache XML-RPC +(http://ws.apache.org/xmlrpc/types.html). This is an extension of the original +XML-RPC format and adds some additional data types. All record I/O types are +not directly expressible in this format, and access to a DDL is required in +order to convert these to valid types. All types primitive or composite are +represented by <value> elements. The particular XML-RPC type is +indicated by a nested element in the <value> element. The encoding for +records is always UTF-8. Primitive types are serialized as follows: + +
      +
    • byte: XML tag <ex:i1>. Values: 1-byte unsigned +integers represented in US-ASCII +
    • boolean: XML tag <boolean>. Values: "0" or "1" +
    • int: XML tags <i4> or <int>. Values: 4-byte +signed integers represented in US-ASCII. +
    • long: XML tag <ex:i8>. Values: 8-byte signed integers +represented in US-ASCII. +
    • float: XML tag <ex:float>. Values: Single precision +floating point numbers represented in US-ASCII. +
    • double: XML tag <double>. Values: Double precision +floating point numbers represented in US-ASCII. +
    • ustring: XML tag <;string>. Values: String values +represented as UTF-8. XML does not permit all Unicode characters in literal +data. In particular, NULLs and control chars are not allowed. Additionally, +XML processors are required to replace carriage returns with line feeds and to +replace CRLF sequences with line feeds. Programming languages that we work +with do not impose these restrictions on string types. To work around these +restrictions, disallowed characters and CRs are percent escaped in strings. +The '%' character is also percent escaped. +
    • buffer: XML tag <string&>. Values: Arbitrary binary +data. Represented as hexBinary, each byte is replaced by its 2-byte +hexadecimal representation. +
    + +Composite types are serialized as follows: + +
      +
    • class: XML tag <struct>. A struct is a sequence of +<member> elements. Each <member> element has a <name> +element and a <value> element. The <name> is a string that must +match /[a-zA-Z][a-zA-Z0-9_]*/. The value of the member is represented +by a <value> element. + +
    • vector: XML tag <array<. An <array> contains a +single <data> element. The <data> element is a sequence of +<value> elements each of which represents an element of the vector. + +
    • map: XML tag <array>. Same as vector. + +
    + +For example: + +
    
+class {
+  int           MY_INT;            // value 5
+  vector MY_VEC;            // values 0.1, -0.89, 2.45e4
+  buffer        MY_BUF;            // value '\00\n\tabc%'
+}
+
    + +is serialized as + +
    
+<value>
+  <struct>
+    <member>
+      <name>MY_INT</name>
+      <value><i4>5</i4></value>
+    </member>
+    <member>
+      <name>MY_VEC</name>
+      <value>
+        <array>
+          <data>
+            <value><ex:float>0.1</ex:float></value>
+            <value><ex:float>-0.89</ex:float></value>
+            <value><ex:float>2.45e4</ex:float></value>
+          </data>
+        </array>
+      </value>
+    </member>
+    <member>
+      <name>MY_BUF</name>
+      <value><string>%00\n\tabc%25</string></value>
+    </member>
+  </struct>
+</value> 
+
    ]]> +     +     + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + Avro.]]> + + + + + + + + + Avro.]]> + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + Avro.]]> + + + + + + + + + Avro.]]> + + + + + + + + + + + + Avro.]]> + + + + + + (DEPRECATED) This package contains classes needed for code generation + from the hadoop record compiler. CppGenerator and JavaGenerator + are the main entry points from the parser. There are classes + corrsponding to every primitive type and compound type + included in Hadoop record I/O syntax. +

    + +

    + DEPRECATED: Replaced by Avro. +

    ]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This task takes the given record definition files and compiles them into + java or c++ + files. It is then up to the user to compile the generated files. + +

    The task requires the file or the nested fileset element to be + specified. Optional attributes are language (set the output + language, default is "java"), + destdir (name of the destination directory for generated java/c++ + code, default is ".") and failonerror (specifies error handling + behavior. default is true). +

    Usage

    + 
    + <recordcc
+       destdir="${basedir}/gensrc"
+       language="java">
+   <fileset include="**\/*.jr" />
+ </recordcc>
+
    + + @deprecated Replaced by Avro.]]> +     +     + +     + + + + + + ]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + (DEPRECATED) This package contains code generated by JavaCC from the + Hadoop record syntax file rcc.jj. For details about the + record file syntax please @see org.apache.hadoop.record. +

    + +

    + DEPRECATED: Replaced by Avro. +

    ]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + + + Avro.]]> + + + + + + + + + + + + + + + + + + + + (cause==null ? null : cause.toString()) (which + typically contains the class and detail message of cause). + @param cause the cause (which is saved for later retrieval by the + {@link #getCause()} method). (A null value is + permitted, and indicates that the cause is nonexistent or + unknown.)]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + mapping + and mapping]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + /host@realm. + @param principalName principal name of format as described above + @return host name if the the string conforms to the above format, else null]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + "jack" + + @param userName + @return userName without login method]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + the return type of the run method + @param action the method to execute + @return the value from the run method]]> + + + + + + + + the return type of the run method + @param action the method to execute + @return the value from the run method + @throws IOException if the action throws an IOException + @throws Error if the action throws an Error + @throws RuntimeException if the action throws a RuntimeException + @throws InterruptedException if the action throws an InterruptedException + @throws UndeclaredThrowableException if the action throws something else]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + (cause==null ? null : cause.toString()) (which + typically contains the class and detail message of cause). + @param cause the cause (which is saved for later retrieval by the + {@link #getCause()} method). (A null value is + permitted, and indicates that the cause is nonexistent or + unknown.)]]> + + + + + + + + + + + + + + does not provide the stack trace for security purposes.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + A User-Agent String is considered to be a browser if it matches + any of the regex patterns from browser-useragent-regex; the default + behavior is to consider everything a browser that matches the following: + "^Mozilla.*,^Opera.*". Subclasses can optionally override + this method to use different behavior. + + @param userAgent The User-Agent String, or null if there isn't one + @return true if the User-Agent String refers to a browser, false if not]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + The type of the token identifier]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + T extends TokenIdentifier]]> + + + + + + + + + + DelegationTokenAuthenticatedURL. +

    + An instance of the default {@link DelegationTokenAuthenticator} will be + used.]]> + + + + + DelegationTokenAuthenticatedURL. + + @param authenticator the {@link DelegationTokenAuthenticator} instance to + use, if null the default one will be used.]]> + + + + + DelegationTokenAuthenticatedURL using the default + {@link DelegationTokenAuthenticator} class. + + @param connConfigurator a connection configurator.]]> + + + + + DelegationTokenAuthenticatedURL. + + @param authenticator the {@link DelegationTokenAuthenticator} instance to + use, if null the default one will be used. + @param connConfigurator a connection configurator.]]> + + + + + + + + + + + + The default class is {@link KerberosDelegationTokenAuthenticator} + + @return the delegation token authenticator class to use as default.]]> + + + + + + + This method is provided to enable WebHDFS backwards compatibility. + + @param useQueryString TRUE if the token is transmitted in the + URL query string, FALSE if the delegation token is transmitted + using the {@link DelegationTokenAuthenticator#DELEGATION_TOKEN_HEADER} HTTP + header.]]> + + + + + TRUE if the token is transmitted in the URL query + string, FALSE if the delegation token is transmitted using the + {@link DelegationTokenAuthenticator#DELEGATION_TOKEN_HEADER} HTTP header.]]> + + + + + + + + + + + + + + + + + + Authenticator. + + @param url the URL to connect to. Only HTTP/S URLs are supported. + @param token the authentication token being used for the user. + @return an authenticated {@link HttpURLConnection}. + @throws IOException if an IO error occurred. + @throws AuthenticationException if an authentication exception occurred.]]> + + + + + + + + + + Authenticator. If the doAs parameter is not NULL, + the request will be done on behalf of the specified doAs user. + + @param url the URL to connect to. Only HTTP/S URLs are supported. + @param token the authentication token being used for the user. + @param doAs user to do the the request on behalf of, if NULL the request is + as self. + @return an authenticated {@link HttpURLConnection}. + @throws IOException if an IO error occurred. + @throws AuthenticationException if an authentication exception occurred.]]> + + + + + + + + + + Authenticator + for authentication. + + @param url the URL to get the delegation token from. Only HTTP/S URLs are + supported. + @param token the authentication token being used for the user where the + Delegation token will be stored. + @param renewer the renewer user. + @return a delegation token. + @throws IOException if an IO error occurred. + @throws AuthenticationException if an authentication exception occurred.]]> + + + + + + + + + + + Authenticator + for authentication. + + @param url the URL to get the delegation token from. Only HTTP/S URLs are + supported. + @param token the authentication token being used for the user where the + Delegation token will be stored. + @param renewer the renewer user. + @param doAsUser the user to do as, which will be the token owner. + @return a delegation token. + @throws IOException if an IO error occurred. + @throws AuthenticationException if an authentication exception occurred.]]> + + + + + + + + + Authenticator for authentication. + + @param url the URL to renew the delegation token from. Only HTTP/S URLs are + supported. + @param token the authentication token with the Delegation Token to renew. + @throws IOException if an IO error occurred. + @throws AuthenticationException if an authentication exception occurred.]]> + + + + + + + + + + Authenticator for authentication. + + @param url the URL to renew the delegation token from. Only HTTP/S URLs are + supported. + @param token the authentication token with the Delegation Token to renew. + @param doAsUser the user to do as, which will be the token owner. + @throws IOException if an IO error occurred. + @throws AuthenticationException if an authentication exception occurred.]]> + + + + + + + + Authenticator. + + @param url the URL to cancel the delegation token from. Only HTTP/S URLs + are supported. + @param token the authentication token with the Delegation Token to cancel. + @throws IOException if an IO error occurred.]]> + + + + + + + + + Authenticator. + + @param url the URL to cancel the delegation token from. Only HTTP/S URLs + are supported. + @param token the authentication token with the Delegation Token to cancel. + @param doAsUser the user to do as, which will be the token owner. + @throws IOException if an IO error occurred.]]> + + + + DelegationTokenAuthenticatedURL is a + {@link AuthenticatedURL} sub-class with built-in Hadoop Delegation Token + functionality. +

    + The authentication mechanisms supported by default are Hadoop Simple + authentication (also known as pseudo authentication) and Kerberos SPNEGO + authentication. +

    + Additional authentication mechanisms can be supported via {@link + DelegationTokenAuthenticator} implementations. +

    + The default {@link DelegationTokenAuthenticator} is the {@link + KerberosDelegationTokenAuthenticator} class which supports + automatic fallback from Kerberos SPNEGO to Hadoop Simple authentication via + the {@link PseudoDelegationTokenAuthenticator} class. +

    + AuthenticatedURL instances are not thread-safe.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Authenticator + for authentication. + + @param url the URL to get the delegation token from. Only HTTP/S URLs are + supported. + @param token the authentication token being used for the user where the + Delegation token will be stored. + @param renewer the renewer user. + @throws IOException if an IO error occurred. + @throws AuthenticationException if an authentication exception occurred.]]> + + + + + + + + + + + Authenticator + for authentication. + + @param url the URL to get the delegation token from. Only HTTP/S URLs are + supported. + @param token the authentication token being used for the user where the + Delegation token will be stored. + @param renewer the renewer user. + @param doAsUser the user to do as, which will be the token owner. + @throws IOException if an IO error occurred. + @throws AuthenticationException if an authentication exception occurred.]]> + + + + + + + + + + Authenticator for authentication. + + @param url the URL to renew the delegation token from. Only HTTP/S URLs are + supported. + @param token the authentication token with the Delegation Token to renew. + @throws IOException if an IO error occurred. + @throws AuthenticationException if an authentication exception occurred.]]> + + + + + + + + + + + Authenticator for authentication. + + @param url the URL to renew the delegation token from. Only HTTP/S URLs are + supported. + @param token the authentication token with the Delegation Token to renew. + @param doAsUser the user to do as, which will be the token owner. + @throws IOException if an IO error occurred. + @throws AuthenticationException if an authentication exception occurred.]]> + + + + + + + + + Authenticator. + + @param url the URL to cancel the delegation token from. Only HTTP/S URLs + are supported. + @param token the authentication token with the Delegation Token to cancel. + @throws IOException if an IO error occurred.]]> + + + + + + + + + + Authenticator. + + @param url the URL to cancel the delegation token from. Only HTTP/S URLs + are supported. + @param token the authentication token with the Delegation Token to cancel. + @param doAsUser the user to do as, which will be the token owner. + @throws IOException if an IO error occurred.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + KerberosDelegationTokenAuthenticator provides support for + Kerberos SPNEGO authentication mechanism and support for Hadoop Delegation + Token operations. +

    + It falls back to the {@link PseudoDelegationTokenAuthenticator} if the HTTP + endpoint does not trigger a SPNEGO authentication]]> + + + + + + + + + PseudoDelegationTokenAuthenticator provides support for + Hadoop's pseudo authentication mechanism that accepts + the user name specified as a query string parameter and support for Hadoop + Delegation Token operations. +

    + This mimics the model of Hadoop Simple authentication trusting the + {@link UserGroupInformation#getCurrentUser()} value.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + live. + @return a (snapshotted) map of blocker name->description values]]> + + + + + + + + + + + + + Do nothing if the service is null or not + in a state in which it can be/needs to be stopped. +

    + The service state is checked before the operation begins. + This process is not thread safe. + @param service a service or null]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

  • Any long-lived operation here will prevent the service state + change from completing in a timely manner.
    • +
  • If another thread is somehow invoked from the listener, and + that thread invokes the methods of the service (including + subclass-specific methods), there is a risk of a deadlock.
    • + + + + @param service the service that has changed.]]> +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +     + + + + + + + + + + + + + + + + The base implementation logs all arguments at the debug level, + then returns the passed in config unchanged.]]> + + + + + + + The action is to signal success by returning the exit code 0.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This method is called before {@link #init(Configuration)}; + Any non-null configuration that is returned from this operation + becomes the one that is passed on to that {@link #init(Configuration)} + operation. +

    + This permits implementations to change the configuration before + the init operation. As the ServiceLauncher only creates + an instance of the base {@link Configuration} class, it is + recommended to instantiate any subclass (such as YarnConfiguration) + that injects new resources. +

    + @param config the initial configuration build up by the + service launcher. + @param args list of arguments passed to the command line + after any launcher-specific commands have been stripped. + @return the configuration to init the service with. + Recommended: pass down the config parameter with any changes + @throws Exception any problem]]> + + + + + + + The return value becomes the exit code of the launched process. +

    + If an exception is raised, the policy is: +

      +
    1. Any subset of {@link org.apache.hadoop.util.ExitUtil.ExitException}: + the exception is passed up unmodified. +
      2. +
    2. Any exception which implements + {@link org.apache.hadoop.util.ExitCodeProvider}: + A new {@link ServiceLaunchException} is created with the exit code + and message of the thrown exception; the thrown exception becomes the + cause.
      3. +
    3. Any other exception: a new {@link ServiceLaunchException} is created + with the exit code {@link LauncherExitCodes#EXIT_EXCEPTION_THROWN} and + the message of the original exception (which becomes the cause).
      4. +
    + @return the exit code + @throws org.apache.hadoop.util.ExitUtil.ExitException an exception passed + up as the exit code and error text. + @throws Exception any exception to report. If it provides an exit code + this is used in a wrapping exception.]]> +     +     + + + The command line options will be passed down before the + {@link Service#init(Configuration)} operation is invoked via an + invocation of {@link LaunchableService#bindArgs(Configuration, List)} + After the service has been successfully started via {@link Service#start()} + the {@link LaunchableService#execute()} method is called to execute the + service. When this method returns, the service launcher will exit, using + the return code from the method as its exit option.]]> + +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Approximate HTTP equivalent: {@code 400 Bad Request}]]> + + + + + + approximate HTTP equivalent: Approximate HTTP equivalent: {@code 401 Unauthorized}]]> + + + + + + + + + + + Approximate HTTP equivalent: Approximate HTTP equivalent: {@code 403: Forbidden}]]> + + + + + + Approximate HTTP equivalent: {@code 404: Not Found}]]> + + + + + + Approximate HTTP equivalent: {@code 405: Not allowed}]]> + + + + + + Approximate HTTP equivalent: {@code 406: Not Acceptable}]]> + + + + + + Approximate HTTP equivalent: {@code 408: Request Timeout}]]> + + + + + + Approximate HTTP equivalent: {@code 409: Conflict}]]> + + + + + + Approximate HTTP equivalent: {@code 500 Internal Server Error}]]> + + + + + + Approximate HTTP equivalent: {@code 501: Not Implemented}]]> + + + + + + Approximate HTTP equivalent: {@code 503 Service Unavailable}]]> + + + + + + If raised, this is expected to be raised server-side and likely due + to client/server version incompatibilities. +

    + Approximate HTTP equivalent: {@code 505: Version Not Supported}]]> + + + + + + + + + + + + + + + Codes with a YARN prefix are YARN-related. +

    + Many of the exit codes are designed to resemble HTTP error codes, + squashed into a single byte. e.g 44 , "not found" is the equivalent + of 404. The various 2XX HTTP error codes aren't followed; + the Unix standard of "0" for success is used. + 

    +    0-10: general command issues
+   30-39: equivalent to the 3XX responses, where those responses are
+          considered errors by the application.
+   40-49: client-side/CLI/config problems
+   50-59: service-side problems.
+   60+  : application specific error codes
+
    ]]> +     +     + + + + + + + + + + + + + + + + + + + This uses {@link String#format(String, Object...)} + to build the formatted exception in the ENGLISH locale. +

    + If the last argument is a throwable, it becomes the cause of the exception. + It will also be used as a parameter for the format. + @param exitCode exit code + @param format format for message to use in exception + @param args list of arguments]]> + + + + + When caught by the ServiceLauncher, it will convert that + into a process exit code. + + The {@link #ServiceLaunchException(int, String, Object...)} constructor + generates formatted exceptions.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Clients and/or applications can use the provided Progressable + to explicitly report progress to the Hadoop framework. This is especially + important for operations which take significant amount of time since, + in-lieu of the reported progress, the framework has to assume that an error + has occured and time-out the operation.

    ]]> +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Class is to be obtained + @return the correctly typed Class of the given object.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + kill -0 command or equivalent]]> + + + + + + + + + + + + + + + + + + + ".cmd" on Windows, or ".sh" otherwise. + + @param parent File parent directory + @param basename String script file basename + @return File referencing the script in the directory]]> + + + + + + ".cmd" on Windows, or ".sh" otherwise. + + @param basename String script file basename + @return String script file name]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + IOException. + @return the path to {@link #WINUTILS_EXE} + @throws RuntimeException if the path is not resolvable]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Shell. + @return the thread that ran runCommand() that spawned this shell + or null if no thread is waiting for this shell to complete]]> + + + + + + + + + + + + Shell interface. + @param cmd shell command to execute. + @return the output of the executed command.]]> + + + + + + + + + Shell interface. + @param env the map of environment key=value + @param cmd shell command to execute. + @param timeout time in milliseconds after which script should be marked timeout + @return the output of the executed command. + @throws IOException on any problem.]]> + + + + + + + + Shell interface. + @param env the map of environment key=value + @param cmd shell command to execute. + @return the output of the executed command. + @throws IOException on any problem.]]> + + + + + Shell processes. + Iterates through a map of all currently running Shell + processes and destroys them one by one. This method is thread safe]]> + + + + + Shell objects.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CreateProcess synchronization object.]]> + + + + + os.name property.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Important: caller must check for this value being null. + The lack of such checks has led to many support issues being raised. +

    + @deprecated use one of the exception-raising getter methods, + specifically {@link #getWinUtilsPath()} or {@link #getWinUtilsFile()}]]> + + + + + + + + + + + + + + Shell can be used to run shell commands like du or + df. It also offers facilities to gate commands by + time-intervals.]]> + + + + + + + + ShutdownHookManager singleton. + + @return ShutdownHookManager singleton.]]> + + + + + + + Runnable + @param priority priority of the shutdownHook.]]> + + + + + + + + + Runnable + @param priority priority of the shutdownHook + @param timeout timeout of the shutdownHook + @param unit unit of the timeout TimeUnit]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ShutdownHookManager enables running shutdownHook + in a deterministic order, higher priority first. +

    + The JVM runs ShutdownHooks in a non-deterministic order or in parallel. + This class registers a single JVM shutdownHook and run all the + shutdownHooks registered to it (to this class) in order based on their + priority. + + Unless a hook was registered with a shutdown explicitly set through + {@link #addShutdownHook(Runnable, int, long, TimeUnit)}, + the shutdown time allocated to it is set by the configuration option + {@link CommonConfigurationKeysPublic#SERVICE_SHUTDOWN_TIMEOUT} in + {@code core-site.xml}, with a default value of + {@link CommonConfigurationKeysPublic#SERVICE_SHUTDOWN_TIMEOUT_DEFAULT} + seconds.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Tool, is the standard for any Map-Reduce tool/application. + The tool/application should delegate the handling of + + standard command-line options to {@link ToolRunner#run(Tool, String[])} + and only handle its custom arguments.

    + +

    Here is how a typical Tool is implemented:

    + 

    +     public class MyApp extends Configured implements Tool {
+     
+       public int run(String[] args) throws Exception {
+         // Configuration processed by ToolRunner
+         Configuration conf = getConf();
+         
+         // Create a JobConf using the processed conf
+         JobConf job = new JobConf(conf, MyApp.class);
+         
+         // Process custom command-line options
+         Path in = new Path(args[1]);
+         Path out = new Path(args[2]);
+         
+         // Specify various job-specific parameters     
+         job.setJobName("my-app");
+         job.setInputPath(in);
+         job.setOutputPath(out);
+         job.setMapperClass(MyMapper.class);
+         job.setReducerClass(MyReducer.class);
+
+         // Submit the job, then poll for progress until the job is complete
+         RunningJob runningJob = JobClient.runJob(job);
+         if (runningJob.isSuccessful()) {
+           return 0;
+         } else {
+           return 1;
+         }
+       }
+       
+       public static void main(String[] args) throws Exception {
+         // Let ToolRunner handle generic command-line options 
+         int res = ToolRunner.run(new Configuration(), new MyApp(), args);
+         
+         System.exit(res);
+       }
+     }
+

    + + @see GenericOptionsParser + @see ToolRunner]]> +     + + + + + + + + + + + + + Tool by {@link Tool#run(String[])}, after + parsing with the given generic arguments. Uses the given + Configuration, or builds one if null. + + Sets the Tool's configuration with the possibly modified + version of the conf. + + @param conf Configuration for the Tool. + @param tool Tool to run. + @param args command-line arguments to the tool. + @return exit code of the {@link Tool#run(String[])} method.]]> + + + + + + + + Tool with its Configuration. + + Equivalent to run(tool.getConf(), tool, args). + + @param tool Tool to run. + @param args command-line arguments to the tool. + @return exit code of the {@link Tool#run(String[])} method.]]> + + + + + + + + + + + + + + + + + ToolRunner can be used to run classes implementing + Tool interface. It works in conjunction with + {@link GenericOptionsParser} to parse the + + generic hadoop command line arguments and modifies the + Configuration of the Tool. The + application-specific options are passed along without being modified. +

    + + @see Tool + @see GenericOptionsParser]]> +     +     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +     + + + + + + + + + + + this filter. + @param nbHash The number of hash function to consider. + @param hashType type of the hashing function (see + {@link org.apache.hadoop.util.hash.Hash}).]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Bloom filter, as defined by Bloom in 1970. +

    + The Bloom filter is a data structure that was introduced in 1970 and that has been adopted by + the networking research community in the past decade thanks to the bandwidth efficiencies that it + offers for the transmission of set membership information between networked hosts. A sender encodes + the information into a bit vector, the Bloom filter, that is more compact than a conventional + representation. Computation and space costs for construction are linear in the number of elements. + The receiver uses the filter to test whether various elements are members of the set. Though the + filter will occasionally return a false positive, it will never return a false negative. When creating + the filter, the sender can choose its desired point in a trade-off between the false positive rate and the size. + +

    + Originally created by + European Commission One-Lab Project 034819. + + @see Filter The general behavior of a filter + + @see Space/Time Trade-Offs in Hash Coding with Allowable Errors]]> + + + + + + + + + + + + + this filter. + @param nbHash The number of hash function to consider. + @param hashType type of the hashing function (see + {@link org.apache.hadoop.util.hash.Hash}).]]> + + + + + + + + + this counting Bloom filter. +

    + Invariant: nothing happens if the specified key does not belong to this counter Bloom filter. + @param key The key to remove.]]> + + + + + + + + + + + + key -> count map. +

    NOTE: due to the bucket size of this filter, inserting the same + key more than 15 times will cause an overflow at all filter positions + associated with this key, and it will significantly increase the error + rate for this and other keys. For this reason the filter can only be + used to store small count values 0 <= N << 15. + @param key key to be tested + @return 0 if the key is not present. Otherwise, a positive value v will + be returned such that v == count with probability equal to the + error rate of this filter, and v > count otherwise. + Additionally, if the filter experienced an underflow as a result of + {@link #delete(Key)} operation, the return value may be lower than the + count with the probability of the false negative rate of such + filter.]]> + + + + + + + + + + + + + + + + + + + + + + counting Bloom filter, as defined by Fan et al. in a ToN + 2000 paper. +

    + A counting Bloom filter is an improvement to standard a Bloom filter as it + allows dynamic additions and deletions of set membership information. This + is achieved through the use of a counting vector instead of a bit vector. +

    + Originally created by + European Commission One-Lab Project 034819. + + @see Filter The general behavior of a filter + + @see Summary cache: a scalable wide-area web cache sharing protocol]]> + + + + + + + + + + + + + + Builds an empty Dynamic Bloom filter. + @param vectorSize The number of bits in the vector. + @param nbHash The number of hash function to consider. + @param hashType type of the hashing function (see + {@link org.apache.hadoop.util.hash.Hash}). + @param nr The threshold for the maximum number of keys to record in a + dynamic Bloom filter row.]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + dynamic Bloom filter, as defined in the INFOCOM 2006 paper. +

    + A dynamic Bloom filter (DBF) makes use of a s * m bit matrix but + each of the s rows is a standard Bloom filter. The creation + process of a DBF is iterative. At the start, the DBF is a 1 * m + bit matrix, i.e., it is composed of a single standard Bloom filter. + It assumes that nr elements are recorded in the + initial bit vector, where nr <= n (n is + the cardinality of the set A to record in the filter). +

    + As the size of A grows during the execution of the application, + several keys must be inserted in the DBF. When inserting a key into the DBF, + one must first get an active Bloom filter in the matrix. A Bloom filter is + active when the number of recorded keys, nr, is + strictly less than the current cardinality of A, n. + If an active Bloom filter is found, the key is inserted and + nr is incremented by one. On the other hand, if there + is no active Bloom filter, a new one is created (i.e., a new row is added to + the matrix) according to the current size of A and the element + is added in this new Bloom filter and the nr value of + this new Bloom filter is set to one. A given key is said to belong to the + DBF if the k positions are set to one in one of the matrix rows. +

    + Originally created by + European Commission One-Lab Project 034819. + + @see Filter The general behavior of a filter + @see BloomFilter A Bloom filter + + @see Theory and Network Applications of Dynamic Bloom Filters]]> + + + + + + + + + Builds a hash function that must obey to a given maximum number of returned values and a highest value. + @param maxValue The maximum highest returned value. + @param nbHash The number of resulting hashed values. + @param hashType type of the hashing function (see {@link Hash}).]]> + + + + + this hash function. A NOOP]]> + + + + + + + + + + + + + + + + + + + The idea is to randomly select a bit to reset.]]> + + + + + + The idea is to select the bit to reset that will generate the minimum + number of false negative.]]> + + + + + + The idea is to select the bit to reset that will remove the maximum number + of false positive.]]> + + + + + + The idea is to select the bit to reset that will, at the same time, remove + the maximum number of false positve while minimizing the amount of false + negative generated.]]> + + + + + Originally created by + European Commission One-Lab Project 034819.]]> + + + + + + + + + + + + + + this filter. + @param nbHash The number of hash function to consider. + @param hashType type of the hashing function (see + {@link org.apache.hadoop.util.hash.Hash}).]]> + + + + + + + + + this retouched Bloom filter. +

    + Invariant: if the false positive is null, nothing happens. + @param key The false positive key to add.]]> + + + + + + this retouched Bloom filter. + @param coll The collection of false positive.]]> + + + + + + this retouched Bloom filter. + @param keys The list of false positive.]]> + + + + + + this retouched Bloom filter. + @param keys The array of false positive.]]> + + + + + + + this retouched Bloom filter. + @param scheme The selective clearing scheme to apply.]]> + + + + + + + + + + + + retouched Bloom filter, as defined in the CoNEXT 2006 paper. +

    + It allows the removal of selected false positives at the cost of introducing + random false negatives, and with the benefit of eliminating some random false + positives at the same time. + +

    + Originally created by + European Commission One-Lab Project 034819. + + @see Filter The general behavior of a filter + @see BloomFilter A Bloom filter + @see RemoveScheme The different selective clearing algorithms + + @see Retouched Bloom Filters: Allowing Networked Applications to Trade Off Selected False Positives Against False Negatives]]> + + + + + + + + + + diff --git a/hadoop-common-project/hadoop-common/pom.xml b/hadoop-common-project/hadoop-common/pom.xml index 10417eb910bf3..84d3ae5b5addc 100644 --- a/hadoop-common-project/hadoop-common/pom.xml +++ b/hadoop-common-project/hadoop-common/pom.xml @@ -15,7 +15,7 @@ + https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -30,7 +30,6 @@ jar - src/test/resources/kdc common true true @@ -161,6 +160,16 @@ junit test + + org.assertj + assertj-core + test + + + org.glassfish.grizzly + grizzly-http-servlet + test + commons-beanutils commons-beanutils @@ -371,6 +380,20 @@ + + org.xolstice.maven.plugins + protobuf-maven-plugin + + + src-compile-protoc + false + + + src-test-compile-protoc + false + + + org.apache.hadoop hadoop-maven-plugins @@ -391,58 +414,6 @@ - - compile-protoc - - protoc - - - ${protobuf.version} - ${protoc.path} - - ${basedir}/src/main/proto - - - ${basedir}/src/main/proto - - HAServiceProtocol.proto - IpcConnectionContext.proto - ProtocolInfo.proto - RpcHeader.proto - ZKFCProtocol.proto - ProtobufRpcEngine.proto - Security.proto - GetUserMappingsProtocol.proto - TraceAdmin.proto - RefreshAuthorizationPolicyProtocol.proto - RefreshUserMappingsProtocol.proto - RefreshCallQueueProtocol.proto - GenericRefreshProtocol.proto - FSProtos.proto - - - - - - compile-test-protoc - - test-protoc - - - ${protobuf.version} - ${protoc.path} - - ${basedir}/src/test/proto - - - ${basedir}/src/test/proto - - test.proto - test_rpc_service.proto - - - - resource-gz generate-resources @@ -462,8 +433,6 @@ maven-surefire-plugin - ${startKdc} - ${kdc.resource.dir} ${runningWithNative} @@ -544,7 +513,6 @@ src/main/native/m4/* src/test/empty-file src/test/all-tests - src/test/resources/kdc/ldif/users.ldif src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.h src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4hc.h @@ -686,6 +654,8 @@ ${require.isal} ${isal.prefix} ${isal.lib} + ${require.pmdk} + ${pmdk.lib} ${require.openssl} ${openssl.prefix} ${openssl.lib} @@ -743,7 +713,7 @@ - false + false true @@ -846,7 +816,7 @@ /p:CustomZstdPrefix=${zstd.prefix} /p:CustomZstdLib=${zstd.lib} /p:CustomZstdInclude=${zstd.include} - /p:RequireZstd=${require.ztsd} + /p:RequireZstd=${require.zstd} /p:CustomOpensslPrefix=${openssl.prefix} /p:CustomOpensslLib=${openssl.lib} /p:CustomOpensslInclude=${openssl.include} @@ -862,87 +832,6 @@ - - - - startKdc - - - startKdc - true - - - - - - org.apache.maven.plugins - maven-enforcer-plugin - - - enforce-os - - enforce - - - - - - mac - unix - - - true - - - - - - org.apache.maven.plugins - maven-antrun-plugin - - - kdc - compile - - run - - - - - - - - - - - - - - - - - - - - - - killKdc - test - - run - - - - - - - - - - - - - parallel-tests diff --git a/hadoop-common-project/hadoop-common/src/CMakeLists.txt b/hadoop-common-project/hadoop-common/src/CMakeLists.txt index b9287c0f4b5ad..10591f6ce2aa8 100644 --- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt +++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt @@ -121,6 +121,7 @@ else () ENDIF(REQUIRE_ZSTD) endif () +#Require ISA-L set(STORED_CMAKE_FIND_LIBRARY_SUFFIXES ${CMAKE_FIND_LIBRARY_SUFFIXES}) hadoop_set_find_shared_library_version("2") find_library(ISAL_LIBRARY @@ -159,6 +160,25 @@ else (ISAL_LIBRARY) ENDIF(REQUIRE_ISAL) endif (ISAL_LIBRARY) +# Build with PMDK library if -Drequire.pmdk option is specified. +if(REQUIRE_PMDK) + set(STORED_CMAKE_FIND_LIBRARY_SUFFIXES ${CMAKE_FIND_LIBRARY_SUFFIXES}) + hadoop_set_find_shared_library_version("1") + find_library(PMDK_LIBRARY + NAMES pmem + PATHS ${CUSTOM_PMDK_LIB} /usr/lib /usr/lib64) + set(CMAKE_FIND_LIBRARY_SUFFIXES ${STORED_CMAKE_FIND_LIBRARY_SUFFIXES}) + + if(PMDK_LIBRARY) + GET_FILENAME_COMPONENT(HADOOP_PMDK_LIBRARY ${PMDK_LIBRARY} REALPATH) + set(PMDK_SOURCE_FILES ${SRC}/io/nativeio/pmdk_load.c) + else(PMDK_LIBRARY) + MESSAGE(FATAL_ERROR "The required PMDK library is NOT found. PMDK_LIBRARY=${PMDK_LIBRARY}") + endif(PMDK_LIBRARY) +else(REQUIRE_PMDK) + MESSAGE(STATUS "Build without PMDK support.") +endif(REQUIRE_PMDK) + # Build hardware CRC32 acceleration, if supported on the platform. if(CMAKE_SYSTEM_PROCESSOR MATCHES "^i.86$" OR CMAKE_SYSTEM_PROCESSOR STREQUAL "x86_64" OR CMAKE_SYSTEM_PROCESSOR STREQUAL "amd64") set(BULK_CRC_ARCH_SOURCE_FIlE "${SRC}/util/bulk_crc32_x86.c") @@ -256,6 +276,7 @@ hadoop_add_dual_library(hadoop ${SRC}/io/compress/zlib/ZlibDecompressor.c ${BZIP2_SOURCE_FILES} ${SRC}/io/nativeio/NativeIO.c + ${PMDK_SOURCE_FILES} ${SRC}/io/nativeio/errno_enum.c ${SRC}/io/nativeio/file_descriptor.c ${SRC}/io/nativeio/SharedFileDescriptorFactory.c diff --git a/hadoop-common-project/hadoop-common/src/config.h.cmake b/hadoop-common-project/hadoop-common/src/config.h.cmake index 40aa467373c8d..7e23a5df3281c 100644 --- a/hadoop-common-project/hadoop-common/src/config.h.cmake +++ b/hadoop-common-project/hadoop-common/src/config.h.cmake @@ -24,6 +24,7 @@ #cmakedefine HADOOP_ZSTD_LIBRARY "@HADOOP_ZSTD_LIBRARY@" #cmakedefine HADOOP_OPENSSL_LIBRARY "@HADOOP_OPENSSL_LIBRARY@" #cmakedefine HADOOP_ISAL_LIBRARY "@HADOOP_ISAL_LIBRARY@" +#cmakedefine HADOOP_PMDK_LIBRARY "@HADOOP_PMDK_LIBRARY@" #cmakedefine HAVE_SYNC_FILE_RANGE #cmakedefine HAVE_POSIX_FADVISE diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop b/hadoop-common-project/hadoop-common/src/main/bin/hadoop index 750dca31457a2..7d9ffc69bc503 100755 --- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop +++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop @@ -140,6 +140,10 @@ function hadoopcmd_case if [[ -n "${YARN_OPTS}" ]] || [[ -n "${YARN_CLIENT_OPTS}" ]]; then hadoop_error "WARNING: Use \"yarn jar\" to launch YARN applications." fi + if [[ -z $1 || $1 = "--help" ]]; then + echo "Usage: hadoop jar [mainClass] args..." + exit 0 + fi HADOOP_CLASSNAME=org.apache.hadoop.util.RunJar ;; jnipath) diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd b/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd index 91c65d1f2d6f2..04e5039d19812 100644 --- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd +++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd @@ -189,6 +189,11 @@ call :updatepath %HADOOP_BIN_PATH% ) else if defined YARN_CLIENT_OPTS ( @echo WARNING: Use "yarn jar" to launch YARN applications. ) + @rem if --help option is used, no need to call command + if [!hadoop-command-arguments[%1%]!]==["--help"] ( + @echo Usage: hadoop jar [mainClass] args... + goto :eof + ) set CLASS=org.apache.hadoop.util.RunJar goto :eof diff --git a/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml b/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml index bd7c11124f5b3..e1640f97546ac 100644 --- a/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml +++ b/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml @@ -109,6 +109,16 @@ active and stand-by states of namenode. + + security.router.admin.protocol.acl + * + ACL for RouterAdmin Protocol. The ACL is a comma-separated + list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + security.zkfc.protocol.acl * diff --git a/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties b/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties index 246a9d12d6715..7f9ea462679b3 100644 --- a/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties +++ b/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties @@ -216,6 +216,36 @@ log4j.appender.RMSUMMARY.MaxBackupIndex=20 log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout log4j.appender.RMSUMMARY.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n +# +# YARN ResourceManager audit logging +# +rm.audit.logger=INFO,NullAppender +rm.audit.log.maxfilesize=256MB +rm.audit.log.maxbackupindex=20 +log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger} +log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false +log4j.appender.RMAUDIT=org.apache.log4j.RollingFileAppender +log4j.appender.RMAUDIT.File=${hadoop.log.dir}/rm-audit.log +log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout +log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n +log4j.appender.RMAUDIT.MaxFileSize=${rm.audit.log.maxfilesize} +log4j.appender.RMAUDIT.MaxBackupIndex=${rm.audit.log.maxbackupindex} + +# +# YARN NodeManager audit logging +# +nm.audit.logger=INFO,NullAppender +nm.audit.log.maxfilesize=256MB +nm.audit.log.maxbackupindex=20 +log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=${nm.audit.logger} +log4j.additivity.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=false +log4j.appender.NMAUDIT=org.apache.log4j.RollingFileAppender +log4j.appender.NMAUDIT.File=${hadoop.log.dir}/nm-audit.log +log4j.appender.NMAUDIT.layout=org.apache.log4j.PatternLayout +log4j.appender.NMAUDIT.layout.ConversionPattern=%d{ISO8601}%p %c{2}: %m%n +log4j.appender.NMAUDIT.MaxFileSize=${nm.audit.log.maxfilesize} +log4j.appender.NMAUDIT.MaxBackupIndex=${nm.audit.log.maxbackupindex} + # HS audit log configs #mapreduce.hs.audit.logger=INFO,HSAUDIT #log4j.logger.org.apache.hadoop.mapreduce.v2.hs.HSAuditLogger=${mapreduce.hs.audit.logger} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java old mode 100644 new mode 100755 index c30ce0db84e22..180bde26574ca --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java @@ -1885,8 +1885,6 @@ private long getTimeDurationHelper(String name, String vStr, vStr = StringUtils.toLowerCase(vStr); ParsedTimeDuration vUnit = ParsedTimeDuration.unitFor(vStr); if (null == vUnit) { - logDeprecation("No unit for " + name + "(" + vStr + ") assuming " + - defaultUnit); vUnit = ParsedTimeDuration.unitFor(defaultUnit); } else { vStr = vStr.substring(0, vStr.lastIndexOf(vUnit.suffix())); @@ -3436,8 +3434,10 @@ private void readTagFromConfig(String attributeValue, String confName, String } private void overlay(Properties to, Properties from) { - for (Entry entry: from.entrySet()) { - to.put(entry.getKey(), entry.getValue()); + synchronized (from) { + for (Entry entry : from.entrySet()) { + to.put(entry.getKey(), entry.getValue()); + } } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/ReconfigurableBase.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/ReconfigurableBase.java index 23e1fda9053ca..8cacbdcdac039 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/ReconfigurableBase.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/ReconfigurableBase.java @@ -146,7 +146,8 @@ public void run() { oldConf.unset(change.prop); } } catch (ReconfigurationException e) { - errorMessage = e.getCause().getMessage(); + Throwable cause = e.getCause(); + errorMessage = cause == null ? e.getMessage() : cause.getMessage(); } results.put(change, Optional.ofNullable(errorMessage)); } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java index 80364ce7fe42a..9e601e26cf944 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java @@ -330,8 +330,8 @@ public int read(long position, byte[] buffer, int offset, int length) throws IOException { checkStream(); if (!(in instanceof PositionedReadable)) { - throw new UnsupportedOperationException("This stream does not support " + - "positioned read."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not support positioned read."); } final int n = ((PositionedReadable) in).read(position, buffer, offset, length); @@ -351,8 +351,8 @@ public int read(long position, final ByteBuffer buf) throws IOException { checkStream(); if (!(in instanceof ByteBufferPositionedReadable)) { - throw new UnsupportedOperationException("This stream does not support " + - "positioned reads with byte buffers."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not support positioned reads with byte buffers."); } int bufPos = buf.position(); final int n = ((ByteBufferPositionedReadable) in).read(position, buf); @@ -363,7 +363,27 @@ public int read(long position, final ByteBuffer buf) return n; } - + + /** + * Positioned readFully using {@link ByteBuffer}s. This method is thread-safe. + */ + @Override + public void readFully(long position, final ByteBuffer buf) + throws IOException { + checkStream(); + if (!(in instanceof ByteBufferPositionedReadable)) { + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not support positioned reads with byte buffers."); + } + int bufPos = buf.position(); + ((ByteBufferPositionedReadable) in).readFully(position, buf); + final int n = buf.position() - bufPos; + if (n > 0) { + // This operation does not change the current offset of the file + decrypt(position, buf, n, bufPos); + } + } + /** * Decrypt length bytes in buffer starting at offset. Output is also put * into buffer starting at offset. It is thread-safe. @@ -480,8 +500,8 @@ public void readFully(long position, byte[] buffer, int offset, int length) throws IOException { checkStream(); if (!(in instanceof PositionedReadable)) { - throw new UnsupportedOperationException("This stream does not support " + - "positioned readFully."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not support positioned readFully."); } ((PositionedReadable) in).readFully(position, buffer, offset, length); if (length > 0) { @@ -513,8 +533,8 @@ public void seek(long pos) throws IOException { } } else { if (!(in instanceof Seekable)) { - throw new UnsupportedOperationException("This stream does not " + - "support seek."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not support seek."); } ((Seekable) in).seek(pos); resetStreamOffset(pos); @@ -672,8 +692,8 @@ public boolean seekToNewSource(long targetPos) throws IOException { "Cannot seek to negative offset."); checkStream(); if (!(in instanceof Seekable)) { - throw new UnsupportedOperationException("This stream does not support " + - "seekToNewSource."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not support seekToNewSource."); } boolean result = ((Seekable) in).seekToNewSource(targetPos); resetStreamOffset(targetPos); @@ -687,16 +707,16 @@ public ByteBuffer read(ByteBufferPool bufferPool, int maxLength, checkStream(); if (outBuffer.remaining() > 0) { if (!(in instanceof Seekable)) { - throw new UnsupportedOperationException("This stream does not " + - "support seek."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not support seek."); } // Have some decrypted data unread, need to reset. ((Seekable) in).seek(getPos()); resetStreamOffset(getPos()); } if (!(in instanceof HasEnhancedByteBufferAccess)) { - throw new UnsupportedOperationException("This stream does not support " + - "enhanced byte buffer access."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not support enhanced byte buffer access."); } final ByteBuffer buffer = ((HasEnhancedByteBufferAccess) in). read(bufferPool, maxLength, opts); @@ -714,8 +734,8 @@ public ByteBuffer read(ByteBufferPool bufferPool, int maxLength, @Override public void releaseBuffer(ByteBuffer buffer) { if (!(in instanceof HasEnhancedByteBufferAccess)) { - throw new UnsupportedOperationException("This stream does not support " + - "release buffer."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not support release buffer."); } ((HasEnhancedByteBufferAccess) in).releaseBuffer(buffer); } @@ -724,8 +744,8 @@ public void releaseBuffer(ByteBuffer buffer) { public void setReadahead(Long readahead) throws IOException, UnsupportedOperationException { if (!(in instanceof CanSetReadahead)) { - throw new UnsupportedOperationException("This stream does not support " + - "setting the readahead caching strategy."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not support setting the readahead caching strategy."); } ((CanSetReadahead) in).setReadahead(readahead); } @@ -734,8 +754,9 @@ public void setReadahead(Long readahead) throws IOException, public void setDropBehind(Boolean dropCache) throws IOException, UnsupportedOperationException { if (!(in instanceof CanSetReadahead)) { - throw new UnsupportedOperationException("This stream does not " + - "support setting the drop-behind caching setting."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " stream does not support setting the drop-behind caching" + + " setting."); } ((CanSetDropBehind) in).setDropBehind(dropCache); } @@ -842,8 +863,8 @@ public boolean hasCapability(String capability) { case StreamCapabilities.READBYTEBUFFER: case StreamCapabilities.PREADBYTEBUFFER: if (!(in instanceof StreamCapabilities)) { - throw new UnsupportedOperationException("This stream does not expose " + - "its stream capabilities."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not expose its stream capabilities."); } return ((StreamCapabilities) in).hasCapability(capability); default: diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java index b2ae084b2dc08..7d26acbf21a03 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java @@ -379,13 +379,15 @@ public List getAtMost(String keyName, int num) throws IOException, if (numToFill > 0) { refiller.fillQueueForKey(keyName, ekvs, numToFill); } - // Asynch task to fill > lowWatermark - if (i <= (int) (lowWatermark * numValues)) { - submitRefillTask(keyName, keyQueue); - } - return ekvs; + + break; + } else { + ekvs.add(val); } - ekvs.add(val); + } + // Schedule a refill task in case queue has gone below the watermark + if (keyQueue.size() < (int) (lowWatermark * numValues)) { + submitRefillTask(keyName, keyQueue); } } catch (Exception e) { throw new IOException("Exception while contacting value generator ", e); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java index 6e82543ca850a..0453ca14537c3 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java @@ -60,6 +60,8 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.apache.hadoop.fs.impl.PathCapabilitiesSupport.validatePathCapabilityArgs; + /** * This class provides an interface for implementors of a Hadoop file system * (analogous to the VFS of Unix). Applications do not access this class; @@ -72,7 +74,7 @@ */ @InterfaceAudience.Public @InterfaceStability.Stable -public abstract class AbstractFileSystem { +public abstract class AbstractFileSystem implements PathCapabilities { static final Logger LOG = LoggerFactory.getLogger(AbstractFileSystem.class); /** Recording statistics per a file system class. */ @@ -1371,4 +1373,16 @@ public CompletableFuture openFileWithOptions(Path path, new CompletableFuture<>(), () -> open(path, bufferSize)); } + public boolean hasPathCapability(final Path path, + final String capability) + throws IOException { + switch (validatePathCapabilityArgs(makeQualified(path), capability)) { + case CommonPathCapabilities.FS_SYMLINKS: + // delegate to the existing supportsSymlinks() call. + return supportsSymlinks(); + default: + // the feature is not implemented. + return false; + } + } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferPositionedReadable.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferPositionedReadable.java index d99ee1624eb19..f8282d88c46c3 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferPositionedReadable.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferPositionedReadable.java @@ -17,6 +17,7 @@ */ package org.apache.hadoop.fs; +import java.io.EOFException; import java.io.IOException; import java.nio.ByteBuffer; @@ -55,6 +56,8 @@ public interface ByteBufferPositionedReadable { *

    * Implementations should treat 0-length requests as legitimate, and must not * signal an error upon their receipt. + *

    + * This does not change the current offset of a file, and is thread-safe. * * @param position position within file * @param buf the ByteBuffer to receive the results of the read operation. @@ -63,4 +66,25 @@ public interface ByteBufferPositionedReadable { * @throws IOException if there is some error performing the read */ int read(long position, ByteBuffer buf) throws IOException; + + /** + * Reads {@code buf.remaining()} bytes into buf from a given position in + * the file or until the end of the data was reached before the read + * operation completed. Callers should use {@code buf.limit(...)} to + * control the size of the desired read and {@code buf.position(...)} to + * control the offset into the buffer the data should be written to. + *

    + * This operation provides similar semantics to + * {@link #read(long, ByteBuffer)}, the difference is that this method is + * guaranteed to read data until the {@link ByteBuffer} is full, or until + * the end of the data stream is reached. + * + * @param position position within file + * @param buf the ByteBuffer to receive the results of the read operation. + * @throws IOException if there is some error performing the read + * @throws EOFException the end of the data was reached before + * the read operation completed + * @see #read(long, ByteBuffer) + */ + void readFully(long position, ByteBuffer buf) throws IOException; } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java index 99aa5d22babb4..5e5d29a28bfce 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java @@ -27,6 +27,7 @@ import java.util.Collections; import java.util.EnumSet; import java.util.List; +import java.util.Locale; import java.util.Set; import java.util.concurrent.CompletableFuture; @@ -42,6 +43,8 @@ import org.apache.hadoop.util.LambdaUtils; import org.apache.hadoop.util.Progressable; +import static org.apache.hadoop.fs.impl.PathCapabilitiesSupport.validatePathCapabilityArgs; + /**************************************************************** * Abstract Checksumed FileSystem. * It provide a basic implementation of a Checksumed FileSystem, @@ -872,4 +875,23 @@ public FSDataOutputStreamBuilder createFile(Path path) { public FSDataOutputStreamBuilder appendFile(Path path) { return createDataOutputStreamBuilder(this, path).append(); } + + /** + * Disable those operations which the checksummed FS blocks. + * {@inheritDoc} + */ + @Override + public boolean hasPathCapability(final Path path, final String capability) + throws IOException { + // query the superclass, which triggers argument validation. + final Path p = makeQualified(path); + switch (validatePathCapabilityArgs(p, capability)) { + case CommonPathCapabilities.FS_APPEND: + case CommonPathCapabilities.FS_CONCAT: + return false; + default: + return super.hasPathCapability(p, capability); + } + } + } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFs.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFs.java index 663b05d9901ba..bc1122c56a2bd 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFs.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFs.java @@ -473,6 +473,32 @@ public void renameInternal(Path src, Path dst) } } + @Override + public void renameInternal(Path src, Path dst, boolean overwrite) + throws AccessControlException, FileAlreadyExistsException, + FileNotFoundException, ParentNotDirectoryException, + UnresolvedLinkException, IOException { + Options.Rename renameOpt = Options.Rename.NONE; + if (overwrite) { + renameOpt = Options.Rename.OVERWRITE; + } + + if (isDirectory(src)) { + getMyFs().rename(src, dst, renameOpt); + } else { + getMyFs().rename(src, dst, renameOpt); + + Path checkFile = getChecksumFile(src); + if (exists(checkFile)) { //try to rename checksum + if (isDirectory(dst)) { + getMyFs().rename(checkFile, dst, renameOpt); + } else { + getMyFs().rename(checkFile, getChecksumFile(dst), renameOpt); + } + } + } + } + /** * Implement the delete(Path, boolean) in checksum * file system. diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java index 384e5d1e5f33c..8c09db1284cff 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java @@ -82,7 +82,7 @@ public class CommonConfigurationKeys extends CommonConfigurationKeysPublic { public static final String IPC_MAXIMUM_DATA_LENGTH = "ipc.maximum.data.length"; /** Default value for IPC_MAXIMUM_DATA_LENGTH. */ - public static final int IPC_MAXIMUM_DATA_LENGTH_DEFAULT = 64 * 1024 * 1024; + public static final int IPC_MAXIMUM_DATA_LENGTH_DEFAULT = 128 * 1024 * 1024; /** Max response size a client will accept. */ public static final String IPC_MAXIMUM_RESPONSE_LENGTH = @@ -106,8 +106,14 @@ public class CommonConfigurationKeys extends CommonConfigurationKeysPublic { public static final String IPC_CALLQUEUE_IMPL_KEY = "callqueue.impl"; public static final String IPC_SCHEDULER_IMPL_KEY = "scheduler.impl"; public static final String IPC_IDENTITY_PROVIDER_KEY = "identity-provider.impl"; + public static final String IPC_COST_PROVIDER_KEY = "cost-provider.impl"; public static final String IPC_BACKOFF_ENABLE = "backoff.enable"; public static final boolean IPC_BACKOFF_ENABLE_DEFAULT = false; + // Callqueue overflow trigger failover for stateless servers. + public static final String IPC_CALLQUEUE_SERVER_FAILOVER_ENABLE = + "callqueue.overflow.trigger.failover"; + public static final boolean IPC_CALLQUEUE_SERVER_FAILOVER_ENABLE_DEFAULT = + false; /** * IPC scheduler priority levels. @@ -218,6 +224,8 @@ public class CommonConfigurationKeys extends CommonConfigurationKeysPublic { SECURITY_CLIENT_PROTOCOL_ACL = "security.client.protocol.acl"; public static final String SECURITY_CLIENT_DATANODE_PROTOCOL_ACL = "security.client.datanode.protocol.acl"; + public static final String SECURITY_ROUTER_ADMIN_PROTOCOL_ACL = + "security.router.admin.protocol.acl"; public static final String SECURITY_DATANODE_PROTOCOL_ACL = "security.datanode.protocol.acl"; public static final String @@ -256,8 +264,6 @@ public class CommonConfigurationKeys extends CommonConfigurationKeysPublic { /** * HA health monitor and failover controller. */ - - /** How often to retry connecting to the service. */ public static final String HA_HM_CONNECT_RETRY_INTERVAL_KEY = "ha.health-monitor.connect-retry-interval.ms"; public static final long HA_HM_CONNECT_RETRY_INTERVAL_DEFAULT = 1000; @@ -271,7 +277,13 @@ public class CommonConfigurationKeys extends CommonConfigurationKeysPublic { public static final String HA_HM_SLEEP_AFTER_DISCONNECT_KEY = "ha.health-monitor.sleep-after-disconnect.ms"; public static final long HA_HM_SLEEP_AFTER_DISCONNECT_DEFAULT = 1000; - + + /** How many time to retry connecting to the service. */ + public static final String HA_HM_RPC_CONNECT_MAX_RETRIES_KEY = + "ha.health-monitor.rpc.connect.max.retries"; + public static final int HA_HM_RPC_CONNECT_MAX_RETRIES_DEFAULT = 1; + + /** How often to retry connecting to the service. */ /* Timeout for the actual monitorHealth() calls. */ public static final String HA_HM_RPC_TIMEOUT_KEY = "ha.health-monitor.rpc-timeout.ms"; @@ -401,4 +413,17 @@ public class CommonConfigurationKeys extends CommonConfigurationKeysPublic { public static final Class HADOOP_DOMAINNAME_RESOLVER_IMPL_DEFAULT = DNSDomainNameResolver.class; + /* + * Ignore KMS default URI returned from NameNode. + * When set to true, kms uri is searched in the following order: + * 1. If there is a mapping in Credential's secrets map for namenode uri. + * 2. Fallback to local conf. + * If client choose to ignore KMS uri provided by NameNode then client + * should set KMS URI using 'hadoop.security.key.provider.path' to access + * the right KMS for encrypted files. + * */ + public static final String DFS_CLIENT_IGNORE_NAMENODE_DEFAULT_KMS_URI = + "dfs.client.ignore.namenode.default.kms.uri"; + public static final boolean + DFS_CLIENT_IGNORE_NAMENODE_DEFAULT_KMS_URI_DEFAULT = false; } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java index 0fa78c2f2aea0..a68012b06d2bc 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java @@ -72,6 +72,25 @@ public class CommonConfigurationKeysPublic { public static final String FS_DU_INTERVAL_KEY = "fs.du.interval"; /** Default value for FS_DU_INTERVAL_KEY */ public static final long FS_DU_INTERVAL_DEFAULT = 600000; + + /** + * @see + * + * core-default.xml + */ + public static final String FS_GETSPACEUSED_CLASSNAME = + "fs.getspaceused.classname"; + + /** + * @see + * + * core-default.xml + */ + public static final String FS_GETSPACEUSED_JITTER_KEY = + "fs.getspaceused.jitterMillis"; + /** Default value for FS_GETSPACEUSED_JITTER_KEY */ + public static final long FS_GETSPACEUSED_JITTER_DEFAULT = 60000; + /** * @see * @@ -398,7 +417,7 @@ public class CommonConfigurationKeysPublic { public static final String IPC_SERVER_LISTEN_QUEUE_SIZE_KEY = "ipc.server.listen.queue.size"; /** Default value for IPC_SERVER_LISTEN_QUEUE_SIZE_KEY */ - public static final int IPC_SERVER_LISTEN_QUEUE_SIZE_DEFAULT = 128; + public static final int IPC_SERVER_LISTEN_QUEUE_SIZE_DEFAULT = 256; /** * @see * @@ -961,5 +980,13 @@ public class CommonConfigurationKeysPublic { /** Default shutdown hook timeout: {@value} seconds. */ public static final long SERVICE_SHUTDOWN_TIMEOUT_DEFAULT = 30; + /** + * @see + * + * core-default.xml + */ + public static final String HADOOP_PROMETHEUS_ENABLED = + "hadoop.prometheus.endpoint.enabled"; + public static final boolean HADOOP_PROMETHEUS_ENABLED_DEFAULT = false; } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java new file mode 100644 index 0000000000000..31e6bac0ccee5 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs; + +/** + * Common path capabilities. + */ +public final class CommonPathCapabilities { + + private CommonPathCapabilities() { + } + + /** + * Does the store support + * {@code FileSystem.setAcl(Path, List)}, + * {@code FileSystem.getAclStatus(Path)} + * and related methods? + * Value: {@value}. + */ + public static final String FS_ACLS = "fs.capability.paths.acls"; + + /** + * Does the store support {@code FileSystem.append(Path)}? + * Value: {@value}. + */ + public static final String FS_APPEND = "fs.capability.paths.append"; + + /** + * Does the store support {@code FileSystem.getFileChecksum(Path)}? + * Value: {@value}. + */ + public static final String FS_CHECKSUMS = "fs.capability.paths.checksums"; + + /** + * Does the store support {@code FileSystem.concat(Path, Path[])}? + * Value: {@value}. + */ + public static final String FS_CONCAT = "fs.capability.paths.concat"; + + /** + * Does the store support {@code FileSystem.listCorruptFileBlocks(Path)} ()}? + * Value: {@value}. + */ + public static final String FS_LIST_CORRUPT_FILE_BLOCKS = + "fs.capability.paths.list-corrupt-file-blocks"; + + /** + * Does the store support + * {@code FileSystem.createPathHandle(FileStatus, Options.HandleOpt...)} + * and related methods? + * Value: {@value}. + */ + public static final String FS_PATHHANDLES = "fs.capability.paths.pathhandles"; + + /** + * Does the store support {@code FileSystem.setPermission(Path, FsPermission)} + * and related methods? + * Value: {@value}. + */ + public static final String FS_PERMISSIONS = "fs.capability.paths.permissions"; + + /** + * Does this filesystem connector only support filesystem read operations? + * For example, the {@code HttpFileSystem} is always read-only. + * This is different from "is the specific instance and path read only?", + * which must be determined by checking permissions (where supported), or + * attempting write operations under a path. + * Value: {@value}. + */ + public static final String FS_READ_ONLY_CONNECTOR = + "fs.capability.paths.read-only-connector"; + + /** + * Does the store support snapshots through + * {@code FileSystem.createSnapshot(Path)} and related methods?? + * Value: {@value}. + */ + public static final String FS_SNAPSHOTS = "fs.capability.paths.snapshots"; + + /** + * Does the store support {@code FileSystem.setStoragePolicy(Path, String)} + * and related methods? + * Value: {@value}. + */ + public static final String FS_STORAGEPOLICY = + "fs.capability.paths.storagepolicy"; + + /** + * Does the store support symlinks through + * {@code FileSystem.createSymlink(Path, Path, boolean)} and related methods? + * Value: {@value}. + */ + public static final String FS_SYMLINKS = + "fs.capability.paths.symlinks"; + + /** + * Does the store support {@code FileSystem#truncate(Path, long)} ? + * Value: {@value}. + */ + public static final String FS_TRUNCATE = + "fs.capability.paths.truncate"; + + /** + * Does the store support XAttributes through + * {@code FileSystem#.setXAttr()} and related methods? + * Value: {@value}. + */ + public static final String FS_XATTRS = "fs.capability.paths.xattrs"; + +} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java index 165c56c3d5c37..a8f294f379158 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java @@ -281,4 +281,11 @@ public CompletableFuture openFileWithOptions(Path path, int bufferSize) throws IOException { return fsImpl.openFileWithOptions(path, mandatoryKeys, options, bufferSize); } + + @Override + public boolean hasPathCapability(final Path path, + final String capability) + throws IOException { + return fsImpl.hasPathCapability(path, capability); + } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java index 066cc3d8b1926..31f82975899e1 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java @@ -52,8 +52,8 @@ public class FSDataInputStream extends DataInputStream public FSDataInputStream(InputStream in) { super(in); if( !(in instanceof Seekable) || !(in instanceof PositionedReadable) ) { - throw new IllegalArgumentException( - "In is not an instance of Seekable or PositionedReadable"); + throw new IllegalArgumentException(in.getClass().getCanonicalName() + + " is not an instance of Seekable or PositionedReadable"); } } @@ -150,7 +150,7 @@ public int read(ByteBuffer buf) throws IOException { } throw new UnsupportedOperationException("Byte-buffer read unsupported " + - "by input stream"); + "by " + in.getClass().getCanonicalName()); } @Override @@ -170,9 +170,8 @@ public void setReadahead(Long readahead) try { ((CanSetReadahead)in).setReadahead(readahead); } catch (ClassCastException e) { - throw new UnsupportedOperationException( - "this stream does not support setting the readahead " + - "caching strategy."); + throw new UnsupportedOperationException(in.getClass().getCanonicalName() + + " does not support setting the readahead caching strategy."); } } @@ -256,6 +255,16 @@ public int read(long position, ByteBuffer buf) throws IOException { return ((ByteBufferPositionedReadable) in).read(position, buf); } throw new UnsupportedOperationException("Byte-buffer pread unsupported " + - "by input stream"); + "by " + in.getClass().getCanonicalName()); + } + + @Override + public void readFully(long position, ByteBuffer buf) throws IOException { + if (in instanceof ByteBufferPositionedReadable) { + ((ByteBufferPositionedReadable) in).readFully(position, buf); + } else { + throw new UnsupportedOperationException("Byte-buffer pread " + + "unsupported by " + in.getClass().getCanonicalName()); + } } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java index f65074856bf3e..b2c1369a9c1fe 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java @@ -46,6 +46,8 @@ import org.apache.hadoop.fs.FileSystem.Statistics; import org.apache.hadoop.fs.Options.CreateOpts; import org.apache.hadoop.fs.impl.FutureDataInputStreamBuilderImpl; +import org.apache.hadoop.fs.impl.FsLinkResolution; +import org.apache.hadoop.fs.impl.PathCapabilitiesSupport; import org.apache.hadoop.fs.permission.AclEntry; import org.apache.hadoop.fs.permission.AclStatus; import org.apache.hadoop.fs.permission.FsAction; @@ -68,6 +70,8 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.apache.hadoop.fs.impl.PathCapabilitiesSupport.validatePathCapabilityArgs; + /** * The FileContext class provides an interface for users of the Hadoop * file system. It exposes a number of file system operations, e.g. create, @@ -171,7 +175,7 @@ @InterfaceAudience.Public @InterfaceStability.Stable -public class FileContext { +public class FileContext implements PathCapabilities { public static final Logger LOG = LoggerFactory.getLogger(FileContext.class); /** @@ -2934,4 +2938,21 @@ public CompletableFuture next( }.resolve(FileContext.this, absF); } } + + /** + * Return the path capabilities of the bonded {@code AbstractFileSystem}. + * @param path path to query the capability of. + * @param capability string to query the stream support for. + * @return true iff the capability is supported under that FS. + * @throws IOException path resolution or other IO failure + * @throws IllegalArgumentException invalid arguments + */ + public boolean hasPathCapability(Path path, String capability) + throws IOException { + validatePathCapabilityArgs(path, capability); + return FsLinkResolution.resolve(this, + fixRelativePart(path), + (fs, p) -> fs.hasPathCapability(p, capability)); + } + } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java index 19f38af69998f..2376c051c99f9 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java @@ -88,6 +88,7 @@ import static com.google.common.base.Preconditions.checkArgument; import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.*; +import static org.apache.hadoop.fs.impl.PathCapabilitiesSupport.validatePathCapabilityArgs; /**************************************************************** * An abstract base class for a fairly generic filesystem. It @@ -134,7 +135,7 @@ @InterfaceAudience.Public @InterfaceStability.Stable public abstract class FileSystem extends Configured - implements Closeable, DelegationTokenIssuer { + implements Closeable, DelegationTokenIssuer, PathCapabilities { public static final String FS_DEFAULT_NAME_KEY = CommonConfigurationKeys.FS_DEFAULT_NAME_KEY; public static final String DEFAULT_FS = @@ -206,6 +207,17 @@ static void addFileSystemForTesting(URI uri, Configuration conf, CACHE.map.put(new Cache.Key(uri, conf), fs); } + @VisibleForTesting + static void removeFileSystemForTesting(URI uri, Configuration conf, + FileSystem fs) throws IOException { + CACHE.map.remove(new Cache.Key(uri, conf), fs); + } + + @VisibleForTesting + static int cacheSize() { + return CACHE.map.size(); + } + /** * Get a FileSystem instance based on the uri, the passed in * configuration and the user. @@ -709,6 +721,7 @@ protected FileSystem() { * */ protected void checkPath(Path path) { + Preconditions.checkArgument(path != null, "null path"); URI uri = path.toUri(); String thatScheme = uri.getScheme(); if (thatScheme == null) // fs is relative @@ -1782,6 +1795,33 @@ public QuotaUsage getQuotaUsage(Path f) throws IOException { return getContentSummary(f); } + /** + * Set quota for the given {@link Path}. + * + * @param src the target path to set quota for + * @param namespaceQuota the namespace quota (i.e., # of files/directories) + * to set + * @param storagespaceQuota the storage space quota to set + * @throws IOException IO failure + */ + public void setQuota(Path src, final long namespaceQuota, + final long storagespaceQuota) throws IOException { + methodNotSupported(); + } + + /** + * Set per storage type quota for the given {@link Path}. + * + * @param src the target path to set storage type quota for + * @param type the storage type to set + * @param quota the quota to set for the given storage type + * @throws IOException IO failure + */ + public void setQuotaByStorageType(Path src, final StorageType type, + final long quota) throws IOException { + methodNotSupported(); + } + /** * The default filter accepts all paths. */ @@ -2024,7 +2064,12 @@ public FileStatus[] listStatus(Path[] files, PathFilter filter) * @throws IOException IO failure */ public FileStatus[] globStatus(Path pathPattern) throws IOException { - return new Globber(this, pathPattern, DEFAULT_FILTER).glob(); + return Globber.createGlobber(this) + .withPathPattern(pathPattern) + .withPathFiltern(DEFAULT_FILTER) + .withResolveSymlinks(true) + .build() + .glob(); } /** @@ -3221,6 +3266,25 @@ public Collection getTrashRoots(boolean allUsers) { return ret; } + /** + * The base FileSystem implementation generally has no knowledge + * of the capabilities of actual implementations. + * Unless it has a way to explicitly determine the capabilities, + * this method returns false. + * {@inheritDoc} + */ + public boolean hasPathCapability(final Path path, final String capability) + throws IOException { + switch (validatePathCapabilityArgs(makeQualified(path), capability)) { + case CommonPathCapabilities.FS_SYMLINKS: + // delegate to the existing supportsSymlinks() call. + return supportsSymlinks() && areSymlinksEnabled(); + default: + // the feature is not implemented. + return false; + } + } + // making it volatile to be able to do a double checked locking private volatile static boolean FILE_SYSTEMS_LOADED = false; @@ -3379,6 +3443,9 @@ private FileSystem getInternal(URI uri, Configuration conf, Key key) } fs = createFileSystem(uri, conf); + final long timeout = conf.getTimeDuration(SERVICE_SHUTDOWN_TIMEOUT, + SERVICE_SHUTDOWN_TIMEOUT_DEFAULT, + ShutdownHookManager.TIME_UNIT_DEFAULT); synchronized (this) { // refetch the lock again FileSystem oldfs = map.get(key); if (oldfs != null) { // a file system is created while lock is releasing @@ -3389,7 +3456,9 @@ private FileSystem getInternal(URI uri, Configuration conf, Key key) // now insert the new file system into the map if (map.isEmpty() && !ShutdownHookManager.get().isShutdownInProgress()) { - ShutdownHookManager.get().addShutdownHook(clientFinalizer, SHUTDOWN_HOOK_PRIORITY); + ShutdownHookManager.get().addShutdownHook(clientFinalizer, + SHUTDOWN_HOOK_PRIORITY, timeout, + ShutdownHookManager.TIME_UNIT_DEFAULT); } fs.key = key; map.put(key, fs); @@ -4444,6 +4513,22 @@ protected CompletableFuture openFileWithOptions( return result; } + /** + * Helper method that throws an {@link UnsupportedOperationException} for the + * current {@link FileSystem} method being called. + */ + private void methodNotSupported() { + // The order of the stacktrace elements is (from top to bottom): + // - java.lang.Thread.getStackTrace + // - org.apache.hadoop.fs.FileSystem.methodNotSupported + // - + // therefore, to find out the current method name, we use the element at + // index 2. + String name = Thread.currentThread().getStackTrace()[2].getMethodName(); + throw new UnsupportedOperationException(getClass().getCanonicalName() + + " does not support method " + name); + } + /** * Create instance of the standard {@link FSDataInputStreamBuilder} for the * given filesystem and path. diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java index 99c18b6646cd6..3bc3cb2e9b07a 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java @@ -334,6 +334,10 @@ public boolean mkdirs(Path f, FsPermission permission) throws IOException { return fs.mkdirs(f, permission); } + @Override + public boolean mkdirs(Path f) throws IOException { + return fs.mkdirs(f); + } /** * The src file is on the local disk. Add it to FS at @@ -725,4 +729,11 @@ protected CompletableFuture openFileWithOptions( return fs.openFileWithOptions(pathHandle, mandatoryKeys, options, bufferSize); } + + @Override + public boolean hasPathCapability(final Path path, final String capability) + throws IOException { + return fs.hasPathCapability(path, capability); + } + } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFs.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFs.java index f5430d6026160..731a52a7b4137 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFs.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFs.java @@ -446,4 +446,9 @@ public CompletableFuture openFileWithOptions( return myFs.openFileWithOptions(path, mandatoryKeys, options, bufferSize); } + public boolean hasPathCapability(final Path path, + final String capability) + throws IOException { + return myFs.hasPathCapability(path, capability); + } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShell.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShell.java index 5be6e5f829b8c..680e742a36059 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShell.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShell.java @@ -97,7 +97,7 @@ protected Help getHelp() throws IOException { return this.help; } - protected void init() throws IOException { + protected void init() { getConf().setQuietMode(true); UserGroupInformation.setConfiguration(getConf()); if (commandFactory == null) { @@ -298,7 +298,7 @@ private TableListing createOptionTableListing() { * run */ @Override - public int run(String argv[]) throws Exception { + public int run(String[] argv) { // initialize FsShell init(); Tracer tracer = new Tracer.Builder("FsShell"). diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java index 4d1f9efe117b8..3439317893b2b 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java @@ -26,7 +26,6 @@ import java.io.IOException; import java.lang.reflect.Constructor; import java.lang.reflect.InvocationTargetException; -import java.util.concurrent.TimeUnit; public interface GetSpaceUsed { @@ -36,20 +35,16 @@ public interface GetSpaceUsed { /** * The builder class */ - final class Builder { + class Builder { static final Logger LOG = LoggerFactory.getLogger(Builder.class); - static final String CLASSNAME_KEY = "fs.getspaceused.classname"; - static final String JITTER_KEY = "fs.getspaceused.jitterMillis"; - static final long DEFAULT_JITTER = TimeUnit.MINUTES.toMillis(1); - - private Configuration conf; private Class klass = null; private File path = null; private Long interval = null; private Long jitter = null; private Long initialUsed = null; + private Constructor cons; public Configuration getConf() { return conf; @@ -89,7 +84,8 @@ public Class getKlass() { if (conf == null) { return result; } - return conf.getClass(CLASSNAME_KEY, result, GetSpaceUsed.class); + return conf.getClass(CommonConfigurationKeys.FS_GETSPACEUSED_CLASSNAME, + result, GetSpaceUsed.class); } public Builder setKlass(Class klass) { @@ -124,9 +120,10 @@ public long getJitter() { Configuration configuration = this.conf; if (configuration == null) { - return DEFAULT_JITTER; + return CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_DEFAULT; } - return configuration.getLong(JITTER_KEY, DEFAULT_JITTER); + return configuration.getLong(CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_KEY, + CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_DEFAULT); } return jitter; } @@ -136,11 +133,21 @@ public Builder setJitter(Long jit) { return this; } + public Constructor getCons() { + return cons; + } + + public void setCons(Constructor cons) { + this.cons = cons; + } + public GetSpaceUsed build() throws IOException { GetSpaceUsed getSpaceUsed = null; try { - Constructor cons = - getKlass().getConstructor(Builder.class); + if (cons == null) { + cons = getKlass().getConstructor(Builder.class); + } + getSpaceUsed = cons.newInstance(this); } catch (InstantiationException e) { LOG.warn("Error trying to create an instance of " + getKlass(), e); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java index b241a949b1533..f301f22057925 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java @@ -25,15 +25,24 @@ import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.util.DurationInfo; import org.apache.htrace.core.TraceScope; import org.apache.htrace.core.Tracer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static com.google.common.base.Preconditions.checkNotNull; + +/** + * Implementation of {@link FileSystem#globStatus(Path, PathFilter)}. + * This has historically been package-private; it has been opened + * up for object stores within the {@code hadoop-*} codebase ONLY. + * It could be expanded for external store implementations in future. + */ @InterfaceAudience.Private @InterfaceStability.Unstable -class Globber { +public class Globber { public static final Logger LOG = LoggerFactory.getLogger(Globber.class.getName()); @@ -42,21 +51,62 @@ class Globber { private final Path pathPattern; private final PathFilter filter; private final Tracer tracer; - - public Globber(FileSystem fs, Path pathPattern, PathFilter filter) { + private final boolean resolveSymlinks; + + Globber(FileSystem fs, Path pathPattern, PathFilter filter) { + this.fs = fs; + this.fc = null; + this.pathPattern = pathPattern; + this.filter = filter; + this.tracer = FsTracer.get(fs.getConf()); + this.resolveSymlinks = true; + } + + Globber(FileContext fc, Path pathPattern, PathFilter filter) { + this.fs = null; + this.fc = fc; + this.pathPattern = pathPattern; + this.filter = filter; + this.tracer = fc.getTracer(); + this.resolveSymlinks = true; + } + + /** + * Filesystem constructor for use by {@link GlobBuilder}. + * @param fs filesystem + * @param pathPattern path pattern + * @param filter optional filter + * @param resolveSymlinks should symlinks be resolved. + */ + private Globber(FileSystem fs, Path pathPattern, PathFilter filter, + boolean resolveSymlinks) { this.fs = fs; this.fc = null; this.pathPattern = pathPattern; this.filter = filter; + this.resolveSymlinks = resolveSymlinks; this.tracer = FsTracer.get(fs.getConf()); + LOG.debug("Created Globber for path={}, symlinks={}", + pathPattern, resolveSymlinks); } - public Globber(FileContext fc, Path pathPattern, PathFilter filter) { + /** + * File Context constructor for use by {@link GlobBuilder}. + * @param fc file context + * @param pathPattern path pattern + * @param filter optional filter + * @param resolveSymlinks should symlinks be resolved. + */ + private Globber(FileContext fc, Path pathPattern, PathFilter filter, + boolean resolveSymlinks) { this.fs = null; this.fc = fc; this.pathPattern = pathPattern; this.filter = filter; + this.resolveSymlinks = resolveSymlinks; this.tracer = fc.getTracer(); + LOG.debug("Created Globber path={}, symlinks={}", + pathPattern, resolveSymlinks); } private FileStatus getFileStatus(Path path) throws IOException { @@ -67,6 +117,7 @@ private FileStatus getFileStatus(Path path) throws IOException { return fc.getFileStatus(path); } } catch (FileNotFoundException e) { + LOG.debug("getFileStatus({}) failed; returning null", path, e); return null; } } @@ -79,6 +130,7 @@ private FileStatus[] listStatus(Path path) throws IOException { return fc.util().listStatus(path); } } catch (FileNotFoundException e) { + LOG.debug("listStatus({}) failed; returning empty array", path, e); return new FileStatus[0]; } } @@ -107,7 +159,7 @@ private static String unescapePathComponent(String name) { */ private static List getPathComponents(String path) throws IOException { - ArrayList ret = new ArrayList(); + ArrayList ret = new ArrayList<>(); for (String component : path.split(Path.SEPARATOR)) { if (!component.isEmpty()) { ret.add(component); @@ -145,7 +197,8 @@ private String authorityFromPath(Path path) throws IOException { public FileStatus[] glob() throws IOException { TraceScope scope = tracer.newScope("Globber#glob"); scope.addKVAnnotation("pattern", pathPattern.toUri().getPath()); - try { + try (DurationInfo ignored = new DurationInfo(LOG, false, + "glob %s", pathPattern)) { return doGlob(); } finally { scope.close(); @@ -164,10 +217,11 @@ private FileStatus[] doGlob() throws IOException { String pathPatternString = pathPattern.toUri().getPath(); List flattenedPatterns = GlobExpander.expand(pathPatternString); + LOG.debug("Filesystem glob {}", pathPatternString); // Now loop over all flattened patterns. In every case, we'll be trying to // match them to entries in the filesystem. ArrayList results = - new ArrayList(flattenedPatterns.size()); + new ArrayList<>(flattenedPatterns.size()); boolean sawWildcard = false; for (String flatPattern : flattenedPatterns) { // Get the absolute path for this flattened pattern. We couldn't do @@ -175,13 +229,14 @@ private FileStatus[] doGlob() throws IOException { // path you go down influences how the path must be made absolute. Path absPattern = fixRelativePart(new Path( flatPattern.isEmpty() ? Path.CUR_DIR : flatPattern)); + LOG.debug("Pattern: {}", absPattern); // Now we break the flattened, absolute pattern into path components. // For example, /a/*/c would be broken into the list [a, *, c] List components = getPathComponents(absPattern.toUri().getPath()); // Starting out at the root of the filesystem, we try to match // filesystem entries against pattern components. - ArrayList candidates = new ArrayList(1); + ArrayList candidates = new ArrayList<>(1); // To get the "real" FileStatus of root, we'd have to do an expensive // RPC to the NameNode. So we create a placeholder FileStatus which has // the correct path, but defaults for the rest of the information. @@ -206,12 +261,13 @@ private FileStatus[] doGlob() throws IOException { for (int componentIdx = 0; componentIdx < components.size(); componentIdx++) { ArrayList newCandidates = - new ArrayList(candidates.size()); + new ArrayList<>(candidates.size()); GlobFilter globFilter = new GlobFilter(components.get(componentIdx)); String component = unescapePathComponent(components.get(componentIdx)); if (globFilter.hasPattern()) { sawWildcard = true; } + LOG.debug("Component {}, patterned={}", component, sawWildcard); if (candidates.isEmpty() && sawWildcard) { // Optimization: if there are no more candidates left, stop examining // the path components. We can only do this if we've already seen @@ -245,19 +301,31 @@ private FileStatus[] doGlob() throws IOException { // incorrectly conclude that /a/b was a file and should not match // /a/*/*. So we use getFileStatus of the path we just listed to // disambiguate. - Path path = candidate.getPath(); - FileStatus status = getFileStatus(path); - if (status == null) { - // null means the file was not found - LOG.warn("File/directory {} not found:" - + " it may have been deleted." - + " If this is an object store, this can be a sign of" - + " eventual consistency problems.", - path); - continue; - } - if (!status.isDirectory()) { - continue; + if (resolveSymlinks) { + LOG.debug("listStatus found one entry; disambiguating {}", + children[0]); + Path path = candidate.getPath(); + FileStatus status = getFileStatus(path); + if (status == null) { + // null means the file was not found + LOG.warn("File/directory {} not found:" + + " it may have been deleted." + + " If this is an object store, this can be a sign of" + + " eventual consistency problems.", + path); + continue; + } + if (!status.isDirectory()) { + LOG.debug("Resolved entry is a file; skipping: {}", status); + continue; + } + } else { + // there's no symlinks in this store, so no need to issue + // another call, just see if the result is a directory or a file + if (children[0].getPath().equals(candidate.getPath())) { + // the listing status is of a file + continue; + } } } for (FileStatus child : children) { @@ -312,6 +380,8 @@ private FileStatus[] doGlob() throws IOException { */ if ((!sawWildcard) && results.isEmpty() && (flattenedPatterns.size() <= 1)) { + LOG.debug("No matches found and there was no wildcard in the path {}", + pathPattern); return null; } /* @@ -324,4 +394,98 @@ private FileStatus[] doGlob() throws IOException { Arrays.sort(ret); return ret; } + + /** + * Create a builder for a Globber, bonded to the specific filesystem. + * @param filesystem filesystem + * @return the builder to finish configuring. + */ + public static GlobBuilder createGlobber(FileSystem filesystem) { + return new GlobBuilder(filesystem); + } + + /** + * Create a builder for a Globber, bonded to the specific file + * context. + * @param fileContext file context. + * @return the builder to finish configuring. + */ + public static GlobBuilder createGlobber(FileContext fileContext) { + return new GlobBuilder(fileContext); + } + + /** + * Builder for Globber instances. + */ + @InterfaceAudience.Private + public static class GlobBuilder { + + private final FileSystem fs; + + private final FileContext fc; + + private Path pathPattern; + + private PathFilter filter; + + private boolean resolveSymlinks = true; + + /** + * Construct bonded to a file context. + * @param fc file context. + */ + public GlobBuilder(final FileContext fc) { + this.fs = null; + this.fc = checkNotNull(fc); + } + + /** + * Construct bonded to a filesystem. + * @param fs file system. + */ + public GlobBuilder(final FileSystem fs) { + this.fs = checkNotNull(fs); + this.fc = null; + } + + /** + * Set the path pattern. + * @param pattern pattern to use. + * @return the builder + */ + public GlobBuilder withPathPattern(Path pattern) { + pathPattern = pattern; + return this; + } + + /** + * Set the path filter. + * @param pathFilter filter + * @return the builder + */ + public GlobBuilder withPathFiltern(PathFilter pathFilter) { + filter = pathFilter; + return this; + } + + /** + * Set the symlink resolution policy. + * @param resolve resolution flag. + * @return the builder + */ + public GlobBuilder withResolveSymlinks(boolean resolve) { + resolveSymlinks = resolve; + return this; + } + + /** + * Build the Globber. + * @return a new instance. + */ + public Globber build() { + return fs != null + ? new Globber(fs, pathPattern, filter, resolveSymlinks) + : new Globber(fc, pathPattern, filter, resolveSymlinks); + } + } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java index f7da819ed6c17..5f4c4a236e96c 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java @@ -36,6 +36,8 @@ import java.net.URLDecoder; import java.util.*; +import static org.apache.hadoop.fs.impl.PathCapabilitiesSupport.validatePathCapabilityArgs; + /** * This is an implementation of the Hadoop Archive * Filesystem. This archive Filesystem has index files @@ -899,7 +901,22 @@ public void setPermission(Path p, FsPermission permission) throws IOException { throw new IOException("Har: setPermission not allowed"); } - + + /** + * Declare that this filesystem connector is always read only. + * {@inheritDoc} + */ + @Override + public boolean hasPathCapability(final Path path, final String capability) + throws IOException { + switch (validatePathCapabilityArgs(path, capability)) { + case CommonPathCapabilities.FS_READ_ONLY_CONNECTOR: + return true; + default: + return false; + } + } + /** * Hadoop archives input stream. This input stream fakes EOF * since archive files are part of bigger part files. diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java new file mode 100644 index 0000000000000..d3492568f468f --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs; + +import java.io.IOException; + +/** + * The Path counterpoint to {@link StreamCapabilities}; a query to see if, + * a FileSystem/FileContext instance has a specific capability under the given + * path. + * Other classes may also implement the interface, as desired. + * + * See {@link CommonPathCapabilities} for the well-known capabilities. + */ +public interface PathCapabilities { + + /** + * Probe for a specific capability under the given path. + * If the function returns {@code true}, this instance is explicitly + * declaring that the capability is available. + * If the function returns {@code false}, it can mean one of: + *

      + *
    • The capability is not known.
      • + *
    • The capability is known but it is not supported.
      • + *
    • The capability is known but the filesystem does not know if it + * is supported under the supplied path.
      • + *
    + * The core guarantee which a caller can rely on is: if the predicate + * returns true, then the specific operation/behavior can be expected to be + * supported. However a specific call may be rejected for permission reasons, + * the actual file/directory not being present, or some other failure during + * the attempted execution of the operation. + *

    + * Implementors: {@link org.apache.hadoop.fs.impl.PathCapabilitiesSupport} + * can be used to help implement this method. + * @param path path to query the capability of. + * @param capability non-null, non-empty string to query the path for support. + * @return true if the capability is supported under that part of the FS. + * @throws IOException this should not be raised, except on problems + * resolving paths or relaying the call. + * @throws IllegalArgumentException invalid arguments + */ + boolean hasPathCapability(Path path, String capability) + throws IOException; +} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java index bd003ae90ab4c..cf2210575da15 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java @@ -53,6 +53,8 @@ import org.apache.hadoop.util.Shell; import org.apache.hadoop.util.StringUtils; +import static org.apache.hadoop.fs.impl.PathCapabilitiesSupport.validatePathCapabilityArgs; + /**************************************************************** * Implement the FileSystem API for the raw local filesystem. * @@ -1060,4 +1062,21 @@ public Path getLinkTarget(Path f) throws IOException { // return an unqualified symlink target return fi.getSymlink(); } + + @Override + public boolean hasPathCapability(final Path path, final String capability) + throws IOException { + switch (validatePathCapabilityArgs(makeQualified(path), capability)) { + case CommonPathCapabilities.FS_APPEND: + case CommonPathCapabilities.FS_CONCAT: + case CommonPathCapabilities.FS_PATHHANDLES: + case CommonPathCapabilities.FS_PERMISSIONS: + case CommonPathCapabilities.FS_TRUNCATE: + return true; + case CommonPathCapabilities.FS_SYMLINKS: + return FileSystem.areSymlinksEnabled(); + default: + return super.hasPathCapability(path, capability); + } + } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java index 9f07d3de1ee98..18972ea3ecf79 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java @@ -271,9 +271,9 @@ protected class Emptier implements Runnable { public void run() { if (emptierInterval == 0) return; // trash disabled - long now = Time.now(); - long end; + long now, end; while (true) { + now = Time.now(); end = ceiling(now, emptierInterval); try { // sleep for interval Thread.sleep(end - now); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/http/AbstractHttpFileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/http/AbstractHttpFileSystem.java index fa0b2cf6c314e..baf0a8187efd0 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/http/AbstractHttpFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/http/AbstractHttpFileSystem.java @@ -20,6 +20,7 @@ import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.CommonPathCapabilities; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.FileStatus; @@ -36,6 +37,8 @@ import java.net.URI; import java.net.URLConnection; +import static org.apache.hadoop.fs.impl.PathCapabilitiesSupport.validatePathCapabilityArgs; + abstract class AbstractHttpFileSystem extends FileSystem { private static final long DEFAULT_BLOCK_SIZE = 4096; private static final Path WORKING_DIR = new Path("/"); @@ -111,6 +114,21 @@ public FileStatus getFileStatus(Path path) throws IOException { return new FileStatus(-1, false, 1, DEFAULT_BLOCK_SIZE, 0, path); } + /** + * Declare that this filesystem connector is always read only. + * {@inheritDoc} + */ + @Override + public boolean hasPathCapability(final Path path, final String capability) + throws IOException { + switch (validatePathCapabilityArgs(path, capability)) { + case CommonPathCapabilities.FS_READ_ONLY_CONNECTOR: + return true; + default: + return super.hasPathCapability(path, capability); + } + } + private static class HttpDataInputStream extends FilterInputStream implements Seekable, PositionedReadable { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FsLinkResolution.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FsLinkResolution.java new file mode 100644 index 0000000000000..f5ef8c4923328 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FsLinkResolution.java @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.impl; + +import java.io.IOException; + +import com.google.common.base.Preconditions; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.fs.AbstractFileSystem; +import org.apache.hadoop.fs.FSLinkResolver; +import org.apache.hadoop.fs.FileContext; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.UnresolvedLinkException; + +/** + * Class to allow Lambda expressions to be used in {@link FileContext} + * link resolution. + * @param type of the returned value. + */ +@InterfaceAudience.Private +@InterfaceStability.Evolving +public class FsLinkResolution extends FSLinkResolver { + + /** + * The function to invoke in the {@link #next(AbstractFileSystem, Path)} call. + */ + private final FsLinkResolutionFunction fn; + + /** + * Construct an instance with the given function. + * @param fn function to invoke. + */ + public FsLinkResolution(final FsLinkResolutionFunction fn) { + this.fn = Preconditions.checkNotNull(fn); + } + + @Override + public T next(final AbstractFileSystem fs, final Path p) + throws UnresolvedLinkException, IOException { + return fn.apply(fs, p); + } + + /** + * The signature of the function to invoke. + * @param type resolved to + */ + @FunctionalInterface + public interface FsLinkResolutionFunction { + + /** + * + * @param fs filesystem to resolve against. + * @param path path to resolve + * @return a result of type T + * @throws UnresolvedLinkException link resolution failure + * @throws IOException other IO failure. + */ + T apply(final AbstractFileSystem fs, final Path path) + throws IOException, UnresolvedLinkException; + } + + /** + * Apply the given function to the resolved path under the the supplied + * FileContext. + * @param fileContext file context to resolve under + * @param path path to resolve + * @param fn function to invoke + * @param return type. + * @return the return value of the function as revoked against the resolved + * path. + * @throws UnresolvedLinkException link resolution failure + * @throws IOException other IO failure. + */ + public static T resolve( + final FileContext fileContext, final Path path, + final FsLinkResolutionFunction fn) + throws UnresolvedLinkException, IOException { + return new FsLinkResolution<>(fn).resolve(fileContext, path); + } +} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java new file mode 100644 index 0000000000000..7bbb34622647d --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.impl; + +import java.io.IOException; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; + +/** + * Evolving support for functional programming/lambda-expressions. + */ +@InterfaceAudience.Private +@InterfaceStability.Unstable +public final class FunctionsRaisingIOE { + + private FunctionsRaisingIOE() { + } + + /** + * Function of arity 1 which may raise an IOException. + * @param type of arg1 + * @param type of return value. + */ + @FunctionalInterface + public interface FunctionRaisingIOE { + + R apply(T t) throws IOException; + } + + /** + * Function of arity 2 which may raise an IOException. + * @param type of arg1 + * @param type of arg2 + * @param type of return value. + */ + @FunctionalInterface + public interface BiFunctionRaisingIOE { + + R apply(T t, U u) throws IOException; + } + + /** + * This is a callable which only raises an IOException. + * @param return type + */ + @FunctionalInterface + public interface CallableRaisingIOE { + + R apply() throws IOException; + } + +} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java index 9d5f2bf4b6ed1..26856e5b935e0 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java @@ -21,6 +21,7 @@ import java.io.IOException; import java.io.InterruptedIOException; import java.util.Map; +import java.util.concurrent.CompletionException; import java.util.concurrent.ExecutionException; import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; @@ -108,20 +109,55 @@ public static T awaitFuture(final Future future, */ public static T raiseInnerCause(final ExecutionException e) throws IOException { + throw unwrapInnerException(e); + } + + /** + * Extract the cause of a completion failure and rethrow it if an IOE + * or RTE. + * @param e exception. + * @param type of return value. + * @return nothing, ever. + * @throws IOException either the inner IOException, or a wrapper around + * any non-Runtime-Exception + * @throws RuntimeException if that is the inner cause. + */ + public static T raiseInnerCause(final CompletionException e) + throws IOException { + throw unwrapInnerException(e); + } + + /** + * From the inner cause of an execution exception, extract the inner cause. + * If it is an RTE: throw immediately. + * If it is an IOE: Return. + * If it is a WrappedIOException: Unwrap and return + * Else: create a new IOException. + * + * Recursively handles wrapped Execution and Completion Exceptions in + * case something very complicated has happened. + * @param e exception. + * @return an IOException extracted or built from the cause. + * @throws RuntimeException if that is the inner cause. + */ + private static IOException unwrapInnerException(final Throwable e) { Throwable cause = e.getCause(); if (cause instanceof IOException) { - throw (IOException) cause; - } else if (cause instanceof WrappedIOException){ - throw ((WrappedIOException) cause).getCause(); - } else if (cause instanceof RuntimeException){ + return (IOException) cause; + } else if (cause instanceof WrappedIOException) { + return ((WrappedIOException) cause).getCause(); + } else if (cause instanceof CompletionException) { + return unwrapInnerException(cause); + } else if (cause instanceof ExecutionException) { + return unwrapInnerException(cause); + } else if (cause instanceof RuntimeException) { throw (RuntimeException) cause; } else if (cause != null) { // other type: wrap with a new IOE - throw new IOException(cause); + return new IOException(cause); } else { - // this only happens if somebody deliberately raises - // an ExecutionException - throw new IOException(e); + // this only happens if there was no cause. + return new IOException(e); } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/PathCapabilitiesSupport.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/PathCapabilitiesSupport.java new file mode 100644 index 0000000000000..9332ac6e7eedb --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/PathCapabilitiesSupport.java @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.impl; + +import java.util.Locale; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.PathCapabilities; + +import static com.google.common.base.Preconditions.checkArgument; + +@InterfaceAudience.Private +@InterfaceStability.Evolving +public class PathCapabilitiesSupport { + + /** + * Validate the arguments to + * {@link PathCapabilities#hasPathCapability(Path, String)}. + * @param path path to query the capability of. + * @param capability non-null, non-empty string to query the path for support. + * @return the string to use in a switch statement. + * @throws IllegalArgumentException if a an argument is invalid. + */ + public static String validatePathCapabilityArgs( + final Path path, final String capability) { + checkArgument(path != null, "null path"); + checkArgument(capability != null, "capability parameter is null"); + checkArgument(!capability.isEmpty(), + "capability parameter is empty string"); + return capability.toLowerCase(Locale.ENGLISH); + } +} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/AclCommands.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/AclCommands.java index 701c9deb9c2cf..5a60ef2ae9b03 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/AclCommands.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/AclCommands.java @@ -173,7 +173,11 @@ public static class SetfaclCommand extends FsCommand { + " -x :Remove specified ACL entries. Other ACL entries are retained.\n" + " --set :Fully replace the ACL, discarding all existing entries." + " The must include entries for user, group, and others" - + " for compatibility with permission bits.\n" + + " for compatibility with permission bits. If the ACL spec contains" + + " only access entries, then the existing default entries are retained" + + ". If the ACL spec contains only default entries, then the existing" + + " access entries are retained. If the ACL spec contains both access" + + " and default entries, then both are replaced.\n" + " : Comma separated list of ACL entries.\n" + " : File or directory to modify.\n"; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Command.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Command.java index 3eef2787e7e74..c81825776a613 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Command.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Command.java @@ -458,7 +458,7 @@ public void displayError(Exception e) { if (e instanceof InterruptedIOException) { throw new CommandInterruptException(); } - + LOG.debug("{} failure", getName(), e); String errorMessage = e.getLocalizedMessage(); if (errorMessage == null) { // this is an unexpected condition, so dump the whole exception since diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java index 2421f06dd9461..0802a00b01bc8 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java @@ -30,6 +30,9 @@ import java.util.Map.Entry; import java.util.NoSuchElementException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import org.apache.hadoop.fs.CreateFlag; import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.FileSystem; @@ -58,7 +61,11 @@ * a source and resolved target. Sources are resolved as children of * a destination directory. */ -abstract class CommandWithDestination extends FsCommand { +abstract class CommandWithDestination extends FsCommand { + + protected static final Logger LOG = LoggerFactory.getLogger( + CommandWithDestination.class); + protected PathData dst; private boolean overwrite = false; private boolean verifyChecksum = true; @@ -220,6 +227,7 @@ protected void processArguments(LinkedList args) } } else if (dst.exists) { if (!dst.stat.isDirectory() && !overwrite) { + LOG.debug("Destination file exists: {}", dst.stat); throw new PathExistsException(dst.toString()); } } else if (!dst.parentExists()) { @@ -407,6 +415,7 @@ protected void copyStreamToTarget(InputStream in, PathData target) targetFs.setWriteChecksum(writeChecksum); targetFs.writeStreamToFile(in, tempTarget, lazyPersist, direct); if (!direct) { + targetFs.deleteOnExit(tempTarget.path); targetFs.rename(tempTarget, target); } } finally { @@ -484,6 +493,15 @@ void writeStreamToFile(InputStream in, PathData target, try { out = create(target, lazyPersist, direct); IOUtils.copyBytes(in, out, getConf(), true); + } catch (IOException e) { + // failure: clean up if we got as far as creating the file + if (!direct && out != null) { + try { + fs.delete(target.path, false); + } catch (IOException ignored) { + } + } + throw e; } finally { IOUtils.closeStream(out); // just in case copyBytes didn't } @@ -493,37 +511,31 @@ void writeStreamToFile(InputStream in, PathData target, FSDataOutputStream create(PathData item, boolean lazyPersist, boolean direct) throws IOException { - try { - if (lazyPersist) { - long defaultBlockSize; - try { - defaultBlockSize = getDefaultBlockSize(); - } catch (NotInMountpointException ex) { - // ViewFileSystem#getDefaultBlockSize() throws an exception as it - // needs a target FS to retrive the default block size from. - // Hence, for ViewFs, we should call getDefaultBlockSize with the - // target path. - defaultBlockSize = getDefaultBlockSize(item.path); - } - - EnumSet createFlags = EnumSet.of(CREATE, LAZY_PERSIST); - return create(item.path, - FsPermission.getFileDefault().applyUMask( - FsPermission.getUMask(getConf())), - createFlags, - getConf().getInt(IO_FILE_BUFFER_SIZE_KEY, - IO_FILE_BUFFER_SIZE_DEFAULT), - (short) 1, - defaultBlockSize, - null, - null); - } else { - return create(item.path, true); - } - } finally { // might have been created but stream was interrupted - if (!direct) { - deleteOnExit(item.path); + if (lazyPersist) { + long defaultBlockSize; + try { + defaultBlockSize = getDefaultBlockSize(); + } catch (NotInMountpointException ex) { + // ViewFileSystem#getDefaultBlockSize() throws an exception as it + // needs a target FS to retrive the default block size from. + // Hence, for ViewFs, we should call getDefaultBlockSize with the + // target path. + defaultBlockSize = getDefaultBlockSize(item.path); } + + EnumSet createFlags = EnumSet.of(CREATE, LAZY_PERSIST); + return create(item.path, + FsPermission.getFileDefault().applyUMask( + FsPermission.getUMask(getConf())), + createFlags, + getConf().getInt(IO_FILE_BUFFER_SIZE_KEY, + IO_FILE_BUFFER_SIZE_DEFAULT), + (short) 1, + defaultBlockSize, + null, + null); + } else { + return create(item.path, true); } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Delete.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Delete.java index 57b543acc21e6..3c9368ca2ed9b 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Delete.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Delete.java @@ -25,6 +25,7 @@ import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.fs.CommonConfigurationKeysPublic; import org.apache.hadoop.fs.ContentSummary; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.PathIOException; @@ -218,26 +219,35 @@ protected void processPath(PathData item) throws IOException { // delete files from the trash that are older // than the retention threshold. static class Expunge extends FsCommand { + private static final String OPTION_FILESYSTEM = "fs"; + public static final String NAME = "expunge"; public static final String USAGE = - "[-immediate]"; + "[-immediate] [-" + OPTION_FILESYSTEM + " ]"; public static final String DESCRIPTION = "Delete files from the trash that are older " + "than the retention threshold"; private boolean emptyImmediately = false; + private String fsArgument; - // TODO: should probably allow path arguments for the filesystems @Override protected void processOptions(LinkedList args) throws IOException { - CommandFormat cf = new CommandFormat(0, 1, "immediate"); + CommandFormat cf = new CommandFormat(0, 2, "immediate"); + cf.addOptionWithValue(OPTION_FILESYSTEM); cf.parse(args); emptyImmediately = cf.getOpt("immediate"); + fsArgument = cf.getOptValue(OPTION_FILESYSTEM); } @Override protected void processArguments(LinkedList args) throws IOException { + if (fsArgument != null && fsArgument.length() != 0) { + getConf().set( + CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, fsArgument); + } + FileSystem[] childFileSystems = FileSystem.get(getConf()).getChildFileSystems(); if (null != childFileSystems) { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java index 5828b0bbf4ddd..6380d0ca788cd 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java @@ -73,6 +73,13 @@ protected void processNonexistentPath(PathData item) throws IOException { // we want a/b final Path itemPath = new Path(item.path.toString()); final Path itemParentPath = itemPath.getParent(); + + if(itemParentPath == null) { + throw new PathNotFoundException(String.format( + "Item: %s parent's path is null. This can happen if mkdir is " + + "called on root, so there's no parent.", itemPath.toString())); + } + if (!item.fs.exists(itemParentPath)) { throw new PathNotFoundException(itemParentPath.toString()); } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java index a2d2529408580..b3acc986d6409 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java @@ -39,7 +39,7 @@ public static void registerCommands(CommandFactory factory) { } public static final String NAME = "test"; - public static final String USAGE = "-[defsz] "; + public static final String USAGE = "-[defswrz] "; public static final String DESCRIPTION = "Answer various questions about , with result via exit status.\n" + " -d return 0 if is a directory.\n" diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java index cf0b84ca366f1..773a7b2def265 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java @@ -98,13 +98,12 @@ protected Path fullPath(final Path path) { /** * Constructor - * @param uri base file system - * @param conf configuration + * @param fs base file system + * @param uri base uri * @throws IOException */ - public ChRootedFileSystem(final URI uri, Configuration conf) - throws IOException { - super(FileSystem.get(uri, conf)); + ChRootedFileSystem(final FileSystem fs, URI uri) throws IOException { + super(fs); String pathString = uri.getPath(); if (pathString.isEmpty()) { pathString = "/"; @@ -115,7 +114,18 @@ public ChRootedFileSystem(final URI uri, Configuration conf) workingDir = getHomeDirectory(); // We don't use the wd of the myFs } - + + /** + * Constructor. + * @param uri base file system + * @param conf configuration + * @throws IOException + */ + public ChRootedFileSystem(final URI uri, Configuration conf) + throws IOException { + this(FileSystem.get(uri, conf), uri); + } + /** * Called after a new FileSystem instance is constructed. * @param name a uri whose authority section names the host, port, etc. @@ -269,6 +279,11 @@ public boolean mkdirs(final Path f, final FsPermission permission) return super.mkdirs(fullPath(f), permission); } + @Override + public boolean mkdirs(final Path f) throws IOException { + return super.mkdirs(fullPath(f)); + } + @Override public FSDataInputStream open(final Path f, final int bufferSize) throws IOException { @@ -476,4 +491,10 @@ public FutureDataInputStreamBuilder openFile(final Path path) throws IOException, UnsupportedOperationException { return super.openFile(fullPath(path)); } + + @Override + public boolean hasPathCapability(final Path path, final String capability) + throws IOException { + return super.hasPathCapability(fullPath(path), capability); + } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java index aa1bc7e63771a..37f1a16800e7d 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java @@ -78,4 +78,11 @@ public interface Constants { FsPermission PERMISSION_555 = new FsPermission((short) 0555); String CONFIG_VIEWFS_RENAME_STRATEGY = "fs.viewfs.rename.strategy"; + + /** + * Enable ViewFileSystem to cache all children filesystems in inner cache. + */ + String CONFIG_VIEWFS_ENABLE_INNER_CACHE = "fs.viewfs.enable.inner.cache"; + + boolean CONFIG_VIEWFS_ENABLE_INNER_CACHE_DEFAULT = true; } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java index 276c07e3576de..faa374a39789b 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java @@ -17,7 +17,10 @@ */ package org.apache.hadoop.fs.viewfs; +import static org.apache.hadoop.fs.impl.PathCapabilitiesSupport.validatePathCapabilityArgs; import static org.apache.hadoop.fs.viewfs.Constants.PERMISSION_555; +import static org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE; +import static org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE_DEFAULT; import java.io.FileNotFoundException; import java.io.IOException; @@ -26,10 +29,13 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.Collection; +import java.util.Collections; import java.util.EnumSet; +import java.util.HashMap; import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.Objects; import java.util.Set; import java.util.Map.Entry; @@ -38,6 +44,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.BlockLocation; import org.apache.hadoop.fs.BlockStoragePolicySpi; +import org.apache.hadoop.fs.CommonPathCapabilities; import org.apache.hadoop.fs.ContentSummary; import org.apache.hadoop.fs.CreateFlag; import org.apache.hadoop.fs.FSDataInputStream; @@ -89,6 +96,72 @@ static AccessControlException readOnlyMountTable(final String operation, return readOnlyMountTable(operation, p.toString()); } + /** + * Caching children filesystems. HADOOP-15565. + */ + static class InnerCache { + private Map map = new HashMap<>(); + + FileSystem get(URI uri, Configuration config) throws IOException { + Key key = new Key(uri); + if (map.get(key) == null) { + FileSystem fs = FileSystem.newInstance(uri, config); + map.put(key, fs); + return fs; + } else { + return map.get(key); + } + } + + void closeAll() { + for (FileSystem fs : map.values()) { + try { + fs.close(); + } catch (IOException e) { + LOG.info("Fail closing ViewFileSystem's child filesystem " + fs, e); + } + } + } + + InnerCache unmodifiableCache() { + map = Collections.unmodifiableMap(map); + return this; + } + + /** + * All the cached instances share the same UGI so there is no need to have a + * URI in the Key. Make the Key simple with just the scheme and authority. + */ + private static class Key { + private final String scheme; + private final String authority; + + Key(URI uri) { + scheme = uri.getScheme() == null ? "" : uri.getScheme().toLowerCase(); + authority = + uri.getAuthority() == null ? "" : uri.getAuthority().toLowerCase(); + } + + @Override + public int hashCode() { + return Objects.hash(scheme, authority); + } + + @Override + public boolean equals(Object obj) { + if (obj == this) { + return true; + } + if (obj != null && obj instanceof Key) { + Key that = (Key) obj; + return this.scheme.equals(that.scheme) && this.authority + .equals(that.authority); + } + return false; + } + } + } + /** * MountPoint representation built from the configuration. */ @@ -125,6 +198,8 @@ public URI[] getTargetFileSystemURIs() { Configuration config; InodeTree fsState; // the fs state; ie the mount table Path homeDir = null; + private boolean enableInnerCache = false; + private InnerCache cache; // Default to rename within same mountpoint private RenameStrategy renameStrategy = RenameStrategy.SAME_MOUNTPOINT; /** @@ -178,6 +253,9 @@ public void initialize(final URI theUri, final Configuration conf) super.initialize(theUri, conf); setConf(conf); config = conf; + enableInnerCache = config.getBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, + CONFIG_VIEWFS_ENABLE_INNER_CACHE_DEFAULT); + final InnerCache innerCache = new InnerCache(); // Now build client side view (i.e. client side mount table) from config. final String authority = theUri.getAuthority(); try { @@ -187,7 +265,13 @@ public void initialize(final URI theUri, final Configuration conf) @Override protected FileSystem getTargetFileSystem(final URI uri) throws URISyntaxException, IOException { - return new ChRootedFileSystem(uri, config); + FileSystem fs; + if (enableInnerCache) { + fs = innerCache.get(uri, config); + } else { + fs = FileSystem.get(uri, config); + } + return new ChRootedFileSystem(fs, uri); } @Override @@ -210,6 +294,12 @@ protected FileSystem getTargetFileSystem(final String settings, throw new IOException("URISyntax exception: " + theUri); } + if (enableInnerCache) { + // All fs instances are created and cached on startup. The cache is + // readonly after the initialize() so the concurrent access of the cache + // is safe. + cache = innerCache.unmodifiableCache(); + } } /** @@ -461,12 +551,19 @@ private Path getChrootedPath(InodeTree.ResolveResult res, suffix.length() == 0 ? f : new Path(res.resolvedPath, suffix)); } + @Override + public boolean mkdirs(Path dir) throws IOException { + InodeTree.ResolveResult res = + fsState.resolve(getUriPath(dir), false); + return res.targetFileSystem.mkdirs(res.remainingPath); + } + @Override public boolean mkdirs(final Path dir, final FsPermission permission) throws IOException { InodeTree.ResolveResult res = - fsState.resolve(getUriPath(dir), false); - return res.targetFileSystem.mkdirs(res.remainingPath, permission); + fsState.resolve(getUriPath(dir), false); + return res.targetFileSystem.mkdirs(res.remainingPath, permission); } @Override @@ -931,6 +1028,36 @@ public Path getLinkTarget(Path path) throws IOException { return res.targetFileSystem.getLinkTarget(res.remainingPath); } + /** + * Reject the concat operation; forward the rest to the viewed FS. + * @param path path to query the capability of. + * @param capability string to query the stream support for. + * @return the capability + * @throws IOException if there is no resolved FS, or it raises an IOE. + */ + @Override + public boolean hasPathCapability(Path path, String capability) + throws IOException { + final Path p = makeQualified(path); + switch (validatePathCapabilityArgs(p, capability)) { + case CommonPathCapabilities.FS_CONCAT: + // concat is not supported, as it may be invoked across filesystems. + return false; + default: + // no break + } + // otherwise, check capabilities of mounted FS. + try { + InodeTree.ResolveResult res + = fsState.resolve(getUriPath(p), true); + return res.targetFileSystem.hasPathCapability(res.remainingPath, + capability); + } catch (FileNotFoundException e) { + // no mount point, nothing will work. + throw new NotInMountpointException(p, "hasPathCapability"); + } + } + /** * An instance of this class represents an internal dir of the viewFs * that is internal dir of the mount table. @@ -1083,6 +1210,12 @@ public boolean mkdirs(Path dir, FsPermission permission) throw readOnlyMountTable("mkdirs", dir); } + @Override + public boolean mkdirs(Path dir) + throws AccessControlException, FileAlreadyExistsException { + return mkdirs(dir, null); + } + @Override public FSDataInputStream open(Path f, int bufferSize) throws AccessControlException, FileNotFoundException, IOException { @@ -1297,4 +1430,12 @@ enum RenameStrategy { SAME_MOUNTPOINT, SAME_TARGET_URI_ACROSS_MOUNTPOINT, SAME_FILESYSTEM_ACROSS_MOUNTPOINT } + + @Override + public void close() throws IOException { + super.close(); + if (enableInnerCache && cache != null) { + cache.closeAll(); + } + } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java index 4a2a21bafb018..9d5c8e7b7ea3b 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java @@ -107,19 +107,30 @@ public HAServiceProtocol getProxy(Configuration conf, int timeoutMs) */ public HAServiceProtocol getHealthMonitorProxy(Configuration conf, int timeoutMs) throws IOException { + return getHealthMonitorProxy(conf, timeoutMs, 1); + } + + public HAServiceProtocol getHealthMonitorProxy(Configuration conf, + int timeoutMs, int retries) throws IOException { InetSocketAddress addr = getHealthMonitorAddress(); if (addr == null) { addr = getAddress(); } - return getProxyForAddress(conf, timeoutMs, addr); + return getProxyForAddress(conf, timeoutMs, retries, addr); } private HAServiceProtocol getProxyForAddress(Configuration conf, int timeoutMs, InetSocketAddress addr) throws IOException { + // Lower the timeout by setting retries to 1, so we quickly fail to connect + return getProxyForAddress(conf, timeoutMs, 1, addr); + } + + private HAServiceProtocol getProxyForAddress(Configuration conf, + int timeoutMs, int retries, InetSocketAddress addr) throws IOException { Configuration confCopy = new Configuration(conf); - // Lower the timeout so we quickly fail to connect confCopy.setInt( - CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_KEY, 1); + CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_KEY, + retries); SocketFactory factory = NetUtils.getDefaultSocketFactory(confCopy); return new HAServiceProtocolClientSideTranslatorPB( addr, diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java index d1a858fd85f68..16c30752edc20 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java @@ -55,6 +55,7 @@ public class HealthMonitor { private long checkIntervalMillis; private long sleepAfterDisconnectMillis; + private int rpcConnectRetries; private int rpcTimeout; private volatile boolean shouldRun = true; @@ -124,6 +125,8 @@ public enum State { this.connectRetryInterval = conf.getLong( HA_HM_CONNECT_RETRY_INTERVAL_KEY, HA_HM_CONNECT_RETRY_INTERVAL_DEFAULT); + this.rpcConnectRetries = conf.getInt(HA_HM_RPC_CONNECT_MAX_RETRIES_KEY, + HA_HM_RPC_CONNECT_MAX_RETRIES_DEFAULT); this.rpcTimeout = conf.getInt( HA_HM_RPC_TIMEOUT_KEY, HA_HM_RPC_TIMEOUT_DEFAULT); @@ -134,19 +137,11 @@ public enum State { public void addCallback(Callback cb) { this.callbacks.add(cb); } - - public void removeCallback(Callback cb) { - callbacks.remove(cb); - } public synchronized void addServiceStateCallback(ServiceStateCallback cb) { this.serviceStateCallbacks.add(cb); } - public synchronized void removeServiceStateCallback(ServiceStateCallback cb) { - serviceStateCallbacks.remove(cb); - } - public void shutdown() { LOG.info("Stopping HealthMonitor thread"); shouldRun = false; @@ -191,7 +186,7 @@ private void tryConnect() { * Connect to the service to be monitored. Stubbed out for easier testing. */ protected HAServiceProtocol createProxy() throws IOException { - return targetToMonitor.getHealthMonitorProxy(conf, rpcTimeout); + return targetToMonitor.getHealthMonitorProxy(conf, rpcTimeout, rpcConnectRetries); } private void doHealthChecks() throws InterruptedException { @@ -258,11 +253,7 @@ private synchronized void enterState(State newState) { synchronized State getHealthState() { return state; } - - synchronized HAServiceStatus getLastServiceStatus() { - return lastServiceState; - } - + boolean isAlive() { return daemon.isAlive(); } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java index d568376d302af..ee4ca1a6084a9 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java @@ -442,14 +442,16 @@ private void recordActiveAttempt( * * * @param timeoutMillis number of millis to wait + * @param onlyAfterNanoTime accept attempt records only after a given + * timestamp. Use this parameter to ignore the old attempt records from a + * previous fail-over attempt. * @return the published record, or null if the timeout elapses or the * service becomes unhealthy * @throws InterruptedException if the thread is interrupted. */ - private ActiveAttemptRecord waitForActiveAttempt(int timeoutMillis) - throws InterruptedException { - long st = System.nanoTime(); - long waitUntil = st + TimeUnit.NANOSECONDS.convert( + private ActiveAttemptRecord waitForActiveAttempt(int timeoutMillis, + long onlyAfterNanoTime) throws InterruptedException { + long waitUntil = onlyAfterNanoTime + TimeUnit.NANOSECONDS.convert( timeoutMillis, TimeUnit.MILLISECONDS); do { @@ -466,7 +468,7 @@ private ActiveAttemptRecord waitForActiveAttempt(int timeoutMillis) synchronized (activeAttemptRecordLock) { if ((lastActiveAttemptRecord != null && - lastActiveAttemptRecord.nanoTime >= st)) { + lastActiveAttemptRecord.nanoTime >= onlyAfterNanoTime)) { return lastActiveAttemptRecord; } // Only wait 1sec so that we periodically recheck the health state @@ -660,6 +662,7 @@ private void doGracefulFailover() List otherZkfcs = new ArrayList(otherNodes.size()); // Phase 3: ask the other nodes to yield from the election. + long st = System.nanoTime(); HAServiceTarget activeNode = null; for (HAServiceTarget remote : otherNodes) { // same location, same node - may not always be == equality @@ -678,7 +681,7 @@ private void doGracefulFailover() // Phase 4: wait for the normal election to make the local node // active. - ActiveAttemptRecord attempt = waitForActiveAttempt(timeout + 60000); + ActiveAttemptRecord attempt = waitForActiveAttempt(timeout + 60000, st); if (attempt == null) { // We didn't even make an attempt to become active. diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpRequestLog.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpRequestLog.java index 05573a8de952d..b2f18538b6c7d 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpRequestLog.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpRequestLog.java @@ -24,7 +24,8 @@ import org.apache.commons.logging.LogConfigurationException; import org.apache.commons.logging.LogFactory; import org.apache.log4j.Appender; -import org.eclipse.jetty.server.NCSARequestLog; +import org.eclipse.jetty.server.AsyncRequestLogWriter; +import org.eclipse.jetty.server.CustomRequestLog; import org.eclipse.jetty.server.RequestLog; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -85,10 +86,11 @@ public static RequestLog getRequestLog(String name) { if (appender instanceof HttpRequestLogAppender) { HttpRequestLogAppender requestLogAppender = (HttpRequestLogAppender)appender; - NCSARequestLog requestLog = new NCSARequestLog(); - requestLog.setFilename(requestLogAppender.getFilename()); - requestLog.setRetainDays(requestLogAppender.getRetainDays()); - return requestLog; + AsyncRequestLogWriter logWriter = new AsyncRequestLogWriter(); + logWriter.setFilename(requestLogAppender.getFilename()); + logWriter.setRetainDays(requestLogAppender.getRetainDays()); + return new CustomRequestLog(logWriter, + CustomRequestLog.EXTENDED_NCSA_FORMAT); } else { LOG.warn("Jetty request log for {} was of the wrong class", loggerName); return null; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java index fb2dff5d02d05..7e7d64423f169 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java @@ -27,6 +27,7 @@ import java.net.MalformedURLException; import java.net.URI; import java.net.URL; +import java.util.Arrays; import java.util.ArrayList; import java.util.Collections; import java.util.Enumeration; @@ -60,12 +61,17 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configuration.IntegerRanges; import org.apache.hadoop.fs.CommonConfigurationKeys; +import org.apache.hadoop.fs.CommonConfigurationKeysPublic; import org.apache.hadoop.jmx.JMXJsonServlet; import org.apache.hadoop.log.LogLevel; +import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; +import org.apache.hadoop.metrics2.sink.PrometheusMetricsSink; import org.apache.hadoop.security.AuthenticationFilterInitializer; import org.apache.hadoop.security.SecurityUtil; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.security.authentication.server.AuthenticationFilter; +import org.apache.hadoop.security.authentication.server.ProxyUserAuthenticationFilterInitializer; +import org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler; import org.apache.hadoop.security.authentication.util.SignerSecretProvider; import org.apache.hadoop.security.authorize.AccessControlList; import org.apache.hadoop.security.ssl.SSLFactory; @@ -82,15 +88,12 @@ import org.eclipse.jetty.server.SecureRequestCustomizer; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; -import org.eclipse.jetty.server.SessionManager; import org.eclipse.jetty.server.SslConnectionFactory; import org.eclipse.jetty.server.handler.AllowSymLinkAliasChecker; import org.eclipse.jetty.server.handler.ContextHandlerCollection; import org.eclipse.jetty.server.handler.HandlerCollection; import org.eclipse.jetty.server.handler.RequestLogHandler; -import org.eclipse.jetty.server.session.AbstractSessionManager; import org.eclipse.jetty.server.session.SessionHandler; -import org.eclipse.jetty.servlet.DefaultServlet; import org.eclipse.jetty.servlet.FilterHolder; import org.eclipse.jetty.servlet.FilterMapping; import org.eclipse.jetty.servlet.ServletContextHandler; @@ -155,7 +158,7 @@ public final class HttpServer2 implements FilterContainer { // gets stored. public static final String CONF_CONTEXT_ATTRIBUTE = "hadoop.conf"; public static final String ADMINS_ACL = "admins.acl"; - public static final String SPNEGO_FILTER = "SpnegoFilter"; + public static final String SPNEGO_FILTER = "authentication"; public static final String NO_CACHE_FILTER = "NoCacheFilter"; public static final String BIND_ADDRESS = "bind.address"; @@ -189,6 +192,11 @@ public final class HttpServer2 implements FilterContainer { private static final String X_FRAME_OPTIONS = "X-FRAME-OPTIONS"; private static final Pattern PATTERN_HTTP_HEADER_REGEX = Pattern.compile(HTTP_HEADER_REGEX); + + private boolean prometheusSupport; + protected static final String PROMETHEUS_SINK = "PROMETHEUS_SINK"; + private PrometheusMetricsSink prometheusMetricsSink; + /** * Class to construct instances of HTTP server with specific options. */ @@ -433,7 +441,9 @@ public HttpServer2 build() throws IOException { HttpServer2 server = new HttpServer2(this); - if (this.securityEnabled) { + if (this.securityEnabled && + !this.conf.get(authFilterConfigurationPrefix + "type"). + equals(PseudoAuthenticationHandler.TYPE)) { server.initSpnego(conf, hostName, usernameConfKey, keytabConfKey); } @@ -507,7 +517,8 @@ private ServerConnector createHttpsChannelConnector( httpConfig.addCustomizer(new SecureRequestCustomizer()); ServerConnector conn = createHttpChannelConnector(server, httpConfig); - SslContextFactory sslContextFactory = new SslContextFactory(); + SslContextFactory.Server sslContextFactory = + new SslContextFactory.Server(); sslContextFactory.setNeedClientAuth(needsClientAuth); sslContextFactory.setKeyManagerPassword(keyPassword); if (keyStore != null) { @@ -526,11 +537,45 @@ private ServerConnector createHttpsChannelConnector( LOG.info("Excluded Cipher List:" + excludeCiphers); } + setEnabledProtocols(sslContextFactory); conn.addFirstConnectionFactory(new SslConnectionFactory(sslContextFactory, HttpVersion.HTTP_1_1.asString())); return conn; } + + private void setEnabledProtocols(SslContextFactory sslContextFactory) { + String enabledProtocols = conf.get(SSLFactory.SSL_ENABLED_PROTOCOLS_KEY, + SSLFactory.SSL_ENABLED_PROTOCOLS_DEFAULT); + if (!enabledProtocols.equals(SSLFactory.SSL_ENABLED_PROTOCOLS_DEFAULT)) { + // Jetty 9.2.4.v20141103 and above excludes certain protocols by + // default. Remove the user enabled protocols from the exclude list, + // and add them into the include list. + String[] jettyExcludedProtocols = + sslContextFactory.getExcludeProtocols(); + String[] enabledProtocolsArray = + StringUtils.getTrimmedStrings(enabledProtocols); + List enabledProtocolsList = + Arrays.asList(enabledProtocolsArray); + + List resetExcludedProtocols = new ArrayList<>(); + for (String jettyExcludedProtocol: jettyExcludedProtocols) { + if (!enabledProtocolsList.contains(jettyExcludedProtocol)) { + resetExcludedProtocols.add(jettyExcludedProtocol); + } else { + LOG.debug("Removed {} from exclude protocol list", + jettyExcludedProtocol); + } + } + + sslContextFactory.setExcludeProtocols( + resetExcludedProtocols.toArray(new String[0])); + LOG.info("Reset exclude protocol list: {}", resetExcludedProtocols); + + sslContextFactory.setIncludeProtocols(enabledProtocolsArray); + LOG.info("Enabled protocols: {}", enabledProtocols); + } + } } private HttpServer2(final Builder b) throws IOException { @@ -575,12 +620,9 @@ private void initializeWebServer(String name, String hostName, threadPool.setMaxThreads(maxThreads); } - SessionManager sm = webAppContext.getSessionHandler().getSessionManager(); - if (sm instanceof AbstractSessionManager) { - AbstractSessionManager asm = (AbstractSessionManager)sm; - asm.setHttpOnly(true); - asm.getSessionCookieConfig().setSecure(true); - } + SessionHandler handler = webAppContext.getSessionHandler(); + handler.setHttpOnly(true); + handler.getSessionCookieConfig().setSecure(true); ContextHandlerCollection contexts = new ContextHandlerCollection(); RequestLog requestLog = HttpRequestLog.getRequestLog(name); @@ -608,12 +650,18 @@ private void initializeWebServer(String name, String hostName, } addDefaultServlets(); + addPrometheusServlet(conf); + } - if (pathSpecs != null) { - for (String path : pathSpecs) { - LOG.info("adding path spec: " + path); - addFilterPathMapping(path, webAppContext); - } + private void addPrometheusServlet(Configuration conf) { + prometheusSupport = conf.getBoolean( + CommonConfigurationKeysPublic.HADOOP_PROMETHEUS_ENABLED, + CommonConfigurationKeysPublic.HADOOP_PROMETHEUS_ENABLED_DEFAULT); + if (prometheusSupport) { + prometheusMetricsSink = new PrometheusMetricsSink(); + getWebAppContext().getServletContext() + .setAttribute(PROMETHEUS_SINK, prometheusMetricsSink); + addServlet("prometheus", "/prom", PrometheusServlet.class); } } @@ -625,7 +673,7 @@ private static WebAppContext createWebAppContext(Builder b, AccessControlList adminsAcl, final String appDir) { WebAppContext ctx = new WebAppContext(); ctx.setDefaultsDescriptor(null); - ServletHolder holder = new ServletHolder(new DefaultServlet()); + ServletHolder holder = new ServletHolder(new WebServlet()); Map params = ImmutableMap. builder() .put("acceptRanges", "true") .put("dirAllowed", "false") @@ -684,10 +732,16 @@ private static FilterInitializer[] getFilterInitializers(Configuration conf) { return null; } - FilterInitializer[] initializers = new FilterInitializer[classes.length]; - for(int i = 0; i < classes.length; i++) { + List> classList = new ArrayList<>(Arrays.asList(classes)); + if (classList.contains(AuthenticationFilterInitializer.class) && + classList.contains(ProxyUserAuthenticationFilterInitializer.class)) { + classList.remove(AuthenticationFilterInitializer.class); + } + + FilterInitializer[] initializers = new FilterInitializer[classList.size()]; + for(int i = 0; i < classList.size(); i++) { initializers[i] = (FilterInitializer)ReflectionUtils.newInstance( - classes[i], conf); + classList.get(i), conf); } return initializers; } @@ -719,12 +773,8 @@ protected void addDefaultApps(ContextHandlerCollection parent, } logContext.setDisplayName("logs"); SessionHandler handler = new SessionHandler(); - SessionManager sm = handler.getSessionManager(); - if (sm instanceof AbstractSessionManager) { - AbstractSessionManager asm = (AbstractSessionManager) sm; - asm.setHttpOnly(true); - asm.getSessionCookieConfig().setSecure(true); - } + handler.setHttpOnly(true); + handler.getSessionCookieConfig().setSecure(true); logContext.setSessionHandler(handler); logContext.addAliasCheck(new AllowSymLinkAliasChecker()); setContextAttributes(logContext, conf); @@ -735,19 +785,15 @@ protected void addDefaultApps(ContextHandlerCollection parent, ServletContextHandler staticContext = new ServletContextHandler(parent, "/static"); staticContext.setResourceBase(appDir + "/static"); - staticContext.addServlet(DefaultServlet.class, "/*"); + staticContext.addServlet(WebServlet.class, "/*"); staticContext.setDisplayName("static"); @SuppressWarnings("unchecked") Map params = staticContext.getInitParams(); params.put("org.eclipse.jetty.servlet.Default.dirAllowed", "false"); params.put("org.eclipse.jetty.servlet.Default.gzip", "true"); SessionHandler handler = new SessionHandler(); - SessionManager sm = handler.getSessionManager(); - if (sm instanceof AbstractSessionManager) { - AbstractSessionManager asm = (AbstractSessionManager) sm; - asm.setHttpOnly(true); - asm.getSessionCookieConfig().setSecure(true); - } + handler.setHttpOnly(true); + handler.getSessionCookieConfig().setSecure(true); staticContext.setSessionHandler(handler); staticContext.addAliasCheck(new AllowSymLinkAliasChecker()); setContextAttributes(staticContext, conf); @@ -794,12 +840,27 @@ public void setAttribute(String name, Object value) { */ public void addJerseyResourcePackage(final String packageName, final String pathSpec) { + addJerseyResourcePackage(packageName, pathSpec, + Collections.emptyMap()); + } + + /** + * Add a Jersey resource package. + * @param packageName The Java package name containing the Jersey resource. + * @param pathSpec The path spec for the servlet + * @param params properties and features for ResourceConfig + */ + public void addJerseyResourcePackage(final String packageName, + final String pathSpec, Map params) { LOG.info("addJerseyResourcePackage: packageName=" + packageName + ", pathSpec=" + pathSpec); final ServletHolder sh = new ServletHolder(ServletContainer.class); sh.setInitParameter("com.sun.jersey.config.property.resourceConfigClass", "com.sun.jersey.api.core.PackagesResourceConfig"); sh.setInitParameter("com.sun.jersey.config.property.packages", packageName); + for (Map.Entry entry : params.entrySet()) { + sh.setInitParameter(entry.getKey(), entry.getValue()); + } webAppContext.addServlet(sh, pathSpec); } @@ -812,7 +873,6 @@ public void addJerseyResourcePackage(final String packageName, public void addServlet(String name, String pathSpec, Class clazz) { addInternalServlet(name, pathSpec, clazz, false); - addFilterPathMapping(pathSpec, webAppContext); } /** @@ -869,16 +929,6 @@ public void addInternalServlet(String name, String pathSpec, } } webAppContext.addServlet(holder, pathSpec); - - if(requireAuth && UserGroupInformation.isSecurityEnabled()) { - LOG.info("Adding Kerberos (SPNEGO) filter to " + name); - ServletHandler handler = webAppContext.getServletHandler(); - FilterMapping fmap = new FilterMapping(); - fmap.setPathSpec(pathSpec); - fmap.setFilterName(SPNEGO_FILTER); - fmap.setDispatches(FilterMapping.ALL); - handler.addFilterMapping(fmap); - } } /** @@ -945,8 +995,8 @@ public void addFilter(String name, String classname, Map parameters) { FilterHolder filterHolder = getFilterHolder(name, classname, parameters); - final String[] USER_FACING_URLS = { "*.html", "*.jsp" }; - FilterMapping fmap = getFilterMapping(name, USER_FACING_URLS); + final String[] userFacingUrls = {"/", "/*" }; + FilterMapping fmap = getFilterMapping(name, userFacingUrls); defineFilter(webAppContext, filterHolder, fmap); LOG.info( "Added filter " + name + " (class=" + classname + ") to context " @@ -1141,6 +1191,11 @@ public void start() throws IOException { try { openListeners(); webServer.start(); + if (prometheusSupport) { + DefaultMetricsSystem.instance() + .register("prometheus", "Hadoop metrics prometheus exporter", + prometheusMetricsSink); + } } catch (IOException ex) { LOG.info("HttpServer.start() threw a non Bind IOException", ex); throw ex; @@ -1201,7 +1256,7 @@ private static void bindListener(ServerConnector listener) throws Exception { * @return */ private static BindException constructBindException(ServerConnector listener, - BindException ex) { + IOException ex) { BindException be = new BindException("Port in use: " + listener.getHost() + ":" + listener.getPort()); if (ex != null) { @@ -1223,7 +1278,7 @@ private void bindForSinglePort(ServerConnector listener, int port) try { bindListener(listener); break; - } catch (BindException ex) { + } catch (IOException ex) { if (port == 0 || !findPort) { throw constructBindException(listener, ex); } @@ -1243,13 +1298,13 @@ private void bindForSinglePort(ServerConnector listener, int port) */ private void bindForPortRange(ServerConnector listener, int startPort) throws Exception { - BindException bindException = null; + IOException ioException = null; try { bindListener(listener); return; - } catch (BindException ex) { + } catch (IOException ex) { // Ignore exception. - bindException = ex; + ioException = ex; } for(Integer port : portRanges) { if (port == startPort) { @@ -1262,10 +1317,10 @@ private void bindForPortRange(ServerConnector listener, int startPort) return; } catch (BindException ex) { // Ignore exception. Move to next port. - bindException = ex; + ioException = ex; } } - throw constructBindException(listener, bindException); + throw constructBindException(listener, ioException); } /** diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/PrometheusServlet.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/PrometheusServlet.java new file mode 100644 index 0000000000000..abcc088a0319d --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/PrometheusServlet.java @@ -0,0 +1,47 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.http; + +import javax.servlet.ServletException; +import javax.servlet.http.HttpServlet; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import java.io.IOException; + +import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; +import org.apache.hadoop.metrics2.sink.PrometheusMetricsSink; + +/** + * Servlet to publish hadoop metrics in prometheus format. + */ +public class PrometheusServlet extends HttpServlet { + + public PrometheusMetricsSink getPrometheusSink() { + return + (PrometheusMetricsSink) getServletContext().getAttribute( + HttpServer2.PROMETHEUS_SINK); + } + + @Override + protected void doGet(HttpServletRequest req, HttpServletResponse resp) + throws ServletException, IOException { + DefaultMetricsSystem.instance().publishMetricsNow(); + getPrometheusSink().writeMetrics(resp.getWriter()); + resp.getWriter().flush(); + } +} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/WebServlet.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/WebServlet.java new file mode 100644 index 0000000000000..2eb6c2beb16a6 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/WebServlet.java @@ -0,0 +1,59 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.http; + +import java.io.IOException; + +import javax.servlet.ServletException; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import org.eclipse.jetty.servlet.DefaultServlet; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +/** + * Hadoop DefaultServlet for serving static web content. + */ +public class WebServlet extends DefaultServlet { + private static final long serialVersionUID = 3910031415927L; + public static final Logger LOG = LoggerFactory.getLogger(WebServlet.class); + + /** + * Get method is modified to support impersonation and Kerberos + * SPNEGO token by forcing client side redirect when accessing + * "/" (root) of the web application context. + */ + @Override + protected void doGet(HttpServletRequest request, HttpServletResponse response) + throws ServletException, IOException { + if (request.getRequestURI().equals("/")) { + StringBuilder location = new StringBuilder(); + location.append("index.html"); + if (request.getQueryString()!=null) { + // echo query string but prevent HTTP response splitting + location.append("?"); + location.append(request.getQueryString() + .replaceAll("\n", "").replaceAll("\r", "")); + } + response.sendRedirect(location.toString()); + } else { + super.doGet(request, response); + } + } +} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java index 5bbfba39b76e2..121af64b01182 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java @@ -508,8 +508,6 @@ private static T wrapWithMessage( Throwable t = ctor.newInstance(msg); return (T) (t.initCause(exception)); } catch (Throwable e) { - LOG.warn("Unable to wrap exception of type " + - clazz + ": it has no (String) constructor", e); throw exception; } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java index 4e0cd8fdd865c..973afa33e3f35 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java @@ -100,6 +100,51 @@ public static class POSIX { write. */ public static int SYNC_FILE_RANGE_WAIT_AFTER = 4; + /** + * Keeps the support state of PMDK. + */ + public enum SupportState { + UNSUPPORTED(-1), + PMDK_LIB_NOT_FOUND(1), + SUPPORTED(0); + + private byte stateCode; + SupportState(int stateCode) { + this.stateCode = (byte) stateCode; + } + + public int getStateCode() { + return stateCode; + } + + public String getMessage() { + String msg; + switch (stateCode) { + // -1 represents UNSUPPORTED. + case -1: + msg = "The native code was built without PMDK support."; + break; + // 1 represents PMDK_LIB_NOT_FOUND. + case 1: + msg = "The native code was built with PMDK support, but PMDK libs " + + "were NOT found in execution environment or failed to be loaded."; + break; + // 0 represents SUPPORTED. + case 0: + msg = "The native code was built with PMDK support, and PMDK libs " + + "were loaded successfully."; + break; + default: + msg = "The state code: " + stateCode + " is unrecognized!"; + } + return msg; + } + } + + // Denotes the state of supporting PMDK. The value is set by JNI. + private static SupportState pmdkSupportState = + SupportState.UNSUPPORTED; + private static final Logger LOG = LoggerFactory.getLogger(NativeIO.class); // Set to true via JNI if possible @@ -124,6 +169,106 @@ public static void setCacheManipulator(CacheManipulator cacheManipulator) { POSIX.cacheManipulator = cacheManipulator; } + // This method is invoked by JNI. + public static void setPmdkSupportState(int stateCode) { + for (SupportState state : SupportState.values()) { + if (state.getStateCode() == stateCode) { + pmdkSupportState = state; + return; + } + } + LOG.error("The state code: " + stateCode + " is unrecognized!"); + } + + public static String getPmdkSupportStateMessage() { + if (getPmdkLibPath() != null) { + return pmdkSupportState.getMessage() + + " The pmdk lib path: " + getPmdkLibPath(); + } + return pmdkSupportState.getMessage(); + } + + public static boolean isPmdkAvailable() { + LOG.info(pmdkSupportState.getMessage()); + return pmdkSupportState == SupportState.SUPPORTED; + } + + /** + * Denote memory region for a file mapped. + */ + public static class PmemMappedRegion { + private long address; + private long length; + private boolean isPmem; + + public PmemMappedRegion(long address, long length, boolean isPmem) { + this.address = address; + this.length = length; + this.isPmem = isPmem; + } + + public boolean isPmem() { + return this.isPmem; + } + + public long getAddress() { + return this.address; + } + + public long getLength() { + return this.length; + } + } + + /** + * JNI wrapper of persist memory operations. + */ + public static class Pmem { + // check whether the address is a Pmem address or DIMM address + public static boolean isPmem(long address, long length) { + return NativeIO.POSIX.isPmemCheck(address, length); + } + + // create a pmem file and memory map it + public static PmemMappedRegion mapBlock(String path, long length) { + return NativeIO.POSIX.pmemCreateMapFile(path, length); + } + + // unmap a pmem file + public static boolean unmapBlock(long address, long length) { + return NativeIO.POSIX.pmemUnMap(address, length); + } + + // copy data from disk file(src) to pmem file(dest), without flush + public static void memCopy(byte[] src, long dest, boolean isPmem, + long length) { + NativeIO.POSIX.pmemCopy(src, dest, isPmem, length); + } + + // flush the memory content to persistent storage + public static void memSync(PmemMappedRegion region) { + if (region.isPmem()) { + NativeIO.POSIX.pmemDrain(); + } else { + NativeIO.POSIX.pmemSync(region.getAddress(), region.getLength()); + } + } + + public static String getPmdkLibPath() { + return POSIX.getPmdkLibPath(); + } + } + + private static native String getPmdkLibPath(); + private static native boolean isPmemCheck(long address, long length); + private static native PmemMappedRegion pmemCreateMapFile(String path, + long length); + private static native boolean pmemUnMap(long address, long length); + private static native void pmemCopy(byte[] src, long dest, boolean isPmem, + long length); + private static native void pmemDrain(); + private static native void pmemSync(long address, long length); + /** * Used to manipulate the operating system cache. */ @@ -143,8 +288,8 @@ public long getOperatingSystemPageSize() { } public void posixFadviseIfPossible(String identifier, - FileDescriptor fd, long offset, long len, int flags) - throws NativeIOException { + FileDescriptor fd, long offset, long len, int flags) + throws NativeIOException { NativeIO.POSIX.posixFadviseIfPossible(identifier, fd, offset, len, flags); } @@ -748,7 +893,7 @@ public CachedUid(String username, long timestamp) { * user account name, of the format DOMAIN\UserName. This method * will remove the domain part of the full logon name. * - * @param Fthe full principal name containing the domain + * @param name the full principal name containing the domain * @return name with domain removed */ private static String stripDomain(String name) { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java index adf23c075f393..ba07db4c2ae5e 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java @@ -38,6 +38,7 @@ import org.apache.hadoop.ipc.RetriableException; import org.apache.hadoop.ipc.StandbyException; import org.apache.hadoop.net.ConnectTimeoutException; +import org.apache.hadoop.security.AccessControlException; import org.apache.hadoop.security.token.SecretManager.InvalidToken; import org.ietf.jgss.GSSException; @@ -688,6 +689,9 @@ public RetryAction shouldRetry(Exception e, int retries, } else if (e instanceof InvalidToken) { return new RetryAction(RetryAction.RetryDecision.FAIL, 0, "Invalid or Cancelled Token"); + } else if (e instanceof AccessControlException) { + return new RetryAction(RetryAction.RetryDecision.FAIL, 0, + "Access denied"); } else if (e instanceof SocketException || (e instanceof IOException && !(e instanceof RemoteException))) { if (isIdempotentOrAtMostOnce) { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java index 9731e13d86b87..81b7d34d0d1e0 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java @@ -61,6 +61,7 @@ static Class convertSchedulerClass( } private volatile boolean clientBackOffEnabled; + private boolean serverFailOverEnabled; // Atomic refs point to active callQueue // We have two so we can better control swapping @@ -79,6 +80,10 @@ public CallQueueManager(Class> backingClass, BlockingQueue bq = createCallQueueInstance(backingClass, priorityLevels, maxQueueSize, namespace, conf); this.clientBackOffEnabled = clientBackOffEnabled; + this.serverFailOverEnabled = conf.getBoolean( + namespace + "." + + CommonConfigurationKeys.IPC_CALLQUEUE_SERVER_FAILOVER_ENABLE, + CommonConfigurationKeys.IPC_CALLQUEUE_SERVER_FAILOVER_ENABLE_DEFAULT); this.putRef = new AtomicReference>(bq); this.takeRef = new AtomicReference>(bq); LOG.info("Using callQueue: {}, queueCapacity: {}, " + @@ -88,11 +93,12 @@ public CallQueueManager(Class> backingClass, @VisibleForTesting // only! CallQueueManager(BlockingQueue queue, RpcScheduler scheduler, - boolean clientBackOffEnabled) { + boolean clientBackOffEnabled, boolean serverFailOverEnabled) { this.putRef = new AtomicReference>(queue); this.takeRef = new AtomicReference>(queue); this.scheduler = scheduler; this.clientBackOffEnabled = clientBackOffEnabled; + this.serverFailOverEnabled = serverFailOverEnabled; } private static T createScheduler( @@ -193,13 +199,11 @@ boolean shouldBackOff(Schedulable e) { return scheduler.shouldBackOff(e); } - void addResponseTime(String name, int priorityLevel, int queueTime, - int processingTime) { - scheduler.addResponseTime(name, priorityLevel, queueTime, processingTime); + void addResponseTime(String name, Schedulable e, ProcessingDetails details) { + scheduler.addResponseTime(name, e, details); } // This should be only called once per call and cached in the call object - // each getPriorityLevel call will increment the counter for the caller int getPriorityLevel(Schedulable e) { return scheduler.getPriorityLevel(e); } @@ -251,7 +255,9 @@ boolean addInternal(E e, boolean checkBackoff) { // ideally this behavior should be controllable too. private void throwBackoff() throws IllegalStateException { - throw CallQueueOverflowException.DISCONNECT; + throw serverFailOverEnabled ? + CallQueueOverflowException.FAILOVER : + CallQueueOverflowException.DISCONNECT; } /** @@ -423,7 +429,10 @@ static class CallQueueOverflowException extends IllegalStateException { new CallQueueOverflowException( new RetriableException(TOO_BUSY + " - disconnecting"), RpcStatusProto.FATAL); - + static final CallQueueOverflowException FAILOVER = + new CallQueueOverflowException( + new StandbyException(TOO_BUSY + " - disconnect and failover"), + RpcStatusProto.FATAL); CallQueueOverflowException(final IOException ioe, final RpcStatusProto status) { super("Queue full", new RpcServerException(ioe.getMessage(), ioe){ diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java index 3afa6d80b8f4d..358c0d7ac3448 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java @@ -638,6 +638,10 @@ private synchronized boolean updateAddress() throws IOException { LOG.warn("Address change detected. Old: " + server.toString() + " New: " + currentAddr.toString()); server = currentAddr; + UserGroupInformation ticket = remoteId.getTicket(); + this.setName("IPC Client (" + socketFactory.hashCode() + + ") connection to " + server.toString() + " from " + + ((ticket == null) ? "an unknown user" : ticket.getUserName())); return true; } return false; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CostProvider.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CostProvider.java new file mode 100644 index 0000000000000..cf76e7d64f05e --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CostProvider.java @@ -0,0 +1,46 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ipc; + +import org.apache.hadoop.conf.Configuration; + +/** + * Used by {@link DecayRpcScheduler} to get the cost of users' operations. This + * is configurable using + * {@link org.apache.hadoop.fs.CommonConfigurationKeys#IPC_COST_PROVIDER_KEY}. + */ +public interface CostProvider { + + /** + * Initialize this provider using the given configuration, examining only + * ones which fall within the provided namespace. + * + * @param namespace The namespace to use when looking up configurations. + * @param conf The configuration + */ + void init(String namespace, Configuration conf); + + /** + * Get cost from {@link ProcessingDetails} which will be used in scheduler. + * + * @param details Process details + * @return The cost of the call + */ + long getCost(ProcessingDetails details); +} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DecayRpcScheduler.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DecayRpcScheduler.java index 5410aebbd0100..ffeafb5c0dc70 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DecayRpcScheduler.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DecayRpcScheduler.java @@ -55,9 +55,11 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.apache.hadoop.ipc.ProcessingDetails.Timing; + /** - * The decay RPC scheduler counts incoming requests in a map, then - * decays the counts at a fixed time interval. The scheduler is optimized + * The decay RPC scheduler tracks the cost of incoming requests in a map, then + * decays the costs at a fixed time interval. The scheduler is optimized * for large periods (on the order of seconds), as it offloads work to the * decay sweep. */ @@ -75,7 +77,7 @@ public class DecayRpcScheduler implements RpcScheduler, "faircallqueue.decay-scheduler.period-ms"; /** - * Decay factor controls how much each count is suppressed by on each sweep. + * Decay factor controls how much each cost is suppressed by on each sweep. * Valid numbers are > 0 and < 1. Decay factor works in tandem with * period * to control how long the scheduler remembers an identity. @@ -133,15 +135,15 @@ public class DecayRpcScheduler implements RpcScheduler, private static final ObjectWriter WRITER = new ObjectMapper().writer(); // Track the decayed and raw (no decay) number of calls for each schedulable - // identity from all previous decay windows: idx 0 for decayed call count and - // idx 1 for the raw call count - private final ConcurrentHashMap> callCounts = + // identity from all previous decay windows: idx 0 for decayed call cost and + // idx 1 for the raw call cost + private final ConcurrentHashMap> callCosts = new ConcurrentHashMap>(); - // Should be the sum of all AtomicLongs in decayed callCounts - private final AtomicLong totalDecayedCallCount = new AtomicLong(); - // The sum of all AtomicLongs in raw callCounts - private final AtomicLong totalRawCallCount = new AtomicLong(); + // Should be the sum of all AtomicLongs in decayed callCosts + private final AtomicLong totalDecayedCallCost = new AtomicLong(); + // The sum of all AtomicLongs in raw callCosts + private final AtomicLong totalRawCallCost = new AtomicLong(); // Track total call count and response time in current decay window @@ -159,7 +161,7 @@ public class DecayRpcScheduler implements RpcScheduler, // Tune the behavior of the scheduler private final long decayPeriodMillis; // How long between each tick - private final double decayFactor; // nextCount = currentCount * decayFactor + private final double decayFactor; // nextCost = currentCost * decayFactor private final int numLevels; private final double[] thresholds; private final IdentityProvider identityProvider; @@ -169,9 +171,10 @@ public class DecayRpcScheduler implements RpcScheduler, private final int topUsersCount; // e.g., report top 10 users' metrics private static final double PRECISION = 0.0001; private MetricsProxy metricsProxy; + private final CostProvider costProvider; /** - * This TimerTask will call decayCurrentCounts until + * This TimerTask will call decayCurrentCosts until * the scheduler has been garbage collected. */ public static class DecayTask extends TimerTask { @@ -187,7 +190,7 @@ public DecayTask(DecayRpcScheduler scheduler, Timer timer) { public void run() { DecayRpcScheduler sched = schedulerRef.get(); if (sched != null) { - sched.decayCurrentCounts(); + sched.decayCurrentCosts(); } else { // Our scheduler was garbage collected since it is no longer in use, // so we should terminate the timer as well @@ -214,6 +217,7 @@ public DecayRpcScheduler(int numLevels, String ns, Configuration conf) { this.decayFactor = parseDecayFactor(ns, conf); this.decayPeriodMillis = parseDecayPeriodMillis(ns, conf); this.identityProvider = this.parseIdentityProvider(ns, conf); + this.costProvider = this.parseCostProvider(ns, conf); this.thresholds = parseThresholds(ns, conf, numLevels); this.backOffByResponseTimeEnabled = parseBackOffByResponseTimeEnabled(ns, conf); @@ -241,6 +245,24 @@ public DecayRpcScheduler(int numLevels, String ns, Configuration conf) { recomputeScheduleCache(); } + private CostProvider parseCostProvider(String ns, Configuration conf) { + List providers = conf.getInstances( + ns + "." + CommonConfigurationKeys.IPC_COST_PROVIDER_KEY, + CostProvider.class); + + if (providers.size() < 1) { + LOG.info("CostProvider not specified, defaulting to DefaultCostProvider"); + return new DefaultCostProvider(); + } else if (providers.size() > 1) { + LOG.warn("Found multiple CostProviders; using: {}", + providers.get(0).getClass()); + } + + CostProvider provider = providers.get(0); // use the first + provider.init(ns, conf); + return provider; + } + // Load configs private IdentityProvider parseIdentityProvider(String ns, Configuration conf) { @@ -387,69 +409,69 @@ private static Boolean parseBackOffByResponseTimeEnabled(String ns, } /** - * Decay the stored counts for each user and clean as necessary. + * Decay the stored costs for each user and clean as necessary. * This method should be called periodically in order to keep - * counts current. + * costs current. */ - private void decayCurrentCounts() { - LOG.debug("Start to decay current counts."); + private void decayCurrentCosts() { + LOG.debug("Start to decay current costs."); try { - long totalDecayedCount = 0; - long totalRawCount = 0; + long totalDecayedCost = 0; + long totalRawCost = 0; Iterator>> it = - callCounts.entrySet().iterator(); + callCosts.entrySet().iterator(); while (it.hasNext()) { Map.Entry> entry = it.next(); - AtomicLong decayedCount = entry.getValue().get(0); - AtomicLong rawCount = entry.getValue().get(1); + AtomicLong decayedCost = entry.getValue().get(0); + AtomicLong rawCost = entry.getValue().get(1); // Compute the next value by reducing it by the decayFactor - totalRawCount += rawCount.get(); - long currentValue = decayedCount.get(); + totalRawCost += rawCost.get(); + long currentValue = decayedCost.get(); long nextValue = (long) (currentValue * decayFactor); - totalDecayedCount += nextValue; - decayedCount.set(nextValue); + totalDecayedCost += nextValue; + decayedCost.set(nextValue); - LOG.debug("Decaying counts for the user: {}, " + - "its decayedCount: {}, rawCount: {}", entry.getKey(), - nextValue, rawCount.get()); + LOG.debug( + "Decaying costs for the user: {}, its decayedCost: {}, rawCost: {}", + entry.getKey(), nextValue, rawCost.get()); if (nextValue == 0) { - LOG.debug("The decayed count for the user {} is zero " + + LOG.debug("The decayed cost for the user {} is zero " + "and being cleaned.", entry.getKey()); // We will clean up unused keys here. An interesting optimization - // might be to have an upper bound on keyspace in callCounts and only + // might be to have an upper bound on keyspace in callCosts and only // clean once we pass it. it.remove(); } } // Update the total so that we remain in sync - totalDecayedCallCount.set(totalDecayedCount); - totalRawCallCount.set(totalRawCount); + totalDecayedCallCost.set(totalDecayedCost); + totalRawCallCost.set(totalRawCost); - LOG.debug("After decaying the stored counts, totalDecayedCount: {}, " + - "totalRawCallCount: {}.", totalDecayedCount, totalRawCount); + LOG.debug("After decaying the stored costs, totalDecayedCost: {}, " + + "totalRawCallCost: {}.", totalDecayedCost, totalRawCost); // Now refresh the cache of scheduling decisions recomputeScheduleCache(); // Update average response time with decay updateAverageResponseTime(true); } catch (Exception ex) { - LOG.error("decayCurrentCounts exception: " + + LOG.error("decayCurrentCosts exception: " + ExceptionUtils.getStackTrace(ex)); throw ex; } } /** - * Update the scheduleCache to match current conditions in callCounts. + * Update the scheduleCache to match current conditions in callCosts. */ private void recomputeScheduleCache() { Map nextCache = new HashMap(); - for (Map.Entry> entry : callCounts.entrySet()) { + for (Map.Entry> entry : callCosts.entrySet()) { Object id = entry.getKey(); AtomicLong value = entry.getValue().get(0); @@ -464,51 +486,52 @@ private void recomputeScheduleCache() { } /** - * Get the number of occurrences and increment atomically. - * @param identity the identity of the user to increment - * @return the value before incrementation + * Adjust the stored cost for a given identity. + * + * @param identity the identity of the user whose cost should be adjusted + * @param costDelta the cost to add for the given identity */ - private long getAndIncrementCallCounts(Object identity) - throws InterruptedException { - // We will increment the count, or create it if no such count exists - List count = this.callCounts.get(identity); - if (count == null) { - // Create the counts since no such count exists. - // idx 0 for decayed call count - // idx 1 for the raw call count - count = new ArrayList(2); - count.add(new AtomicLong(0)); - count.add(new AtomicLong(0)); + private void addCost(Object identity, long costDelta) { + // We will increment the cost, or create it if no such cost exists + List cost = this.callCosts.get(identity); + if (cost == null) { + // Create the costs since no such cost exists. + // idx 0 for decayed call cost + // idx 1 for the raw call cost + cost = new ArrayList(2); + cost.add(new AtomicLong(0)); + cost.add(new AtomicLong(0)); // Put it in, or get the AtomicInteger that was put in by another thread - List otherCount = callCounts.putIfAbsent(identity, count); - if (otherCount != null) { - count = otherCount; + List otherCost = callCosts.putIfAbsent(identity, cost); + if (otherCost != null) { + cost = otherCost; } } // Update the total - totalDecayedCallCount.getAndIncrement(); - totalRawCallCount.getAndIncrement(); + totalDecayedCallCost.getAndAdd(costDelta); + totalRawCallCost.getAndAdd(costDelta); // At this point value is guaranteed to be not null. It may however have - // been clobbered from callCounts. Nonetheless, we return what + // been clobbered from callCosts. Nonetheless, we return what // we have. - count.get(1).getAndIncrement(); - return count.get(0).getAndIncrement(); + cost.get(1).getAndAdd(costDelta); + cost.get(0).getAndAdd(costDelta); } /** - * Given the number of occurrences, compute a scheduling decision. - * @param occurrences how many occurrences + * Given the cost for an identity, compute a scheduling decision. + * + * @param cost the cost for an identity * @return scheduling decision from 0 to numLevels - 1 */ - private int computePriorityLevel(long occurrences) { - long totalCallSnapshot = totalDecayedCallCount.get(); + private int computePriorityLevel(long cost) { + long totalCallSnapshot = totalDecayedCallCost.get(); double proportion = 0; if (totalCallSnapshot > 0) { - proportion = (double) occurrences / totalCallSnapshot; + proportion = (double) cost / totalCallSnapshot; } // Start with low priority levels, since they will be most common @@ -529,31 +552,23 @@ private int computePriorityLevel(long occurrences) { * @return integer scheduling decision from 0 to numLevels - 1 */ private int cachedOrComputedPriorityLevel(Object identity) { - try { - long occurrences = this.getAndIncrementCallCounts(identity); - - // Try the cache - Map scheduleCache = scheduleCacheRef.get(); - if (scheduleCache != null) { - Integer priority = scheduleCache.get(identity); - if (priority != null) { - LOG.debug("Cache priority for: {} with priority: {}", identity, - priority); - return priority; - } + // Try the cache + Map scheduleCache = scheduleCacheRef.get(); + if (scheduleCache != null) { + Integer priority = scheduleCache.get(identity); + if (priority != null) { + LOG.debug("Cache priority for: {} with priority: {}", identity, + priority); + return priority; } - - // Cache was no good, compute it - int priority = computePriorityLevel(occurrences); - LOG.debug("compute priority for " + identity + " priority " + priority); - return priority; - - } catch (InterruptedException ie) { - LOG.warn("Caught InterruptedException, returning low priority level"); - LOG.debug("Fallback priority for: {} with priority: {}", identity, - numLevels - 1); - return numLevels - 1; } + + // Cache was no good, compute it + List costList = callCosts.get(identity); + long currentCost = costList == null ? 0 : costList.get(0).get(); + int priority = computePriorityLevel(currentCost); + LOG.debug("compute priority for {} priority {}", identity, priority); + return priority; } /** @@ -601,14 +616,22 @@ public boolean shouldBackOff(Schedulable obj) { } @Override - public void addResponseTime(String name, int priorityLevel, int queueTime, - int processingTime) { + public void addResponseTime(String callName, Schedulable schedulable, + ProcessingDetails details) { + String user = identityProvider.makeIdentity(schedulable); + long processingCost = costProvider.getCost(details); + addCost(user, processingCost); + + int priorityLevel = schedulable.getPriorityLevel(); + long queueTime = details.get(Timing.QUEUE, TimeUnit.MILLISECONDS); + long processingTime = details.get(Timing.PROCESSING, TimeUnit.MILLISECONDS); + responseTimeCountInCurrWindow.getAndIncrement(priorityLevel); responseTimeTotalInCurrWindow.getAndAdd(priorityLevel, queueTime+processingTime); if (LOG.isDebugEnabled()) { LOG.debug("addResponseTime for call: {} priority: {} queueTime: {} " + - "processingTime: {} ", name, priorityLevel, queueTime, + "processingTime: {} ", callName, priorityLevel, queueTime, processingTime); } } @@ -646,22 +669,30 @@ void updateAverageResponseTime(boolean enableDecay) { // For testing @VisibleForTesting - public double getDecayFactor() { return decayFactor; } + double getDecayFactor() { + return decayFactor; + } @VisibleForTesting - public long getDecayPeriodMillis() { return decayPeriodMillis; } + long getDecayPeriodMillis() { + return decayPeriodMillis; + } @VisibleForTesting - public double[] getThresholds() { return thresholds; } + double[] getThresholds() { + return thresholds; + } @VisibleForTesting - public void forceDecay() { decayCurrentCounts(); } + void forceDecay() { + decayCurrentCosts(); + } @VisibleForTesting - public Map getCallCountSnapshot() { + Map getCallCostSnapshot() { HashMap snapshot = new HashMap(); - for (Map.Entry> entry : callCounts.entrySet()) { + for (Map.Entry> entry : callCosts.entrySet()) { snapshot.put(entry.getKey(), entry.getValue().get(0).get()); } @@ -669,8 +700,8 @@ public Map getCallCountSnapshot() { } @VisibleForTesting - public long getTotalCallSnapshot() { - return totalDecayedCallCount.get(); + long getTotalCallSnapshot() { + return totalDecayedCallCost.get(); } /** @@ -803,15 +834,15 @@ public void getMetrics(MetricsCollector collector, boolean all) { } public int getUniqueIdentityCount() { - return callCounts.size(); + return callCosts.size(); } public long getTotalCallVolume() { - return totalDecayedCallCount.get(); + return totalDecayedCallCost.get(); } public long getTotalRawCallVolume() { - return totalRawCallCount.get(); + return totalRawCallCost.get(); } public long[] getResponseTimeCountInLastWindow() { @@ -904,17 +935,17 @@ private void addTopNCallerSummary(MetricsRecordBuilder rb) { } } - // Get the top N callers' raw call count and scheduler decision + // Get the top N callers' raw call cost and scheduler decision private TopN getTopCallers(int n) { TopN topNCallers = new TopN(n); Iterator>> it = - callCounts.entrySet().iterator(); + callCosts.entrySet().iterator(); while (it.hasNext()) { Map.Entry> entry = it.next(); String caller = entry.getKey().toString(); - Long count = entry.getValue().get(1).get(); - if (count > 0) { - topNCallers.offer(new NameValuePair(caller, count)); + Long cost = entry.getValue().get(1).get(); + if (cost > 0) { + topNCallers.offer(new NameValuePair(caller, cost)); } } return topNCallers; @@ -935,25 +966,25 @@ public String getSchedulingDecisionSummary() { public String getCallVolumeSummary() { try { - return WRITER.writeValueAsString(getDecayedCallCounts()); + return WRITER.writeValueAsString(getDecayedCallCosts()); } catch (Exception e) { return "Error: " + e.getMessage(); } } - private Map getDecayedCallCounts() { - Map decayedCallCounts = new HashMap<>(callCounts.size()); + private Map getDecayedCallCosts() { + Map decayedCallCosts = new HashMap<>(callCosts.size()); Iterator>> it = - callCounts.entrySet().iterator(); + callCosts.entrySet().iterator(); while (it.hasNext()) { Map.Entry> entry = it.next(); Object user = entry.getKey(); - Long decayedCount = entry.getValue().get(0).get(); - if (decayedCount > 0) { - decayedCallCounts.put(user, decayedCount); + Long decayedCost = entry.getValue().get(0).get(); + if (decayedCost > 0) { + decayedCallCosts.put(user, decayedCost); } } - return decayedCallCounts; + return decayedCallCosts; } @Override diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DefaultCostProvider.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DefaultCostProvider.java new file mode 100644 index 0000000000000..ad56ddfb2e655 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DefaultCostProvider.java @@ -0,0 +1,43 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ipc; + +import org.apache.hadoop.conf.Configuration; + +/** + * Ignores process details and returns a constant value for each call. + */ +public class DefaultCostProvider implements CostProvider { + + @Override + public void init(String namespace, Configuration conf) { + // No-op + } + + /** + * Returns 1, regardless of the processing details. + * + * @param details Process details (ignored) + * @return 1 + */ + @Override + public long getCost(ProcessingDetails details) { + return 1; + } +} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DefaultRpcScheduler.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DefaultRpcScheduler.java index 0847af7f37b7a..696160ecb6c2b 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DefaultRpcScheduler.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DefaultRpcScheduler.java @@ -35,8 +35,8 @@ public boolean shouldBackOff(Schedulable obj) { } @Override - public void addResponseTime(String name, int priorityLevel, int queueTime, - int processingTime) { + public void addResponseTime(String callName, Schedulable schedulable, + ProcessingDetails details) { } public DefaultRpcScheduler(int priorityLevels, String namespace, diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ExternalCall.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ExternalCall.java index 5cc366561f091..39e55348c81e8 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ExternalCall.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ExternalCall.java @@ -37,6 +37,11 @@ public ExternalCall(PrivilegedExceptionAction action) { this.action = action; } + @Override + public String getDetailedMetricsName() { + return "(external)"; + } + public abstract UserGroupInformation getRemoteUser(); public final T get() throws InterruptedException, ExecutionException { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/FairCallQueue.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/FairCallQueue.java index 380426fe5b07d..d15a71000bd54 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/FairCallQueue.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/FairCallQueue.java @@ -34,6 +34,7 @@ import com.google.common.annotations.VisibleForTesting; import org.apache.commons.lang3.NotImplementedException; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.CommonConfigurationKeys; import org.apache.hadoop.ipc.CallQueueManager.CallQueueOverflowException; import org.apache.hadoop.metrics2.MetricsCollector; import org.apache.hadoop.metrics2.MetricsRecordBuilder; @@ -77,6 +78,8 @@ private void signalNotEmpty() { /* Statistic tracking */ private final ArrayList overflowedCalls; + /* Failover if queue is filled up */ + private boolean serverFailOverEnabled; /** * Create a FairCallQueue. * @param capacity the total size of all sub-queues @@ -108,6 +111,10 @@ public FairCallQueue(int priorityLevels, int capacity, String ns, } this.overflowedCalls.add(new AtomicLong(0)); } + this.serverFailOverEnabled = conf.getBoolean( + ns + "." + + CommonConfigurationKeys.IPC_CALLQUEUE_SERVER_FAILOVER_ENABLE, + CommonConfigurationKeys.IPC_CALLQUEUE_SERVER_FAILOVER_ENABLE_DEFAULT); this.multiplexer = new WeightedRoundRobinMultiplexer(numQueues, ns, conf); // Make this the active source of metrics @@ -158,10 +165,18 @@ public boolean add(E e) { final int priorityLevel = e.getPriorityLevel(); // try offering to all queues. if (!offerQueues(priorityLevel, e, true)) { - // only disconnect the lowest priority users that overflow the queue. - throw (priorityLevel == queues.size() - 1) - ? CallQueueOverflowException.DISCONNECT - : CallQueueOverflowException.KEEPALIVE; + + CallQueueOverflowException ex; + if (serverFailOverEnabled) { + // Signal clients to failover and try a separate server. + ex = CallQueueOverflowException.FAILOVER; + } else if (priorityLevel == queues.size() - 1){ + // only disconnect the lowest priority users that overflow the queue. + ex = CallQueueOverflowException.DISCONNECT; + } else { + ex = CallQueueOverflowException.KEEPALIVE; + } + throw ex; } return true; } @@ -377,9 +392,21 @@ public void setDelegate(FairCallQueue obj) { this.revisionNumber++; } + /** + * Fetch the current call queue from the weak reference delegate. If there + * is no delegate, or the delegate is empty, this will return null. + */ + private FairCallQueue getCallQueue() { + WeakReference> ref = this.delegate; + if (ref == null) { + return null; + } + return ref.get(); + } + @Override public int[] getQueueSizes() { - FairCallQueue obj = this.delegate.get(); + FairCallQueue obj = getCallQueue(); if (obj == null) { return new int[]{}; } @@ -389,7 +416,7 @@ public int[] getQueueSizes() { @Override public long[] getOverflowedCalls() { - FairCallQueue obj = this.delegate.get(); + FairCallQueue obj = getCallQueue(); if (obj == null) { return new long[]{}; } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/IpcException.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/IpcException.java index e55eda93778e3..61c42b80887f0 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/IpcException.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/IpcException.java @@ -26,9 +26,8 @@ */ public class IpcException extends IOException { private static final long serialVersionUID = 1L; - - final String errMsg; + public IpcException(final String err) { - errMsg = err; + super(err); } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProcessingDetails.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProcessingDetails.java new file mode 100644 index 0000000000000..5b97eec9c11cd --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProcessingDetails.java @@ -0,0 +1,96 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ipc; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.concurrent.TimeUnit; + +/** + * Stores the times that a call takes to be processed through each step. + */ +@InterfaceStability.Unstable +@InterfaceAudience.Private +public class ProcessingDetails { + public static final Logger LOG = + LoggerFactory.getLogger(ProcessingDetails.class); + private final TimeUnit valueTimeUnit; + + /** + * The different stages to track the time of. + */ + public enum Timing { + ENQUEUE, // time for reader to insert in call queue. + QUEUE, // time in the call queue. + HANDLER, // handler overhead not spent in processing/response. + PROCESSING, // time handler spent processing the call. always equal to + // lock_free + lock_wait + lock_shared + lock_exclusive + LOCKFREE, // processing with no lock. + LOCKWAIT, // processing while waiting for lock. + LOCKSHARED, // processing with a read lock. + LOCKEXCLUSIVE, // processing with a write lock. + RESPONSE; // time to encode and send response. + } + + private long[] timings = new long[Timing.values().length]; + + ProcessingDetails(TimeUnit timeUnit) { + this.valueTimeUnit = timeUnit; + } + + public long get(Timing type) { + // When using nanoTime to fetch timing information, it is possible to see + // time "move backward" slightly under unusual/rare circumstances. To avoid + // displaying a confusing number, round such timings to 0 here. + long ret = timings[type.ordinal()]; + return ret < 0 ? 0 : ret; + } + + public long get(Timing type, TimeUnit timeUnit) { + return timeUnit.convert(get(type), valueTimeUnit); + } + + public void set(Timing type, long value) { + timings[type.ordinal()] = value; + } + + public void set(Timing type, long value, TimeUnit timeUnit) { + set(type, valueTimeUnit.convert(value, timeUnit)); + } + + public void add(Timing type, long value, TimeUnit timeUnit) { + timings[type.ordinal()] += valueTimeUnit.convert(value, timeUnit); + } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder(256); + for (Timing type : Timing.values()) { + if (sb.length() > 0) { + sb.append(" "); + } + sb.append(type.name().toLowerCase()) + .append("Time=").append(get(type)); + } + return sb.toString(); + } +} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java index ae305279d02d2..c6b3fded7c731 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java @@ -521,46 +521,29 @@ public Writable call(RPC.Server server, String connectionProtocolName, Message param = request.getValue(prototype); Message result; - long startTime = Time.now(); - int qTime = (int) (startTime - receiveTime); - Exception exception = null; - boolean isDeferred = false; + Call currentCall = Server.getCurCall().get(); try { server.rpcDetailedMetrics.init(protocolImpl.protocolClass); currentCallInfo.set(new CallInfo(server, methodName)); + currentCall.setDetailedMetricsName(methodName); result = service.callBlockingMethod(methodDescriptor, null, param); // Check if this needs to be a deferred response, // by checking the ThreadLocal callback being set if (currentCallback.get() != null) { - Server.getCurCall().get().deferResponse(); - isDeferred = true; + currentCall.deferResponse(); currentCallback.set(null); return null; } } catch (ServiceException e) { - exception = (Exception) e.getCause(); + Exception exception = (Exception) e.getCause(); + currentCall.setDetailedMetricsName( + exception.getClass().getSimpleName()); throw (Exception) e.getCause(); } catch (Exception e) { - exception = e; + currentCall.setDetailedMetricsName(e.getClass().getSimpleName()); throw e; } finally { currentCallInfo.set(null); - int processingTime = (int) (Time.now() - startTime); - if (LOG.isDebugEnabled()) { - String msg = - "Served: " + methodName + (isDeferred ? ", deferred" : "") + - ", queueTime= " + qTime + - " procesingTime= " + processingTime; - if (exception != null) { - msg += " exception= " + exception.getClass().getSimpleName(); - } - LOG.debug(msg); - } - String detailedMetricsName = (exception == null) ? - methodName : - exception.getClass().getSimpleName(); - server.updateMetrics(detailedMetricsName, qTime, processingTime, - isDeferred); } return RpcWritable.wrap(result); } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RemoteException.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RemoteException.java index 36e280f39990b..f1142d35e72c2 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RemoteException.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RemoteException.java @@ -66,7 +66,7 @@ public String getClassName() { * @return may be null if the code was newer than our protobuf definitions or none was given. */ public RpcErrorCodeProto getErrorCode() { - return RpcErrorCodeProto.valueOf(errorCode); + return RpcErrorCodeProto.forNumber(errorCode); } /** diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcScheduler.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcScheduler.java index 95c5a13cdfa88..63812f47f2db0 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcScheduler.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcScheduler.java @@ -18,6 +18,8 @@ package org.apache.hadoop.ipc; +import java.util.concurrent.TimeUnit; + /** * Implement this interface to be used for RPC scheduling and backoff. * @@ -30,8 +32,43 @@ public interface RpcScheduler { boolean shouldBackOff(Schedulable obj); - void addResponseTime(String name, int priorityLevel, int queueTime, - int processingTime); + /** + * This method only exists to maintain backwards compatibility with old + * implementations. It will not be called by any Hadoop code, and should not + * be implemented by new implementations. + * + * @deprecated Use + * {@link #addResponseTime(String, Schedulable, ProcessingDetails)} instead. + */ + @Deprecated + @SuppressWarnings("unused") + default void addResponseTime(String name, int priorityLevel, int queueTime, + int processingTime) { + throw new UnsupportedOperationException( + "This method is deprecated: use the other addResponseTime"); + } + + /** + * Store a processing time value for an RPC call into this scheduler. + * + * @param callName The name of the call. + * @param schedulable The schedulable representing the incoming call. + * @param details The details of processing time. + */ + @SuppressWarnings("deprecation") + default void addResponseTime(String callName, Schedulable schedulable, + ProcessingDetails details) { + // For the sake of backwards compatibility with old implementations of + // this interface, a default implementation is supplied which uses the old + // method. All new implementations MUST override this interface and should + // NOT use the other addResponseTime method. + int queueTimeMs = (int) + details.get(ProcessingDetails.Timing.QUEUE, TimeUnit.MILLISECONDS); + int processingTimeMs = (int) + details.get(ProcessingDetails.Timing.PROCESSING, TimeUnit.MILLISECONDS); + addResponseTime(callName, schedulable.getPriorityLevel(), + queueTimeMs, processingTimeMs); + } void stop(); } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcWritable.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcWritable.java index 54fb98e80d858..a97af87bdfb01 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcWritable.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcWritable.java @@ -106,7 +106,7 @@ Message getMessage() { @Override void writeTo(ResponseBuffer out) throws IOException { int length = message.getSerializedSize(); - length += CodedOutputStream.computeRawVarint32Size(length); + length += CodedOutputStream.computeUInt32SizeNoTag(length); out.ensureCapacity(length); message.writeDelimitedTo(out); } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java index 94d9bc33137fc..36785e147d757 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java @@ -18,6 +18,7 @@ package org.apache.hadoop.ipc; +import static org.apache.hadoop.ipc.ProcessingDetails.Timing; import static org.apache.hadoop.ipc.RpcConstants.AUTHORIZATION_FAILED_CALL_ID; import static org.apache.hadoop.ipc.RpcConstants.CONNECTION_CONTEXT_CALL_ID; import static org.apache.hadoop.ipc.RpcConstants.CURRENT_VERSION; @@ -64,6 +65,7 @@ import java.util.concurrent.BlockingQueue; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; @@ -79,6 +81,7 @@ import org.apache.hadoop.conf.Configuration.IntegerRanges; import org.apache.hadoop.fs.CommonConfigurationKeys; import org.apache.hadoop.fs.CommonConfigurationKeysPublic; +import org.apache.hadoop.ha.HealthCheckFailedException; import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.io.Writable; import org.apache.hadoop.io.WritableUtils; @@ -505,7 +508,7 @@ protected void setLogSlowRPC(boolean logSlowRPCFlag) { * Logs a Slow RPC Request. * * @param methodName - RPC Request method name - * @param processingTime - Processing Time. + * @param details - Processing Detail. * * if this request took too much time relative to other requests * we consider that as a slow RPC. 3 is a magic number that comes @@ -514,7 +517,8 @@ protected void setLogSlowRPC(boolean logSlowRPCFlag) { * if and only if it falls above 99.7% of requests. We start this logic * only once we have enough sample size. */ - void logSlowRpcCalls(String methodName, int processingTime) { + void logSlowRpcCalls(String methodName, Call call, + ProcessingDetails details) { final int deviation = 3; // 1024 for minSampleSize just a guess -- not a number computed based on @@ -525,29 +529,54 @@ void logSlowRpcCalls(String methodName, int processingTime) { final double threeSigma = rpcMetrics.getProcessingMean() + (rpcMetrics.getProcessingStdDev() * deviation); + long processingTime = + details.get(Timing.PROCESSING, RpcMetrics.TIMEUNIT); if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) && (processingTime > threeSigma)) { - if(LOG.isWarnEnabled()) { - String client = CurCall.get().toString(); - LOG.warn( - "Slow RPC : " + methodName + " took " + processingTime + - " milliseconds to process from client " + client); - } + LOG.warn( + "Slow RPC : {} took {} {} to process from client {}," + + " the processing detail is {}", + methodName, processingTime, RpcMetrics.TIMEUNIT, call, + details.toString()); rpcMetrics.incrSlowRpc(); } } - void updateMetrics(String name, int queueTime, int processingTime, - boolean deferredCall) { + void updateMetrics(Call call, long startTime, boolean connDropped) { + // delta = handler + processing + response + long deltaNanos = Time.monotonicNowNanos() - startTime; + long timestampNanos = call.timestampNanos; + + ProcessingDetails details = call.getProcessingDetails(); + // queue time is the delta between when the call first arrived and when it + // began being serviced, minus the time it took to be put into the queue + details.set(Timing.QUEUE, + startTime - timestampNanos - details.get(Timing.ENQUEUE)); + deltaNanos -= details.get(Timing.PROCESSING); + deltaNanos -= details.get(Timing.RESPONSE); + details.set(Timing.HANDLER, deltaNanos); + + long queueTime = details.get(Timing.QUEUE, RpcMetrics.TIMEUNIT); rpcMetrics.addRpcQueueTime(queueTime); - if (!deferredCall) { - rpcMetrics.addRpcProcessingTime(processingTime); - rpcDetailedMetrics.addProcessingTime(name, processingTime); - callQueue.addResponseTime(name, getPriorityLevel(), queueTime, - processingTime); - if (isLogSlowRPC()) { - logSlowRpcCalls(name, processingTime); - } + + if (call.isResponseDeferred() || connDropped) { + // call was skipped; don't include it in processing metrics + return; + } + + long processingTime = + details.get(Timing.PROCESSING, RpcMetrics.TIMEUNIT); + long waitTime = + details.get(Timing.LOCKWAIT, RpcMetrics.TIMEUNIT); + rpcMetrics.addRpcLockWaitTime(waitTime); + rpcMetrics.addRpcProcessingTime(processingTime); + // don't include lock wait for detailed metrics. + processingTime -= waitTime; + String name = call.getDetailedMetricsName(); + rpcDetailedMetrics.addProcessingTime(name, processingTime); + callQueue.addResponseTime(name, call, details); + if (isLogSlowRPC()) { + logSlowRpcCalls(name, call, details); } } @@ -716,9 +745,13 @@ static boolean getClientBackoffEnable( /** A generic call queued for handling. */ public static class Call implements Schedulable, PrivilegedExceptionAction { + private final ProcessingDetails processingDetails = + new ProcessingDetails(TimeUnit.NANOSECONDS); + // the method name to use in metrics + private volatile String detailedMetricsName = ""; final int callId; // the client's call id final int retryCount; // the retry count of the call - long timestamp; // time received when response is null + long timestampNanos; // time received when response is null // time served when response is not null private AtomicInteger responseWaitCount = new AtomicInteger(1); final RPC.RpcKind rpcKind; @@ -755,7 +788,7 @@ public Call(int id, int retryCount, Void ignore1, Void ignore2, TraceScope traceScope, CallerContext callerContext) { this.callId = id; this.retryCount = retryCount; - this.timestamp = Time.now(); + this.timestampNanos = Time.monotonicNowNanos(); this.rpcKind = kind; this.clientId = clientId; this.traceScope = traceScope; @@ -764,6 +797,28 @@ public Call(int id, int retryCount, Void ignore1, Void ignore2, this.isCallCoordinated = false; } + /** + * Indicates whether the call has been processed. Always true unless + * overridden. + * + * @return true + */ + boolean isOpen() { + return true; + } + + String getDetailedMetricsName() { + return detailedMetricsName; + } + + void setDetailedMetricsName(String name) { + detailedMetricsName = name; + } + + public ProcessingDetails getProcessingDetails() { + return processingDetails; + } + @Override public String toString() { return "Call#" + callId + " Retry#" + retryCount; @@ -911,6 +966,11 @@ private class RpcCall extends Call { this.rpcRequest = param; } + @Override + boolean isOpen() { + return connection.channel.isOpen(); + } + void setResponseFields(Writable returnValue, ResponseParams responseParams) { this.rv = returnValue; @@ -938,18 +998,33 @@ public Void run() throws Exception { Server.LOG.info(Thread.currentThread().getName() + ": skipped " + this); return null; } + + long startNanos = Time.monotonicNowNanos(); Writable value = null; ResponseParams responseParams = new ResponseParams(); try { value = call( - rpcKind, connection.protocolName, rpcRequest, timestamp); + rpcKind, connection.protocolName, rpcRequest, timestampNanos); } catch (Throwable e) { populateResponseParamsOnError(e, responseParams); } if (!isResponseDeferred()) { + long deltaNanos = Time.monotonicNowNanos() - startNanos; + ProcessingDetails details = getProcessingDetails(); + + details.set(Timing.PROCESSING, deltaNanos, TimeUnit.NANOSECONDS); + deltaNanos -= details.get(Timing.LOCKWAIT, TimeUnit.NANOSECONDS); + deltaNanos -= details.get(Timing.LOCKSHARED, TimeUnit.NANOSECONDS); + deltaNanos -= details.get(Timing.LOCKEXCLUSIVE, TimeUnit.NANOSECONDS); + details.set(Timing.LOCKFREE, deltaNanos, TimeUnit.NANOSECONDS); + startNanos = Time.monotonicNowNanos(); + setResponseFields(value, responseParams); sendResponse(); + + deltaNanos = Time.monotonicNowNanos() - startNanos; + details.set(Timing.RESPONSE, deltaNanos, TimeUnit.NANOSECONDS); } else { if (LOG.isDebugEnabled()) { LOG.debug("Deferring response for callId: " + this.callId); @@ -1377,12 +1452,13 @@ Reader getReader() { } } + private final static long PURGE_INTERVAL_NANOS = TimeUnit.NANOSECONDS.convert( + 15, TimeUnit.MINUTES); + // Sends responses of RPC back to clients. private class Responder extends Thread { private final Selector writeSelector; private int pending; // connections waiting to register - - final static int PURGE_INTERVAL = 900000; // 15mins Responder() throws IOException { this.setName("IPC Server Responder"); @@ -1408,12 +1484,13 @@ public void run() { } private void doRunLoop() { - long lastPurgeTime = 0; // last check for old calls. + long lastPurgeTimeNanos = 0; // last check for old calls. while (running) { try { waitPending(); // If a channel is being registered, wait. - writeSelector.select(PURGE_INTERVAL); + writeSelector.select( + TimeUnit.NANOSECONDS.toMillis(PURGE_INTERVAL_NANOS)); Iterator iter = writeSelector.selectedKeys().iterator(); while (iter.hasNext()) { SelectionKey key = iter.next(); @@ -1435,11 +1512,11 @@ private void doRunLoop() { LOG.info(Thread.currentThread().getName() + ": doAsyncWrite threw exception " + e); } } - long now = Time.now(); - if (now < lastPurgeTime + PURGE_INTERVAL) { + long nowNanos = Time.monotonicNowNanos(); + if (nowNanos < lastPurgeTimeNanos + PURGE_INTERVAL_NANOS) { continue; } - lastPurgeTime = now; + lastPurgeTimeNanos = nowNanos; // // If there were some calls that have not been sent out for a // long time, discard them. @@ -1463,7 +1540,7 @@ private void doRunLoop() { } for (RpcCall call : calls) { - doPurge(call, now); + doPurge(call, nowNanos); } } catch (OutOfMemoryError e) { // @@ -1514,7 +1591,7 @@ private void doPurge(RpcCall call, long now) { Iterator iter = responseQueue.listIterator(0); while (iter.hasNext()) { call = iter.next(); - if (now > call.timestamp + PURGE_INTERVAL) { + if (now > call.timestampNanos + PURGE_INTERVAL_NANOS) { closeConnection(call.connection); break; } @@ -1578,7 +1655,7 @@ private boolean processResponse(LinkedList responseQueue, if (inHandler) { // set the serve time when the response has to be sent later - call.timestamp = Time.now(); + call.timestampNanos = Time.monotonicNowNanos(); incPending(); try { @@ -2084,7 +2161,7 @@ private void doSaslReply(Message message) throws IOException { private void doSaslReply(Exception ioe) throws IOException { setupResponse(authFailedCall, RpcStatusProto.FATAL, RpcErrorCodeProto.FATAL_UNAUTHORIZED, - null, ioe.getClass().getName(), ioe.getLocalizedMessage()); + null, ioe.getClass().getName(), ioe.toString()); sendResponse(authFailedCall); } @@ -2164,11 +2241,17 @@ public int readAndProcess() throws IOException, InterruptedException { setupHttpRequestOnIpcPortResponse(); return -1; } - - if (!RpcConstants.HEADER.equals(dataLengthBuffer) - || version != CURRENT_VERSION) { + + if(!RpcConstants.HEADER.equals(dataLengthBuffer)) { + LOG.warn("Incorrect RPC Header length from {}:{} " + + "expected length: {} got length: {}", + hostAddress, remotePort, RpcConstants.HEADER, dataLengthBuffer); + setupBadVersionResponse(version); + return -1; + } + if (version != CURRENT_VERSION) { //Warning is ok since this is not supposed to happen. - LOG.warn("Incorrect header or version mismatch from " + + LOG.warn("Version mismatch from " + hostAddress + ":" + remotePort + " got version " + version + " expected version " + CURRENT_VERSION); @@ -2473,7 +2556,8 @@ private void processOneRpc(ByteBuffer bb) final RpcCall call = new RpcCall(this, callId, retry); setupResponse(call, rse.getRpcStatusProto(), rse.getRpcErrorCodeProto(), null, - t.getClass().getName(), t.getMessage()); + t.getClass().getName(), + t.getMessage() != null ? t.getMessage() : t.toString()); sendResponse(call); } } @@ -2600,8 +2684,6 @@ private void processRpcRequest(RpcRequestHeaderProto header, stateId = alignmentContext.receiveRequestState( header, getMaxIdleTime()); call.setClientStateId(stateId); - LOG.trace("Client State ID= {} and Server State ID= {}", - call.getClientStateId(), alignmentContext.getLastSeenStateId()); } } catch (IOException ioe) { throw new RpcServerException("Processing RPC request caught ", ioe); @@ -2770,6 +2852,9 @@ private void internalQueueCall(Call call, boolean blocking) } else { callQueue.add(call); } + long deltaNanos = Time.monotonicNowNanos() - call.timestampNanos; + call.getProcessingDetails().set(Timing.ENQUEUE, deltaNanos, + TimeUnit.NANOSECONDS); } catch (CallQueueOverflowException cqe) { // If rpc scheduler indicates back off based on performance degradation // such as response time or rpc queue is full, we will ask the client @@ -2797,8 +2882,16 @@ public void run() { SERVER.set(Server.this); while (running) { TraceScope traceScope = null; + Call call = null; + long startTimeNanos = 0; + // True iff the connection for this call has been dropped. + // Set to true by default and update to false later if the connection + // can be succesfully read. + boolean connDropped = true; + try { - final Call call = callQueue.take(); // pop the queue; maybe blocked here + call = callQueue.take(); // pop the queue; maybe blocked here + startTimeNanos = Time.monotonicNowNanos(); if (alignmentContext != null && call.isCallCoordinated() && call.getClientStateId() > alignmentContext.getLastSeenStateId()) { /* @@ -2829,6 +2922,7 @@ public void run() { // always update the current call context CallerContext.setCurrent(call.callerContext); UserGroupInformation remoteUser = call.getRemoteUser(); + connDropped = !call.isOpen(); if (remoteUser != null) { remoteUser.doAs(call); } else { @@ -2851,6 +2945,14 @@ public void run() { } finally { CurCall.set(null); IOUtils.cleanupWithLogger(LOG, traceScope); + if (call != null) { + updateMetrics(call, startTimeNanos, connDropped); + ProcessingDetails.LOG.debug( + "Served: [{}]{} name={} user={} details={}", + call, (call.isResponseDeferred() ? ", deferred" : ""), + call.getDetailedMetricsName(), call.getRemoteUser(), + call.getProcessingDetails()); + } } } LOG.debug(Thread.currentThread().getName() + ": exiting"); @@ -2996,6 +3098,8 @@ protected Server(String bindAddress, int port, } this.exceptionsHandler.addTerseLoggingExceptions(StandbyException.class); + this.exceptionsHandler.addTerseLoggingExceptions( + HealthCheckFailedException.class); } public synchronized void addAuxiliaryListener(int auxiliaryPort) @@ -3170,10 +3274,10 @@ private byte[] setupResponseForProtobuf( cos.writeRawByte((byte)((length >>> 16) & 0xFF)); cos.writeRawByte((byte)((length >>> 8) & 0xFF)); cos.writeRawByte((byte)((length >>> 0) & 0xFF)); - cos.writeRawVarint32(header.getSerializedSize()); + cos.writeUInt32NoTag(header.getSerializedSize()); header.writeTo(cos); if (payload != null) { - cos.writeRawVarint32(payload.getSerializedSize()); + cos.writeUInt32NoTag(payload.getSerializedSize()); payload.writeTo(cos); } return buf; @@ -3181,7 +3285,7 @@ private byte[] setupResponseForProtobuf( private static int getDelimitedLength(Message message) { int length = message.getSerializedSize(); - return length + CodedOutputStream.computeRawVarint32Size(length); + return length + CodedOutputStream.computeUInt32SizeNoTag(length); } /** diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WeightedTimeCostProvider.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WeightedTimeCostProvider.java new file mode 100644 index 0000000000000..4304b24299f29 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WeightedTimeCostProvider.java @@ -0,0 +1,110 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ipc; + +import java.util.Locale; +import org.apache.hadoop.conf.Configuration; + +import static org.apache.hadoop.ipc.ProcessingDetails.Timing; + +/** + * A {@link CostProvider} that calculates the cost for an operation + * as a weighted sum of its processing time values (see + * {@link ProcessingDetails}). This can be used by specifying the + * {@link org.apache.hadoop.fs.CommonConfigurationKeys#IPC_COST_PROVIDER_KEY} + * configuration key. + * + *

    This allows for configuration of how heavily each of the operations + * within {@link ProcessingDetails} is weighted. By default, + * {@link ProcessingDetails.Timing#LOCKFREE}, + * {@link ProcessingDetails.Timing#RESPONSE}, and + * {@link ProcessingDetails.Timing#HANDLER} times have a weight of + * {@value #DEFAULT_LOCKFREE_WEIGHT}, + * {@link ProcessingDetails.Timing#LOCKSHARED} has a weight of + * {@value #DEFAULT_LOCKSHARED_WEIGHT}, + * {@link ProcessingDetails.Timing#LOCKEXCLUSIVE} has a weight of + * {@value #DEFAULT_LOCKEXCLUSIVE_WEIGHT}, and others are ignored. + * These values can all be configured using the {@link #WEIGHT_CONFIG_PREFIX} + * key, prefixed with the IPC namespace, and suffixed with the name of the + * timing measurement from {@link ProcessingDetails} (all lowercase). + * For example, to set the lock exclusive weight to be 1000, set: + * 

    + *   ipc.8020.cost-provider.impl=org.apache.hadoop.ipc.WeightedTimeCostProvider
+ *   ipc.8020.weighted-cost.lockexclusive=1000
+ *
    + */ +public class WeightedTimeCostProvider implements CostProvider { + + /** + * The prefix used in configuration values specifying the weight to use when + * determining the cost of an operation. See the class Javadoc for more info. + */ + public static final String WEIGHT_CONFIG_PREFIX = ".weighted-cost."; + static final int DEFAULT_LOCKFREE_WEIGHT = 1; + static final int DEFAULT_LOCKSHARED_WEIGHT = 10; + static final int DEFAULT_LOCKEXCLUSIVE_WEIGHT = 100; + + private long[] weights; + + @Override + public void init(String namespace, Configuration conf) { + weights = new long[Timing.values().length]; + for (Timing timing : ProcessingDetails.Timing.values()) { + final int defaultValue; + switch (timing) { + case LOCKFREE: + case RESPONSE: + case HANDLER: + defaultValue = DEFAULT_LOCKFREE_WEIGHT; + break; + case LOCKSHARED: + defaultValue = DEFAULT_LOCKSHARED_WEIGHT; + break; + case LOCKEXCLUSIVE: + defaultValue = DEFAULT_LOCKEXCLUSIVE_WEIGHT; + break; + default: + // by default don't bill for queueing or lock wait time + defaultValue = 0; + } + String key = namespace + WEIGHT_CONFIG_PREFIX + + timing.name().toLowerCase(Locale.ENGLISH); + weights[timing.ordinal()] = conf.getInt(key, defaultValue); + } + } + + /** + * Calculates a weighted sum of the times stored on the provided processing + * details to be used as the cost in {@link DecayRpcScheduler}. + * + * @param details Processing details + * @return The weighted sum of the times. The returned unit is the same + * as the default unit used by the provided processing details. + */ + @Override + public long getCost(ProcessingDetails details) { + assert weights != null : "Cost provider must be initialized before use"; + long cost = 0; + // weights was initialized to the same length as Timing.values() + for (int i = 0; i < Timing.values().length; i++) { + cost += details.get(Timing.values()[i]) * weights[i]; + } + return cost; + } +} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java index c590dbdaf2aab..b303f8494b63c 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java @@ -537,15 +537,15 @@ public Writable call(org.apache.hadoop.ipc.RPC.Server server, } // Invoke the protocol method - long startTime = Time.now(); - int qTime = (int) (startTime-receivedTime); Exception exception = null; + Call currentCall = Server.getCurCall().get(); try { Method method = protocolImpl.protocolClass.getMethod(call.getMethodName(), call.getParameterClasses()); method.setAccessible(true); server.rpcDetailedMetrics.init(protocolImpl.protocolClass); + currentCall.setDetailedMetricsName(call.getMethodName()); Object value = method.invoke(protocolImpl.protocolImpl, call.getParameters()); if (server.verbose) log("Return: "+value); @@ -571,20 +571,10 @@ public Writable call(org.apache.hadoop.ipc.RPC.Server server, exception = ioe; throw ioe; } finally { - int processingTime = (int) (Time.now() - startTime); - if (LOG.isDebugEnabled()) { - String msg = "Served: " + call.getMethodName() + - " queueTime= " + qTime + " procesingTime= " + processingTime; - if (exception != null) { - msg += " exception= " + exception.getClass().getSimpleName(); - } - LOG.debug(msg); + if (exception != null) { + currentCall.setDetailedMetricsName( + exception.getClass().getSimpleName()); } - String detailedMetricsName = (exception == null) ? - call.getMethodName() : - exception.getClass().getSimpleName(); - server - .updateMetrics(detailedMetricsName, qTime, processingTime, false); } } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcDetailedMetrics.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcDetailedMetrics.java index 0160b0e5b71b1..98b9f262b8549 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcDetailedMetrics.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcDetailedMetrics.java @@ -61,17 +61,17 @@ public static RpcDetailedMetrics create(int port) { */ public void init(Class protocol) { rates.init(protocol); - deferredRpcRates.init(protocol); + deferredRpcRates.init(protocol, "Deferred"); } /** * Add an RPC processing time sample - * @param name of the RPC call + * @param rpcCallName of the RPC call * @param processingTime the processing time */ //@Override // some instrumentation interface - public void addProcessingTime(String name, int processingTime) { - rates.add(name, processingTime); + public void addProcessingTime(String rpcCallName, long processingTime) { + rates.add(rpcCallName, processingTime); } public void addDeferredProcessingTime(String name, long processingTime) { @@ -82,5 +82,7 @@ public void addDeferredProcessingTime(String name, long processingTime) { * Shutdown the instrumentation for the process */ //@Override // some instrumentation interface - public void shutdown() {} + public void shutdown() { + DefaultMetricsSystem.instance().unregisterSource(name); + } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java index a36bcd8648c58..bb4bfcfd08be5 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java @@ -17,6 +17,8 @@ */ package org.apache.hadoop.ipc.metrics; +import java.util.concurrent.TimeUnit; + import com.google.common.annotations.VisibleForTesting; import org.apache.hadoop.fs.CommonConfigurationKeys; import org.apache.hadoop.ipc.Server; @@ -27,7 +29,6 @@ import org.apache.hadoop.metrics2.annotation.Metrics; import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; import org.apache.hadoop.metrics2.lib.MetricsRegistry; -import org.apache.hadoop.metrics2.lib.MutableCounterInt; import org.apache.hadoop.metrics2.lib.MutableCounterLong; import org.apache.hadoop.metrics2.lib.MutableQuantiles; import org.apache.hadoop.metrics2.lib.MutableRate; @@ -47,6 +48,8 @@ public class RpcMetrics { final MetricsRegistry registry; final String name; final boolean rpcQuantileEnable; + /** The time unit used when storing/accessing time durations. */ + public final static TimeUnit TIMEUNIT = TimeUnit.MILLISECONDS; RpcMetrics(Server server, Configuration conf) { String port = String.valueOf(server.getListenerAddress().getPort()); @@ -61,24 +64,31 @@ public class RpcMetrics { CommonConfigurationKeys.RPC_METRICS_QUANTILE_ENABLE, CommonConfigurationKeys.RPC_METRICS_QUANTILE_ENABLE_DEFAULT); if (rpcQuantileEnable) { - rpcQueueTimeMillisQuantiles = + rpcQueueTimeQuantiles = + new MutableQuantiles[intervals.length]; + rpcLockWaitTimeQuantiles = new MutableQuantiles[intervals.length]; - rpcProcessingTimeMillisQuantiles = + rpcProcessingTimeQuantiles = new MutableQuantiles[intervals.length]; - deferredRpcProcessingTimeMillisQuantiles = + deferredRpcProcessingTimeQuantiles = new MutableQuantiles[intervals.length]; for (int i = 0; i < intervals.length; i++) { int interval = intervals[i]; - rpcQueueTimeMillisQuantiles[i] = registry.newQuantiles("rpcQueueTime" - + interval + "s", "rpc queue time in milli second", "ops", + rpcQueueTimeQuantiles[i] = registry.newQuantiles("rpcQueueTime" + + interval + "s", "rpc queue time in " + TIMEUNIT, "ops", "latency", interval); - rpcProcessingTimeMillisQuantiles[i] = registry.newQuantiles( + rpcLockWaitTimeQuantiles[i] = registry.newQuantiles( + "rpcLockWaitTime" + interval + "s", + "rpc lock wait time in " + TIMEUNIT, "ops", + "latency", interval); + rpcProcessingTimeQuantiles[i] = registry.newQuantiles( "rpcProcessingTime" + interval + "s", - "rpc processing time in milli second", "ops", "latency", interval); - deferredRpcProcessingTimeMillisQuantiles[i] = registry - .newQuantiles("deferredRpcProcessingTime" + interval + "s", - "deferred rpc processing time in milli seconds", "ops", - "latency", interval); + "rpc processing time in " + TIMEUNIT, "ops", + "latency", interval); + deferredRpcProcessingTimeQuantiles[i] = registry.newQuantiles( + "deferredRpcProcessingTime" + interval + "s", + "deferred rpc processing time in " + TIMEUNIT, "ops", + "latency", interval); } } LOG.debug("Initialized " + registry); @@ -94,11 +104,13 @@ public static RpcMetrics create(Server server, Configuration conf) { @Metric("Number of received bytes") MutableCounterLong receivedBytes; @Metric("Number of sent bytes") MutableCounterLong sentBytes; @Metric("Queue time") MutableRate rpcQueueTime; - MutableQuantiles[] rpcQueueTimeMillisQuantiles; + MutableQuantiles[] rpcQueueTimeQuantiles; + @Metric("Lock wait time") MutableRate rpcLockWaitTime; + MutableQuantiles[] rpcLockWaitTimeQuantiles; @Metric("Processing time") MutableRate rpcProcessingTime; - MutableQuantiles[] rpcProcessingTimeMillisQuantiles; + MutableQuantiles[] rpcProcessingTimeQuantiles; @Metric("Deferred Processing time") MutableRate deferredRpcProcessingTime; - MutableQuantiles[] deferredRpcProcessingTimeMillisQuantiles; + MutableQuantiles[] deferredRpcProcessingTimeQuantiles; @Metric("Number of authentication failures") MutableCounterLong rpcAuthenticationFailures; @Metric("Number of authentication successes") @@ -170,7 +182,9 @@ public void incrAuthorizationFailures() { * Shutdown the instrumentation for the process */ //@Override - public void shutdown() {} + public void shutdown() { + DefaultMetricsSystem.instance().unregisterSource(name); + } /** * Increment sent bytes by count @@ -194,25 +208,32 @@ public void incrReceivedBytes(int count) { * Add an RPC queue time sample * @param qTime the queue time */ - //@Override - public void addRpcQueueTime(int qTime) { + public void addRpcQueueTime(long qTime) { rpcQueueTime.add(qTime); if (rpcQuantileEnable) { - for (MutableQuantiles q : rpcQueueTimeMillisQuantiles) { + for (MutableQuantiles q : rpcQueueTimeQuantiles) { q.add(qTime); } } } + public void addRpcLockWaitTime(long waitTime) { + rpcLockWaitTime.add(waitTime); + if (rpcQuantileEnable) { + for (MutableQuantiles q : rpcLockWaitTimeQuantiles) { + q.add(waitTime); + } + } + } + /** * Add an RPC processing time sample * @param processingTime the processing time */ - //@Override - public void addRpcProcessingTime(int processingTime) { + public void addRpcProcessingTime(long processingTime) { rpcProcessingTime.add(processingTime); if (rpcQuantileEnable) { - for (MutableQuantiles q : rpcProcessingTimeMillisQuantiles) { + for (MutableQuantiles q : rpcProcessingTimeQuantiles) { q.add(processingTime); } } @@ -221,7 +242,7 @@ public void addRpcProcessingTime(int processingTime) { public void addDeferredRpcProcessingTime(long processingTime) { deferredRpcProcessingTime.add(processingTime); if (rpcQuantileEnable) { - for (MutableQuantiles q : deferredRpcProcessingTimeMillisQuantiles) { + for (MutableQuantiles q : deferredRpcProcessingTimeQuantiles) { q.add(processingTime); } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java index c404ebedebbe1..f20933b5c8668 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java @@ -348,7 +348,8 @@ private void writeAttribute(JsonGenerator jg, ObjectName oname, MBeanAttributeIn } catch (RuntimeErrorException e) { // RuntimeErrorException happens when an unexpected failure occurs in getAttribute // for example https://issues.apache.org/jira/browse/DAEMON-120 - LOG.debug("getting attribute "+attName+" of "+oname+" threw an exception", e); + LOG.error("getting attribute {} of {} threw an exception", + attName, oname, e); return; } catch (AttributeNotFoundException e) { //Ignored the attribute was not found, which should never happen because the bean diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRatesWithAggregation.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRatesWithAggregation.java index 26a15063bb498..aa7b7596173e7 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRatesWithAggregation.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRatesWithAggregation.java @@ -58,6 +58,8 @@ public class MutableRatesWithAggregation extends MutableMetric { weakReferenceQueue = new ConcurrentLinkedDeque<>(); private final ThreadLocal> threadLocalMetricsMap = new ThreadLocal<>(); + // prefix for metric name + private String typePrefix = ""; /** * Initialize the registry with all the methods in a protocol @@ -148,7 +150,7 @@ Map getGlobalMetrics() { private synchronized MutableRate addMetricIfNotExists(String name) { MutableRate metric = globalMetrics.get(name); if (metric == null) { - metric = new MutableRate(name, name, false); + metric = new MutableRate(name + typePrefix, name + typePrefix, false); globalMetrics.put(name, metric); } return metric; @@ -170,4 +172,9 @@ synchronized void snapshotInto(MutableRate metric) { } } + public void init(Class protocol, String prefix) { + this.typePrefix = prefix; + init(protocol); + } + } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/PrometheusMetricsSink.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/PrometheusMetricsSink.java new file mode 100644 index 0000000000000..10df76941caf4 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/PrometheusMetricsSink.java @@ -0,0 +1,124 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

    + * http://www.apache.org/licenses/LICENSE-2.0 + *

    + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.metrics2.sink; + +import org.apache.commons.configuration2.SubsetConfiguration; +import org.apache.hadoop.metrics2.AbstractMetric; +import org.apache.hadoop.metrics2.MetricType; +import org.apache.hadoop.metrics2.MetricsRecord; +import org.apache.hadoop.metrics2.MetricsSink; +import org.apache.hadoop.metrics2.MetricsTag; + +import java.io.IOException; +import java.io.Writer; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.regex.Pattern; + +import org.apache.commons.lang3.StringUtils; + +/** + * Metrics sink for prometheus exporter. + *

    + * Stores the metric data in-memory and return with it on request. + */ +public class PrometheusMetricsSink implements MetricsSink { + + /** + * Cached output lines for each metrics. + */ + private final Map metricLines = new ConcurrentHashMap<>(); + + private static final Pattern SPLIT_PATTERN = + Pattern.compile("(? { /** @return its children */ List getChildren(); + /** @return the number of children this node has. */ + int getNumOfChildren(); + /** @return the number of leave nodes. */ int getNumOfLeaves(); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/InnerNodeImpl.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/InnerNodeImpl.java index 5a2931bf6a153..923515b6efe7e 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/InnerNodeImpl.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/InnerNodeImpl.java @@ -41,26 +41,30 @@ public InnerNodeImpl newInnerNode(String path) { protected final Map childrenMap = new HashMap<>(); protected int numOfLeaves; - /** Construct an InnerNode from a path-like string */ + /** Construct an InnerNode from a path-like string. */ protected InnerNodeImpl(String path) { super(path); } /** Construct an InnerNode - * from its name, its network location, its parent, and its level */ - protected InnerNodeImpl(String name, String location, InnerNode parent, int level) { + * from its name, its network location, its parent, and its level. */ + protected InnerNodeImpl(String name, String location, + InnerNode parent, int level) { super(name, location, parent, level); } @Override - public List getChildren() {return children;} + public List getChildren() { + return children; + } - /** @return the number of children this node has */ - int getNumOfChildren() { + /** @return the number of children this node has. */ + @Override + public int getNumOfChildren() { return children.size(); } - /** Judge if this node represents a rack + /** Judge if this node represents a rack. * @return true if it has no child or its children are not InnerNodes */ public boolean isRack() { @@ -76,7 +80,7 @@ public boolean isRack() { return true; } - /** Judge if this node is an ancestor of node n + /** Judge if this node is an ancestor of node n. * * @param n a node * @return true if this node is an ancestor of n @@ -87,7 +91,7 @@ public boolean isAncestor(Node n) { startsWith(getPath(this)+NodeBase.PATH_SEPARATOR_STR); } - /** Judge if this node is the parent of node n + /** Judge if this node is the parent of node n. * * @param n a node * @return true if this node is the parent of n @@ -107,8 +111,9 @@ public String getNextAncestorName(Node n) { name = name.substring(1); } int index=name.indexOf(PATH_SEPARATOR); - if (index !=-1) + if (index != -1) { name = name.substring(0, index); + } return name; } @@ -168,7 +173,8 @@ public boolean add(Node n) { * @see InnerNodeImpl(String, String, InnerNode, int) */ private InnerNodeImpl createParentNode(String parentName) { - return new InnerNodeImpl(parentName, getPath(this), this, this.getLevel()+1); + return new InnerNodeImpl(parentName, + getPath(this), this, this.getLevel() + 1); } @Override @@ -220,14 +226,16 @@ public boolean remove(Node n) { @Override public Node getLoc(String loc) { - if (loc == null || loc.length() == 0) return this; + if (loc == null || loc.length() == 0) { + return this; + } String[] path = loc.split(PATH_SEPARATOR_STR, 2); - Node childnode = childrenMap.get(path[0]); - if (childnode == null) return null; // non-existing node - if (path.length == 1) return childnode; - if (childnode instanceof InnerNode) { - return ((InnerNode)childnode).getLoc(path[1]); + Node childNode = childrenMap.get(path[0]); + if (childNode == null || path.length == 1) { + return childNode; + } else if (childNode instanceof InnerNode) { + return ((InnerNode)childNode).getLoc(path[1]); } else { return null; } @@ -237,11 +245,10 @@ public Node getLoc(String loc) { public Node getLeaf(int leafIndex, Node excludedNode) { int count=0; // check if the excluded node a leaf - boolean isLeaf = - excludedNode == null || !(excludedNode instanceof InnerNode); + boolean isLeaf = !(excludedNode instanceof InnerNode); // calculate the total number of excluded leaf nodes int numOfExcludedLeaves = - isLeaf ? 1 : ((InnerNode)excludedNode).getNumOfLeaves(); + isLeaf ? 1 : ((InnerNode)excludedNode).getNumOfLeaves(); if (isLeafParent()) { // children are leaves if (isLeaf) { // excluded node is a leaf node if (excludedNode != null && diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java index acdec9388e75d..d98254cb1ca25 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java @@ -804,7 +804,11 @@ public static IOException wrapException(final String destHost, + ";" + see("SocketException")); } else { - // Return instance of same type if Exception has a String constructor + // 1. Return instance of same type with exception msg if Exception has a + // String constructor. + // 2. Return instance of same type if Exception doesn't have a String + // constructor. + // Related HADOOP-16453. return wrapWithMessage(exception, "DestHost:destPort " + destHost + ":" + destPort + " , LocalHost:localPort " + localHost @@ -832,9 +836,9 @@ private static T wrapWithMessage( Constructor ctor = clazz.getConstructor(String.class); Throwable t = ctor.newInstance(msg); return (T)(t.initCause(exception)); + } catch (NoSuchMethodException e) { + return exception; } catch (Throwable e) { - LOG.trace("Unable to wrap exception of type {}: it has no (String) " - + "constructor", clazz, e); throw exception; } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java index 053c95972ce52..724cec37ba503 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java @@ -146,10 +146,8 @@ public void add(Node node) { if (rack == null) { incrementRacks(); } - if (!(node instanceof InnerNode)) { - if (depthOfAllLeaves == -1) { - depthOfAllLeaves = node.getLevel(); - } + if (depthOfAllLeaves == -1) { + depthOfAllLeaves = node.getLevel(); } } LOG.debug("NetworkTopology became:\n{}", this); @@ -295,22 +293,12 @@ public String getRack(String loc) { /** @return the total number of racks */ public int getNumOfRacks() { - netlock.readLock().lock(); - try { - return numOfRacks; - } finally { - netlock.readLock().unlock(); - } + return numOfRacks; } /** @return the total number of leaf nodes */ public int getNumOfLeaves() { - netlock.readLock().lock(); - try { - return clusterMap.getNumOfLeaves(); - } finally { - netlock.readLock().unlock(); - } + return clusterMap.getNumOfLeaves(); } /** Return the distance between two nodes @@ -569,10 +557,11 @@ protected Node chooseRandom(final String scope, String excludedScope, private Node chooseRandom(final InnerNode parentNode, final Node excludedScopeNode, final Collection excludedNodes, final int totalInScopeNodes, final int availableNodes) { - Preconditions.checkArgument( - totalInScopeNodes >= availableNodes && availableNodes > 0, String - .format("%d should >= %d, and both should be positive.", - totalInScopeNodes, availableNodes)); + if (totalInScopeNodes < availableNodes) { + LOG.warn("Total Nodes in scope : {} are less than Available Nodes : {}", + totalInScopeNodes, availableNodes); + return null; + } if (excludedNodes == null || excludedNodes.isEmpty()) { // if there are no excludedNodes, randomly choose a node final int index = r.nextInt(totalInScopeNodes); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java index 9255a844c4e43..8e71f69c858d1 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java @@ -17,12 +17,18 @@ */ package org.apache.hadoop.security; +import java.io.FileInputStream; import java.io.IOException; +import java.io.InputStream; import java.io.InputStreamReader; import java.io.Reader; +import java.net.InetAddress; +import java.net.Socket; import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Paths; +import java.security.GeneralSecurityException; +import java.security.KeyStore; import java.util.ArrayList; import java.util.Collections; import java.util.Hashtable; @@ -44,6 +50,13 @@ import javax.naming.ldap.LdapName; import javax.naming.ldap.Rdn; import javax.naming.spi.InitialContextFactory; +import javax.net.SocketFactory; +import javax.net.ssl.KeyManager; +import javax.net.ssl.KeyManagerFactory; +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLSocketFactory; +import javax.net.ssl.TrustManager; +import javax.net.ssl.TrustManagerFactory; import com.google.common.collect.Iterators; import org.apache.hadoop.classification.InterfaceAudience; @@ -273,6 +286,13 @@ public class LdapGroupsMapping public static final String LDAP_CTX_FACTORY_CLASS_DEFAULT = "com.sun.jndi.ldap.LdapCtxFactory"; + /** + * The env key used for specifying a custom socket factory to be used for + * creating connections to the LDAP server. This is not a Hadoop conf key. + */ + private static final String LDAP_SOCKET_FACTORY_ENV_KEY = + "java.naming.ldap.factory.socket"; + private static final Logger LOG = LoggerFactory.getLogger(LdapGroupsMapping.class); @@ -640,19 +660,13 @@ private DirContext getDirContext() throws NamingException { // Set up SSL security, if necessary if (useSsl) { env.put(Context.SECURITY_PROTOCOL, "ssl"); - if (!keystore.isEmpty()) { - System.setProperty("javax.net.ssl.keyStore", keystore); - } - if (!keystorePass.isEmpty()) { - System.setProperty("javax.net.ssl.keyStorePassword", keystorePass); - } - if (!truststore.isEmpty()) { - System.setProperty("javax.net.ssl.trustStore", truststore); - } - if (!truststorePass.isEmpty()) { - System.setProperty("javax.net.ssl.trustStorePassword", - truststorePass); - } + // It is necessary to use a custom socket factory rather than setting + // system properties to configure these options to avoid interfering + // with other SSL factories throughout the system + LdapSslSocketFactory.setConfigurations(keystore, keystorePass, + truststore, truststorePass); + env.put("java.naming.ldap.factory.socket", + LdapSslSocketFactory.class.getName()); } env.put(Context.SECURITY_PRINCIPAL, currentBindUser.username); @@ -929,4 +943,130 @@ public String toString() { return this.username; } } + + /** + * An private internal socket factory used to create SSL sockets with custom + * configuration. There is no way to pass a specific instance of a factory to + * the Java naming services, and the instantiated socket factory is not + * passed any contextual information, so all information must be encapsulated + * directly in the class. Static fields are used here to achieve this. This is + * safe since the only usage of {@link LdapGroupsMapping} is within + * {@link Groups}, which is a singleton (see the GROUPS field). + *

    + * This has nearly the same behavior as an {@link SSLSocketFactory}. The only + * additional logic is to configure the key store and trust store. + *

    + * This is public only to be accessible by the Java naming services. + */ + @InterfaceAudience.Private + public static class LdapSslSocketFactory extends SocketFactory { + + /** Cached value lazy-loaded by {@link #getDefault()}. */ + private static LdapSslSocketFactory defaultSslFactory; + + private static String keyStoreLocation; + private static String keyStorePassword; + private static String trustStoreLocation; + private static String trustStorePassword; + + private final SSLSocketFactory socketFactory; + + LdapSslSocketFactory(SSLSocketFactory wrappedSocketFactory) { + this.socketFactory = wrappedSocketFactory; + } + + public static synchronized SocketFactory getDefault() { + if (defaultSslFactory == null) { + try { + SSLContext context = SSLContext.getInstance("TLS"); + context.init(createKeyManagers(), createTrustManagers(), null); + defaultSslFactory = + new LdapSslSocketFactory(context.getSocketFactory()); + LOG.info("Successfully instantiated LdapSslSocketFactory with " + + "keyStoreLocation = {} and trustStoreLocation = {}", + keyStoreLocation, trustStoreLocation); + } catch (IOException | GeneralSecurityException e) { + throw new RuntimeException("Unable to create SSLSocketFactory", e); + } + } + return defaultSslFactory; + } + + static synchronized void setConfigurations(String newKeyStoreLocation, + String newKeyStorePassword, String newTrustStoreLocation, + String newTrustStorePassword) { + LdapSslSocketFactory.keyStoreLocation = newKeyStoreLocation; + LdapSslSocketFactory.keyStorePassword = newKeyStorePassword; + LdapSslSocketFactory.trustStoreLocation = newTrustStoreLocation; + LdapSslSocketFactory.trustStorePassword = newTrustStorePassword; + } + + private static KeyManager[] createKeyManagers() + throws IOException, GeneralSecurityException { + if (keyStoreLocation.isEmpty()) { + return null; + } + KeyManagerFactory keyMgrFactory = KeyManagerFactory + .getInstance(KeyManagerFactory.getDefaultAlgorithm()); + keyMgrFactory.init(createKeyStore(keyStoreLocation, keyStorePassword), + getPasswordCharArray(keyStorePassword)); + return keyMgrFactory.getKeyManagers(); + } + + private static TrustManager[] createTrustManagers() + throws IOException, GeneralSecurityException { + if (trustStoreLocation.isEmpty()) { + return null; + } + TrustManagerFactory trustMgrFactory = TrustManagerFactory + .getInstance(TrustManagerFactory.getDefaultAlgorithm()); + trustMgrFactory.init( + createKeyStore(trustStoreLocation, trustStorePassword)); + return trustMgrFactory.getTrustManagers(); + } + + private static KeyStore createKeyStore(String location, String password) + throws IOException, GeneralSecurityException { + KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType()); + try (InputStream keyStoreInput = new FileInputStream(location)) { + keyStore.load(keyStoreInput, getPasswordCharArray(password)); + } + return keyStore; + } + + private static char[] getPasswordCharArray(String password) { + if (password == null || password.isEmpty()) { + return null; + } + return password.toCharArray(); + } + + @Override + public Socket createSocket() throws IOException { + return socketFactory.createSocket(); + } + + @Override + public Socket createSocket(String host, int port) throws IOException { + return socketFactory.createSocket(host, port); + } + + @Override + public Socket createSocket(String host, int port, InetAddress localHost, + int localPort) throws IOException { + return socketFactory.createSocket(host, port, localHost, localPort); + } + + @Override + public Socket createSocket(InetAddress host, int port) throws IOException { + return socketFactory.createSocket(host, port); + } + + @Override + public Socket createSocket(InetAddress address, int port, + InetAddress localAddress, int localPort) throws IOException { + return socketFactory.createSocket(address, port, localAddress, localPort); + } + } + } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java index 5696118c162b7..603772444bcef 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java @@ -44,7 +44,8 @@ public class CredentialShell extends CommandShell { " [-help]\n" + " [" + CreateCommand.USAGE + "]\n" + " [" + DeleteCommand.USAGE + "]\n" + - " [" + ListCommand.USAGE + "]\n"; + " [" + ListCommand.USAGE + "]\n" + + " [" + CheckCommand.USAGE + "]\n"; @VisibleForTesting public static final String NO_VALID_PROVIDERS = "There are no valid (non-transient) providers configured.\n" + @@ -66,6 +67,7 @@ public class CredentialShell extends CommandShell { * 

        * % hadoop credential create alias [-provider providerPath]
    * % hadoop credential list [-provider providerPath]
+   * % hadoop credential check alias [-provider providerPath]
    * % hadoop credential delete alias [-provider providerPath] [-f]
    *
    * @param args @@ -86,6 +88,11 @@ protected int init(String[] args) throws IOException { return 1; } setSubCommand(new CreateCommand(args[++i])); + } else if (args[i].equals("check")) { + if (i == args.length - 1) { + return 1; + } + setSubCommand(new CheckCommand(args[++i])); } else if (args[i].equals("delete")) { if (i == args.length - 1) { return 1; @@ -293,6 +300,91 @@ public String getUsage() { } } + private class CheckCommand extends Command { + public static final String USAGE = "check [-value alias-value] " + + "[-provider provider-path] [-strict]"; + public static final String DESC = + "The check subcommand check a password for the name\n" + + "specified as the argument within the provider indicated\n" + + "through the -provider argument. If -strict is supplied, fail\n" + + "immediately if the provider requires a password and none is given.\n" + + "If -value is provided, use that for the value of the credential\n" + + "instead of prompting the user."; + + private String alias = null; + + CheckCommand(String alias) { + this.alias = alias; + } + + public boolean validate() { + if (alias == null) { + getOut().println("There is no alias specified. Please provide the" + + "mandatory . See the usage description with -help."); + return false; + } + if (alias.equals("-help")) { + return true; + } + try { + provider = getCredentialProvider(); + if (provider == null) { + return false; + } else if (provider.needsPassword()) { + if (strict) { + getOut().println(provider.noPasswordError()); + return false; + } else { + getOut().println(provider.noPasswordWarning()); + } + } + } catch (IOException e) { + e.printStackTrace(getErr()); + } + return true; + } + + public void execute() throws IOException, NoSuchAlgorithmException { + if (alias.equals("-help")) { + doHelp(); + return; + } + warnIfTransientProvider(); + getOut().println("Checking aliases for CredentialProvider: " + + provider.toString()); + try { + PasswordReader c = getPasswordReader(); + if (c == null) { + throw new IOException("No console available for checking user."); + } + + char[] password = null; + if (value != null) { + // testing only + password = value.toCharArray(); + } else { + password = c.readPassword("Enter alias password: "); + } + char[] storePassword = + provider.getCredentialEntry(alias).getCredential(); + String beMatch = + Arrays.equals(storePassword, password) ? "success" : "failed"; + + getOut().println("Password match " + beMatch + " for " + alias + "."); + } catch (IOException e) { + getOut().println("Cannot check aliases for CredentialProvider: " + + provider.toString() + + ": " + e.getMessage()); + throw e; + } + } + + @Override + public String getUsage() { + return USAGE + ":\n\n" + DESC; + } + } + private class CreateCommand extends Command { public static final String USAGE = "create [-value alias-value] " + "[-provider provider-path] [-strict]"; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authentication/server/ProxyUserAuthenticationFilter.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authentication/server/ProxyUserAuthenticationFilter.java new file mode 100644 index 0000000000000..bd04efeed2202 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authentication/server/ProxyUserAuthenticationFilter.java @@ -0,0 +1,199 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. See accompanying LICENSE file. + */ +package org.apache.hadoop.security.authentication.server; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.security.authorize.AuthorizationException; +import org.apache.hadoop.security.authorize.ProxyUsers; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.util.HttpExceptionUtils; +import org.apache.hadoop.util.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.security.Principal; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import javax.servlet.FilterChain; +import javax.servlet.FilterConfig; +import javax.servlet.ServletException; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletRequestWrapper; +import javax.servlet.http.HttpServletResponse; + +/** + * AuthenticationFilter which adds support to perform operations + * using end user instead of proxy user. Fetches the end user from + * doAs Query Parameter. + */ +public class ProxyUserAuthenticationFilter extends AuthenticationFilter { + + private static final Logger LOG = LoggerFactory.getLogger( + ProxyUserAuthenticationFilter.class); + + private static final String DO_AS = "doas"; + public static final String PROXYUSER_PREFIX = "proxyuser"; + + @Override + public void init(FilterConfig filterConfig) throws ServletException { + Configuration conf = getProxyuserConfiguration(filterConfig); + ProxyUsers.refreshSuperUserGroupsConfiguration(conf, PROXYUSER_PREFIX); + super.init(filterConfig); + } + + @Override + protected void doFilter(FilterChain filterChain, HttpServletRequest request, + HttpServletResponse response) throws IOException, ServletException { + final HttpServletRequest lowerCaseRequest = toLowerCase(request); + String doAsUser = lowerCaseRequest.getParameter(DO_AS); + + if (doAsUser != null && !doAsUser.equals(request.getRemoteUser())) { + LOG.debug("doAsUser = {}, RemoteUser = {} , RemoteAddress = {} ", + doAsUser, request.getRemoteUser(), request.getRemoteAddr()); + UserGroupInformation requestUgi = (request.getUserPrincipal() != null) ? + UserGroupInformation.createRemoteUser(request.getRemoteUser()) + : null; + if (requestUgi != null) { + requestUgi = UserGroupInformation.createProxyUser(doAsUser, + requestUgi); + try { + ProxyUsers.authorize(requestUgi, request.getRemoteAddr()); + + final UserGroupInformation ugiF = requestUgi; + request = new HttpServletRequestWrapper(request) { + @Override + public String getRemoteUser() { + return ugiF.getShortUserName(); + } + + @Override + public Principal getUserPrincipal() { + return new Principal() { + @Override + public String getName() { + return ugiF.getUserName(); + } + }; + } + }; + LOG.debug("Proxy user Authentication successful"); + } catch (AuthorizationException ex) { + HttpExceptionUtils.createServletExceptionResponse(response, + HttpServletResponse.SC_FORBIDDEN, ex); + LOG.warn("Proxy user Authentication exception", ex); + return; + } + } + } + super.doFilter(filterChain, request, response); + } + + protected Configuration getProxyuserConfiguration(FilterConfig filterConfig) + throws ServletException { + Configuration conf = new Configuration(false); + Enumeration names = filterConfig.getInitParameterNames(); + while (names.hasMoreElements()) { + String name = (String) names.nextElement(); + if (name.startsWith(PROXYUSER_PREFIX + ".")) { + String value = filterConfig.getInitParameter(name); + conf.set(name, value); + } + } + return conf; + } + + static boolean containsUpperCase(final Iterable strings) { + for(String s : strings) { + for(int i = 0; i < s.length(); i++) { + if (Character.isUpperCase(s.charAt(i))) { + return true; + } + } + } + return false; + } + + public static HttpServletRequest toLowerCase( + final HttpServletRequest request) { + @SuppressWarnings("unchecked") + final Map original = (Map) + request.getParameterMap(); + if (!containsUpperCase(original.keySet())) { + return request; + } + + final Map> m = new HashMap>(); + for (Map.Entry entry : original.entrySet()) { + final String key = StringUtils.toLowerCase(entry.getKey()); + List strings = m.get(key); + if (strings == null) { + strings = new ArrayList(); + m.put(key, strings); + } + for (String v : entry.getValue()) { + strings.add(v); + } + } + + return new HttpServletRequestWrapper(request) { + private Map parameters = null; + + @Override + public Map getParameterMap() { + if (parameters == null) { + parameters = new HashMap(); + for (Map.Entry> entry : m.entrySet()) { + final List a = entry.getValue(); + parameters.put(entry.getKey(), a.toArray(new String[a.size()])); + } + } + return parameters; + } + + @Override + public String getParameter(String name) { + final List a = m.get(name); + return a == null ? null : a.get(0); + } + + @Override + public String[] getParameterValues(String name) { + return getParameterMap().get(name); + } + + @Override + public Enumeration getParameterNames() { + final Iterator i = m.keySet().iterator(); + return new Enumeration() { + @Override + public boolean hasMoreElements() { + return i.hasNext(); + } + + @Override + public String nextElement() { + return i.next(); + } + }; + } + }; + } + +} + diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authentication/server/ProxyUserAuthenticationFilterInitializer.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authentication/server/ProxyUserAuthenticationFilterInitializer.java new file mode 100644 index 0000000000000..ad79742aeb528 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authentication/server/ProxyUserAuthenticationFilterInitializer.java @@ -0,0 +1,60 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.security.authentication.server; + +import java.util.Map; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.http.FilterContainer; +import org.apache.hadoop.http.FilterInitializer; +import org.apache.hadoop.security.AuthenticationFilterInitializer; +import org.apache.hadoop.security.authorize.ProxyUsers; + +/** + * Filter initializer to initialize + * {@link ProxyUserAuthenticationFilter} which adds support + * to perform operations using end user instead of proxy user. + */ +public class ProxyUserAuthenticationFilterInitializer + extends FilterInitializer { + + private String configPrefix; + + public ProxyUserAuthenticationFilterInitializer() { + this.configPrefix = "hadoop.http.authentication."; + } + + protected Map createFilterConfig(Configuration conf) { + Map filterConfig = AuthenticationFilterInitializer + .getFilterConfigMap(conf, configPrefix); + //Add proxy user configs + for (Map.Entry entry : conf.getPropsWithPrefix( + ProxyUsers.CONF_HADOOP_PROXYUSER).entrySet()) { + filterConfig.put("proxyuser" + entry.getKey(), entry.getValue()); + } + return filterConfig; + } + + @Override + public void initFilter(FilterContainer container, Configuration conf) { + Map filterConfig = createFilterConfig(conf); + container.addFilter("ProxyUserAuthenticationFilter", + ProxyUserAuthenticationFilter.class.getName(), filterConfig); + } + +} \ No newline at end of file diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authentication/server/package-info.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authentication/server/package-info.java new file mode 100644 index 0000000000000..b0accf71b351a --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authentication/server/package-info.java @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * Provides the server-side framework for authentication. + */ +package org.apache.hadoop.security.authentication.server; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ServiceAuthorizationManager.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ServiceAuthorizationManager.java index 4c47348fa554c..a264eb4dcd9fb 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ServiceAuthorizationManager.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ServiceAuthorizationManager.java @@ -97,21 +97,23 @@ public void authorize(UserGroupInformation user, throw new AuthorizationException("Protocol " + protocol + " is not known."); } - - // get client principal key to verify (if available) - KerberosInfo krbInfo = SecurityUtil.getKerberosInfo(protocol, conf); - String clientPrincipal = null; - if (krbInfo != null) { - String clientKey = krbInfo.clientPrincipal(); - if (clientKey != null && !clientKey.isEmpty()) { - try { - clientPrincipal = SecurityUtil.getServerPrincipal( - conf.get(clientKey), addr); - } catch (IOException e) { - throw (AuthorizationException) new AuthorizationException( - "Can't figure out Kerberos principal name for connection from " - + addr + " for user=" + user + " protocol=" + protocol) - .initCause(e); + + String clientPrincipal = null; + if (UserGroupInformation.isSecurityEnabled()) { + // get client principal key to verify (if available) + KerberosInfo krbInfo = SecurityUtil.getKerberosInfo(protocol, conf); + if (krbInfo != null) { + String clientKey = krbInfo.clientPrincipal(); + if (clientKey != null && !clientKey.isEmpty()) { + try { + clientPrincipal = SecurityUtil.getServerPrincipal( + conf.get(clientKey), addr); + } catch (IOException e) { + throw (AuthorizationException) new AuthorizationException( + "Can't figure out Kerberos principal name for connection from " + + addr + " for user=" + user + " protocol=" + protocol) + .initCause(e); + } } } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/CrossOriginFilter.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/CrossOriginFilter.java index 02c168f7b6044..60c2864bbe539 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/CrossOriginFilter.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/CrossOriginFilter.java @@ -197,7 +197,7 @@ private void initializeAllowedOrigins(FilterConfig filterConfig) { LOG.info("Allowed Origins: " + StringUtils.join(allowedOrigins, ',')); LOG.info("Allow All Origins: " + allowAllOrigins); List discouragedAllowedOrigins = allowedOrigins.stream() - .filter(s -> s.length() > 1 && s.contains("*")) + .filter(s -> s.length() > 1 && s.contains("*") && !(s.startsWith(ALLOWED_ORIGINS_REGEX_PREFIX))) .collect(Collectors.toList()); for (String discouragedAllowedOrigin : discouragedAllowedOrigins) { LOG.warn("Allowed Origin pattern '" + discouragedAllowedOrigin + "' is discouraged, use the 'regex:' prefix and use a Java regular expression instead."); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/DelegatingSSLSocketFactory.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/DelegatingSSLSocketFactory.java new file mode 100644 index 0000000000000..ad97a99c6ddc2 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/DelegatingSSLSocketFactory.java @@ -0,0 +1,287 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.security.ssl; + +import java.io.IOException; +import java.net.InetAddress; +import java.net.Socket; +import java.net.SocketException; +import java.security.KeyManagementException; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.logging.Level; + +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLSocket; +import javax.net.ssl.SSLSocketFactory; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.wildfly.openssl.OpenSSLProvider; +import org.wildfly.openssl.SSL; + + +/** + * A {@link SSLSocketFactory} that can delegate to various SSL implementations. + * Specifically, either OpenSSL or JSSE can be used. OpenSSL offers better + * performance than JSSE and is made available via the + * wildlfy-openssl + * library. + * + *

    + * The factory has several different modes of operation: + *

      + *
    • OpenSSL: Uses the wildly-openssl library to delegate to the + * system installed OpenSSL. If the wildfly-openssl integration is not + * properly setup, an exception is thrown.
      • + *
    • Default: Attempts to use the OpenSSL mode, if it cannot load the + * necessary libraries, it falls back to the Default_JSEE mode.
      • + *
    • Default_JSSE: Delegates to the JSSE implementation of SSL, but + * it disables the GCM cipher when running on Java 8.
      • + *
    • Default_JSSE_with_GCM: Delegates to the JSSE implementation of + * SSL with no modification to the list of enabled ciphers.
      • + *
    + *

    + */ +public final class DelegatingSSLSocketFactory extends SSLSocketFactory { + + /** + * Default indicates Ordered, preferred OpenSSL, if failed to load then fall + * back to Default_JSSE. + * + *

    + * Default_JSSE is not truly the the default JSSE implementation because + * the GCM cipher is disabled when running on Java 8. However, the name + * was not changed in order to preserve backwards compatibility. Instead, + * a new mode called Default_JSSE_with_GCM delegates to the default JSSE + * implementation with no changes to the list of enabled ciphers. + *

    + */ + public enum SSLChannelMode { + OpenSSL, + Default, + Default_JSSE, + Default_JSSE_with_GCM + } + + private static DelegatingSSLSocketFactory instance = null; + private static final Logger LOG = LoggerFactory.getLogger( + DelegatingSSLSocketFactory.class); + private String providerName; + private SSLContext ctx; + private String[] ciphers; + private SSLChannelMode channelMode; + + // This should only be modified within the #initializeDefaultFactory + // method which is synchronized + private boolean openSSLProviderRegistered; + + /** + * Initialize a singleton SSL socket factory. + * + * @param preferredMode applicable only if the instance is not initialized. + * @throws IOException if an error occurs. + */ + public static synchronized void initializeDefaultFactory( + SSLChannelMode preferredMode) throws IOException { + if (instance == null) { + instance = new DelegatingSSLSocketFactory(preferredMode); + } + } + + /** + * Singletone instance of the SSLSocketFactory. + * + * SSLSocketFactory must be initialized with appropriate SSLChannelMode + * using initializeDefaultFactory method. + * + * @return instance of the SSLSocketFactory, instance must be initialized by + * initializeDefaultFactory. + */ + public static DelegatingSSLSocketFactory getDefaultFactory() { + return instance; + } + + private DelegatingSSLSocketFactory(SSLChannelMode preferredChannelMode) + throws IOException { + try { + initializeSSLContext(preferredChannelMode); + } catch (NoSuchAlgorithmException e) { + throw new IOException(e); + } catch (KeyManagementException e) { + throw new IOException(e); + } + + // Get list of supported cipher suits from the SSL factory. + SSLSocketFactory factory = ctx.getSocketFactory(); + String[] defaultCiphers = factory.getSupportedCipherSuites(); + String version = System.getProperty("java.version"); + + ciphers = (channelMode == SSLChannelMode.Default_JSSE + && version.startsWith("1.8")) + ? alterCipherList(defaultCiphers) : defaultCiphers; + + providerName = ctx.getProvider().getName() + "-" + + ctx.getProvider().getVersion(); + } + + private void initializeSSLContext(SSLChannelMode preferredChannelMode) + throws NoSuchAlgorithmException, KeyManagementException { + switch (preferredChannelMode) { + case Default: + if (!openSSLProviderRegistered) { + OpenSSLProvider.register(); + openSSLProviderRegistered = true; + } + try { + java.util.logging.Logger logger = java.util.logging.Logger.getLogger( + SSL.class.getName()); + logger.setLevel(Level.WARNING); + ctx = SSLContext.getInstance("openssl.TLS"); + ctx.init(null, null, null); + // Strong reference needs to be kept to logger until initialization of + // SSLContext finished (see HADOOP-16174): + logger.setLevel(Level.INFO); + channelMode = SSLChannelMode.OpenSSL; + } catch (NoSuchAlgorithmException e) { + LOG.debug("Failed to load OpenSSL. Falling back to the JSSE default."); + ctx = SSLContext.getDefault(); + channelMode = SSLChannelMode.Default_JSSE; + } + break; + case OpenSSL: + if (!openSSLProviderRegistered) { + OpenSSLProvider.register(); + openSSLProviderRegistered = true; + } + ctx = SSLContext.getInstance("openssl.TLS"); + ctx.init(null, null, null); + channelMode = SSLChannelMode.OpenSSL; + break; + case Default_JSSE: + ctx = SSLContext.getDefault(); + channelMode = SSLChannelMode.Default_JSSE; + break; + case Default_JSSE_with_GCM: + ctx = SSLContext.getDefault(); + channelMode = SSLChannelMode.Default_JSSE_with_GCM; + break; + default: + throw new NoSuchAlgorithmException("Unknown channel mode: " + + preferredChannelMode); + } + } + + public String getProviderName() { + return providerName; + } + + @Override + public String[] getDefaultCipherSuites() { + return ciphers.clone(); + } + + @Override + public String[] getSupportedCipherSuites() { + return ciphers.clone(); + } + + public Socket createSocket() throws IOException { + SSLSocketFactory factory = ctx.getSocketFactory(); + SSLSocket ss = (SSLSocket) factory.createSocket(); + configureSocket(ss); + return ss; + } + + @Override + public Socket createSocket(Socket s, String host, int port, + boolean autoClose) throws IOException { + SSLSocketFactory factory = ctx.getSocketFactory(); + SSLSocket ss = (SSLSocket) factory.createSocket(s, host, port, autoClose); + + configureSocket(ss); + return ss; + } + + @Override + public Socket createSocket(InetAddress address, int port, + InetAddress localAddress, int localPort) + throws IOException { + SSLSocketFactory factory = ctx.getSocketFactory(); + SSLSocket ss = (SSLSocket) factory + .createSocket(address, port, localAddress, localPort); + + configureSocket(ss); + return ss; + } + + @Override + public Socket createSocket(String host, int port, InetAddress localHost, + int localPort) throws IOException { + SSLSocketFactory factory = ctx.getSocketFactory(); + SSLSocket ss = (SSLSocket) factory + .createSocket(host, port, localHost, localPort); + + configureSocket(ss); + + return ss; + } + + @Override + public Socket createSocket(InetAddress host, int port) throws IOException { + SSLSocketFactory factory = ctx.getSocketFactory(); + SSLSocket ss = (SSLSocket) factory.createSocket(host, port); + + configureSocket(ss); + + return ss; + } + + @Override + public Socket createSocket(String host, int port) throws IOException { + SSLSocketFactory factory = ctx.getSocketFactory(); + SSLSocket ss = (SSLSocket) factory.createSocket(host, port); + + configureSocket(ss); + + return ss; + } + + private void configureSocket(SSLSocket ss) throws SocketException { + ss.setEnabledCipherSuites(ciphers); + } + + private String[] alterCipherList(String[] defaultCiphers) { + + ArrayList preferredSuits = new ArrayList<>(); + + // Remove GCM mode based ciphers from the supported list. + for (int i = 0; i < defaultCiphers.length; i++) { + if (defaultCiphers[i].contains("_GCM_")) { + LOG.debug("Removed Cipher - {} from list of enabled SSLSocket ciphers", + defaultCiphers[i]); + } else { + preferredSuits.add(defaultCiphers[i]); + } + } + + ciphers = preferredSuits.toArray(new String[0]); + return ciphers; + } +} \ No newline at end of file diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/OpenSSLSocketFactory.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/OpenSSLSocketFactory.java deleted file mode 100644 index 99fc195133dc8..0000000000000 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/OpenSSLSocketFactory.java +++ /dev/null @@ -1,248 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.security.ssl; - -import java.io.IOException; -import java.net.InetAddress; -import java.net.Socket; -import java.net.SocketException; -import java.security.KeyManagementException; -import java.security.NoSuchAlgorithmException; -import java.util.ArrayList; -import java.util.logging.Level; - -import javax.net.ssl.SSLContext; -import javax.net.ssl.SSLSocket; -import javax.net.ssl.SSLSocketFactory; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.wildfly.openssl.OpenSSLProvider; -import org.wildfly.openssl.SSL; - - -/** - * Extension to use native OpenSSL library instead of JSSE for better - * performance. - * - */ -public final class OpenSSLSocketFactory extends SSLSocketFactory { - - /** - * Default indicates Ordered, preferred OpenSSL, if failed to load then fall - * back to Default_JSSE. - */ - public enum SSLChannelMode { - OpenSSL, - Default, - Default_JSSE - } - - private static OpenSSLSocketFactory instance = null; - private static final Logger LOG = LoggerFactory.getLogger( - OpenSSLSocketFactory.class); - private String providerName; - private SSLContext ctx; - private String[] ciphers; - private SSLChannelMode channelMode; - - /** - * Initialize a singleton SSL socket factory. - * - * @param preferredMode applicable only if the instance is not initialized. - * @throws IOException if an error occurs. - */ - public static synchronized void initializeDefaultFactory( - SSLChannelMode preferredMode) throws IOException { - if (instance == null) { - instance = new OpenSSLSocketFactory(preferredMode); - } - } - - /** - * Singletone instance of the SSLSocketFactory. - * - * SSLSocketFactory must be initialized with appropriate SSLChannelMode - * using initializeDefaultFactory method. - * - * @return instance of the SSLSocketFactory, instance must be initialized by - * initializeDefaultFactory. - */ - public static OpenSSLSocketFactory getDefaultFactory() { - return instance; - } - - static { - OpenSSLProvider.register(); - } - - private OpenSSLSocketFactory(SSLChannelMode preferredChannelMode) - throws IOException { - try { - initializeSSLContext(preferredChannelMode); - } catch (NoSuchAlgorithmException e) { - throw new IOException(e); - } catch (KeyManagementException e) { - throw new IOException(e); - } - - // Get list of supported cipher suits from the SSL factory. - SSLSocketFactory factory = ctx.getSocketFactory(); - String[] defaultCiphers = factory.getSupportedCipherSuites(); - String version = System.getProperty("java.version"); - - ciphers = (channelMode == SSLChannelMode.Default_JSSE - && version.startsWith("1.8")) - ? alterCipherList(defaultCiphers) : defaultCiphers; - - providerName = ctx.getProvider().getName() + "-" - + ctx.getProvider().getVersion(); - } - - private void initializeSSLContext(SSLChannelMode preferredChannelMode) - throws NoSuchAlgorithmException, KeyManagementException { - switch (preferredChannelMode) { - case Default: - try { - java.util.logging.Logger logger = java.util.logging.Logger.getLogger( - SSL.class.getName()); - logger.setLevel(Level.WARNING); - ctx = SSLContext.getInstance("openssl.TLS"); - ctx.init(null, null, null); - // Strong reference needs to be kept to logger until initialization of - // SSLContext finished (see HADOOP-16174): - logger.setLevel(Level.INFO); - channelMode = SSLChannelMode.OpenSSL; - } catch (NoSuchAlgorithmException e) { - LOG.warn("Failed to load OpenSSL. Falling back to the JSSE default."); - ctx = SSLContext.getDefault(); - channelMode = SSLChannelMode.Default_JSSE; - } - break; - case OpenSSL: - ctx = SSLContext.getInstance("openssl.TLS"); - ctx.init(null, null, null); - channelMode = SSLChannelMode.OpenSSL; - break; - case Default_JSSE: - ctx = SSLContext.getDefault(); - channelMode = SSLChannelMode.Default_JSSE; - break; - default: - throw new AssertionError("Unknown channel mode: " - + preferredChannelMode); - } - } - - public String getProviderName() { - return providerName; - } - - @Override - public String[] getDefaultCipherSuites() { - return ciphers.clone(); - } - - @Override - public String[] getSupportedCipherSuites() { - return ciphers.clone(); - } - - public Socket createSocket() throws IOException { - SSLSocketFactory factory = ctx.getSocketFactory(); - SSLSocket ss = (SSLSocket) factory.createSocket(); - configureSocket(ss); - return ss; - } - - @Override - public Socket createSocket(Socket s, String host, int port, - boolean autoClose) throws IOException { - SSLSocketFactory factory = ctx.getSocketFactory(); - SSLSocket ss = (SSLSocket) factory.createSocket(s, host, port, autoClose); - - configureSocket(ss); - return ss; - } - - @Override - public Socket createSocket(InetAddress address, int port, - InetAddress localAddress, int localPort) - throws IOException { - SSLSocketFactory factory = ctx.getSocketFactory(); - SSLSocket ss = (SSLSocket) factory - .createSocket(address, port, localAddress, localPort); - - configureSocket(ss); - return ss; - } - - @Override - public Socket createSocket(String host, int port, InetAddress localHost, - int localPort) throws IOException { - SSLSocketFactory factory = ctx.getSocketFactory(); - SSLSocket ss = (SSLSocket) factory - .createSocket(host, port, localHost, localPort); - - configureSocket(ss); - - return ss; - } - - @Override - public Socket createSocket(InetAddress host, int port) throws IOException { - SSLSocketFactory factory = ctx.getSocketFactory(); - SSLSocket ss = (SSLSocket) factory.createSocket(host, port); - - configureSocket(ss); - - return ss; - } - - @Override - public Socket createSocket(String host, int port) throws IOException { - SSLSocketFactory factory = ctx.getSocketFactory(); - SSLSocket ss = (SSLSocket) factory.createSocket(host, port); - - configureSocket(ss); - - return ss; - } - - private void configureSocket(SSLSocket ss) throws SocketException { - ss.setEnabledCipherSuites(ciphers); - } - - private String[] alterCipherList(String[] defaultCiphers) { - - ArrayList preferredSuits = new ArrayList<>(); - - // Remove GCM mode based ciphers from the supported list. - for (int i = 0; i < defaultCiphers.length; i++) { - if (defaultCiphers[i].contains("_GCM_")) { - LOG.debug("Removed Cipher - " + defaultCiphers[i]); - } else { - preferredSuits.add(defaultCiphers[i]); - } - } - - ciphers = preferredSuits.toArray(new String[0]); - return ciphers; - } -} \ No newline at end of file diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLFactory.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLFactory.java index a7548aac0c94b..e10741e58a5ce 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLFactory.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLFactory.java @@ -72,7 +72,7 @@ public enum Mode { CLIENT, SERVER } public static final String SSL_ENABLED_PROTOCOLS_KEY = "hadoop.ssl.enabled.protocols"; public static final String SSL_ENABLED_PROTOCOLS_DEFAULT = - "TLSv1.1,TLSv1.2"; + "TLSv1.2"; public static final String SSL_SERVER_NEED_CLIENT_AUTH = "ssl.server.need.client.auth"; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java index 94481a27730e7..487dd4625202e 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java @@ -56,7 +56,6 @@ public class Token implements Writable { private Text kind; private Text service; private TokenRenewer renewer; - private byte[] dnHandshakeSecret; /** * Construct a token given a token identifier and a secret manager for the @@ -69,7 +68,14 @@ public Token(T id, SecretManager mgr) { identifier = id.getBytes(); kind = id.getKind(); service = new Text(); - dnHandshakeSecret = new byte[0]; + } + + public void setID(byte[] bytes) { + identifier = bytes; + } + + public void setPassword(byte[] newPassword) { + password = newPassword; } /** @@ -84,7 +90,6 @@ public Token(byte[] identifier, byte[] password, Text kind, Text service) { this.password = (password == null)? new byte[0] : password; this.kind = (kind == null)? new Text() : kind; this.service = (service == null)? new Text() : service; - this.dnHandshakeSecret = new byte[0]; } /** @@ -95,7 +100,6 @@ public Token() { password = new byte[0]; kind = new Text(); service = new Text(); - dnHandshakeSecret = new byte[0]; } /** @@ -107,7 +111,6 @@ public Token(Token other) { this.password = other.password.clone(); this.kind = new Text(other.kind); this.service = new Text(other.service); - this.dnHandshakeSecret = other.dnHandshakeSecret.clone(); } public Token copyToken() { @@ -123,7 +126,6 @@ public Token(TokenProto tokenPB) { this.password = tokenPB.getPassword().toByteArray(); this.kind = new Text(tokenPB.getKindBytes().toByteArray()); this.service = new Text(tokenPB.getServiceBytes().toByteArray()); - this.dnHandshakeSecret = new byte[0]; } /** @@ -149,14 +151,6 @@ public byte[] getIdentifier() { return identifier; } - public byte[] getDnHandshakeSecret() { - return dnHandshakeSecret; - } - - public void setDNHandshakeSecret(byte[] secret) { - this.dnHandshakeSecret = secret; - } - private static Class getClassForIdentifier(Text kind) { Class cls = null; @@ -351,11 +345,6 @@ public void readFields(DataInput in) throws IOException { in.readFully(password); kind.readFields(in); service.readFields(in); - len = WritableUtils.readVInt(in); - if (dnHandshakeSecret == null || dnHandshakeSecret.length != len) { - dnHandshakeSecret = new byte[len]; - } - in.readFully(dnHandshakeSecret); } @Override @@ -366,8 +355,6 @@ public void write(DataOutput out) throws IOException { out.write(password); kind.write(out); service.write(out); - WritableUtils.writeVInt(out, dnHandshakeSecret.length); - out.write(dnHandshakeSecret); } /** diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java index dca9e2f915994..f61590c28ebce 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java @@ -52,13 +52,14 @@ import org.apache.hadoop.classification.InterfaceAudience.Private; import org.apache.hadoop.classification.InterfaceStability.Unstable; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.security.SecurityUtil; import org.apache.hadoop.security.token.Token; import org.apache.hadoop.security.token.delegation.web.DelegationTokenManager; import org.apache.zookeeper.CreateMode; import org.apache.zookeeper.KeeperException; import org.apache.zookeeper.KeeperException.NoNodeException; import org.apache.zookeeper.ZooDefs.Perms; -import org.apache.zookeeper.client.ZooKeeperSaslClient; +import org.apache.zookeeper.client.ZKClientConfig; import org.apache.zookeeper.data.ACL; import org.apache.zookeeper.data.Id; import org.slf4j.Logger; @@ -172,8 +173,8 @@ public ZKDelegationTokenSecretManager(Configuration conf) { LOG.info("Connecting to ZooKeeper with SASL/Kerberos" + "and using 'sasl' ACLs"); String principal = setJaasConfiguration(conf); - System.setProperty(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY, - JAAS_LOGIN_ENTRY_NAME); + System.setProperty(ZKClientConfig.LOGIN_CONTEXT_NAME_KEY, + JAAS_LOGIN_ENTRY_NAME); System.setProperty("zookeeper.authProvider.1", "org.apache.zookeeper.server.auth.SASLAuthenticationProvider"); aclProvider = new SASLOwnerACLProvider(principal); @@ -221,6 +222,7 @@ private String setJaasConfiguration(Configuration config) throws Exception { } String principal = config.get(ZK_DTSM_ZK_KERBEROS_PRINCIPAL, "").trim(); + principal = SecurityUtil.getServerPrincipal(principal, ""); if (principal == null || principal.length() == 0) { throw new IllegalArgumentException(ZK_DTSM_ZK_KERBEROS_PRINCIPAL + " must be specified"); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/CompositeService.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/CompositeService.java index a5e8c895c6571..4aa2f23fad730 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/CompositeService.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/CompositeService.java @@ -93,7 +93,8 @@ protected boolean addIfService(Object object) { } } - protected synchronized boolean removeService(Service service) { + protected boolean removeService(Service service) { + LOG.debug("Removing service {}", service.getName()); synchronized (serviceList) { return serviceList.remove(service); } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/ServiceLaunchException.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/ServiceLaunchException.java index 1243a1fabfa50..fdb60236fe645 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/ServiceLaunchException.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/ServiceLaunchException.java @@ -78,4 +78,18 @@ public ServiceLaunchException(int exitCode, String format, Object... args) { } } + /** + * Create a formatted exception. + *

    + * This uses {@link String#format(String, Object...)} + * to build the formatted exception in the ENGLISH locale. + * @param exitCode exit code + * @param cause inner cause + * @param format format for message to use in exception + * @param args list of arguments + */ + public ServiceLaunchException(int exitCode, Throwable cause, + String format, Object... args) { + super(exitCode, String.format(Locale.ENGLISH, format, args), cause); + } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/ServiceLauncher.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/ServiceLauncher.java index da91a3d0e6c70..5e8a1f4eb21fb 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/ServiceLauncher.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/ServiceLauncher.java @@ -200,7 +200,7 @@ public ServiceLauncher(String serviceName, String serviceClassName) { * Get the service. * * Null until - * {@link #coreServiceLaunch(Configuration, List, boolean, boolean)} + * {@link #coreServiceLaunch(Configuration, Service, List, boolean, boolean)} * has completed. * @return the service */ @@ -303,7 +303,7 @@ public void launchServiceAndExit(List args) { exitException = e; noteException(exitException); } - if (exitException.getExitCode() != 0) { + if (exitException.getExitCode() == LauncherExitCodes.EXIT_USAGE) { // something went wrong. Print the usage and commands System.err.println(getUsageMessage()); System.err.println("Command: " + argumentString); @@ -328,8 +328,18 @@ protected void bindCommandOptions() { * @param exitException exception */ void noteException(ExitUtil.ExitException exitException) { - LOG.debug("Exception raised", exitException); - serviceExitCode = exitException.getExitCode(); + int exitCode = exitException.getExitCode(); + if (exitCode != 0) { + LOG.debug("Exception raised with exit code {}", + exitCode, + exitException); + Throwable cause = exitException.getCause(); + if (cause != null) { + // log the nested exception in more detail + LOG.warn("{}", cause.toString(), cause); + } + } + serviceExitCode = exitCode; serviceException = exitException; } @@ -451,17 +461,38 @@ public int loadConfigurationClasses() { * @param execute execute/wait for the service to stop. * @return an exit exception, which will have a status code of 0 if it worked */ - @VisibleForTesting public ExitUtil.ExitException launchService(Configuration conf, List processedArgs, boolean addShutdownHook, boolean execute) { - + return launchService(conf, null, processedArgs, addShutdownHook, execute); + } + + /** + * Launch a service catching all exceptions and downgrading them to exit codes + * after logging. + * + * Sets {@link #serviceException} to this value. + * @param conf configuration to use + * @param instance optional instance of the service. + * @param processedArgs command line after the launcher-specific arguments + * have been stripped out. + * @param addShutdownHook should a shutdown hook be added to terminate + * this service on shutdown. Tests should set this to false. + * @param execute execute/wait for the service to stop. + * @return an exit exception, which will have a status code of 0 if it worked + */ + public ExitUtil.ExitException launchService(Configuration conf, + S instance, + List processedArgs, + boolean addShutdownHook, + boolean execute) { + ExitUtil.ExitException exitException; - + try { - int exitCode = coreServiceLaunch(conf, processedArgs, addShutdownHook, - execute); + int exitCode = coreServiceLaunch(conf, instance, processedArgs, + addShutdownHook, execute); if (service != null) { // check to see if the service failed Throwable failure = service.getFailureCause(); @@ -495,6 +526,12 @@ public ExitUtil.ExitException launchService(Configuration conf, // exit exceptions are passed through unchanged exitException = ee; } catch (Throwable thrown) { + // other errors need a full log. + LOG.error("Exception raised {}", + service != null + ? (service.toString() + " in state " + service.getServiceState()) + : "during service instantiation", + thrown); exitException = convertToExitException(thrown); } noteException(exitException); @@ -514,6 +551,7 @@ public ExitUtil.ExitException launchService(Configuration conf, * {@link #getService()}. * * @param conf configuration + * @param instance optional instance of the service. * @param processedArgs arguments after the configuration parameters * have been stripped out. * @param addShutdownHook should a shutdown hook be added to terminate @@ -530,12 +568,19 @@ public ExitUtil.ExitException launchService(Configuration conf, */ protected int coreServiceLaunch(Configuration conf, + S instance, List processedArgs, boolean addShutdownHook, boolean execute) throws Exception { // create the service instance - instantiateService(conf); + if (instance == null) { + instantiateService(conf); + } else { + // service already exists, so instantiate + configuration = conf; + service = instance; + } ServiceShutdownHook shutdownHook = null; // and the shutdown hook if requested @@ -685,8 +730,7 @@ protected static ExitUtil.ExitException convertToExitException( } // construct the new exception with the original message and // an exit code - exitException = new ServiceLaunchException(exitCode, message); - exitException.initCause(thrown); + exitException = new ServiceLaunchException(exitCode, thrown, message); return exitException; } @@ -917,7 +961,7 @@ protected List parseCommandArgs(Configuration conf, throw new ServiceLaunchException(EXIT_COMMAND_ARGUMENT_ERROR, e); } catch (RuntimeException e) { // lower level issue such as XML parse failure - throw new ServiceLaunchException(EXIT_COMMAND_ARGUMENT_ERROR, + throw new ServiceLaunchException(EXIT_COMMAND_ARGUMENT_ERROR, e, E_PARSE_FAILED + " %s : %s", argString, e); } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tools/TableListing.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tools/TableListing.java index 85015fbe305fc..348f86fe1368b 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tools/TableListing.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tools/TableListing.java @@ -234,7 +234,7 @@ public String toString() { Column column = columns[i]; if (column.wrap) { int maxWidth = column.getMaxWidth(); - if (maxWidth > 4) { + if (maxWidth > 10) { column.setWrapWidth(maxWidth-1); modified = true; width -= 1; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverInfo.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverInfo.java index 92aaa10ef8790..546af26b9589a 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverInfo.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverInfo.java @@ -17,7 +17,7 @@ */ package org.apache.hadoop.tracing; -import java.util.LinkedList; +import java.util.ArrayList; import java.util.List; import org.apache.hadoop.classification.InterfaceAudience; @@ -29,7 +29,7 @@ public class SpanReceiverInfo { private final long id; private final String className; final List configPairs = - new LinkedList(); + new ArrayList(); static class ConfigurationPair { private final String key; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java index 06ef8acc52344..32a0adca1979a 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java @@ -143,9 +143,12 @@ public static DataChecksum newDataChecksum(Type type, int bytesPerChecksum ) { * Creates a DataChecksum from HEADER_LEN bytes from arr[offset]. * @return DataChecksum of the type in the array or null in case of an error. */ - public static DataChecksum newDataChecksum( byte bytes[], int offset ) { + public static DataChecksum newDataChecksum(byte[] bytes, int offset) + throws IOException { if (offset < 0 || bytes.length < offset + getChecksumHeaderSize()) { - return null; + throw new InvalidChecksumSizeException("Could not create DataChecksum " + + " from the byte array of length " + bytes.length + + " and offset "+ offset); } // like readInt(): @@ -153,7 +156,14 @@ public static DataChecksum newDataChecksum( byte bytes[], int offset ) { ( (bytes[offset+2] & 0xff) << 16 ) | ( (bytes[offset+3] & 0xff) << 8 ) | ( (bytes[offset+4] & 0xff) ); - return newDataChecksum( Type.valueOf(bytes[offset]), bytesPerChecksum ); + DataChecksum csum = newDataChecksum(mapByteToChecksumType(bytes[offset]), + bytesPerChecksum); + if (csum == null) { + throw new InvalidChecksumSizeException(("Could not create DataChecksum " + + " from the byte array of length " + bytes.length + + " and bytesPerCheckSum of "+ bytesPerChecksum)); + } + return csum; } /** @@ -164,13 +174,23 @@ public static DataChecksum newDataChecksum( DataInputStream in ) throws IOException { int type = in.readByte(); int bpc = in.readInt(); - DataChecksum summer = newDataChecksum(Type.valueOf(type), bpc ); + DataChecksum summer = newDataChecksum(mapByteToChecksumType(type), bpc); if ( summer == null ) { throw new InvalidChecksumSizeException("Could not create DataChecksum " + "of type " + type + " with bytesPerChecksum " + bpc); } return summer; } + + private static Type mapByteToChecksumType(int type) + throws InvalidChecksumSizeException{ + try { + return Type.valueOf(type); + } catch (IllegalArgumentException e) { + throw new InvalidChecksumSizeException("The value "+type+" does not map"+ + " to a valid checksum Type"); + } + } /** * Writes the checksum header to the output stream out. diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DurationInfo.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DurationInfo.java index 9dd75db27c733..605d060270f8e 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DurationInfo.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DurationInfo.java @@ -20,8 +20,8 @@ import org.slf4j.Logger; -import org.apache.hadoop.classification.InterfaceAudience; -import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.classification.InterfaceAudience.Public; +import org.apache.hadoop.classification.InterfaceStability.Unstable; /** * A duration with logging of final state at info or debug @@ -29,8 +29,8 @@ * This allows it to be used in a try-with-resources clause, and have the * duration automatically logged. */ -@InterfaceAudience.Private -@InterfaceStability.Unstable +@Public +@Unstable public class DurationInfo extends OperationDuration implements AutoCloseable { private final String text; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java index 366c8c787f84c..12d1ef01201a2 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java @@ -154,18 +154,20 @@ public static void validateResponse(HttpURLConnection conn, toThrow = (Exception) constr.newInstance(exMsg); } catch (Exception ex) { toThrow = new IOException(String.format( - "HTTP status [%d], exception [%s], message [%s] ", - conn.getResponseCode(), exClass, exMsg)); + "HTTP status [%d], exception [%s], message [%s], URL [%s]", + conn.getResponseCode(), exClass, exMsg, conn.getURL())); } } else { String msg = (exMsg != null) ? exMsg : conn.getResponseMessage(); toThrow = new IOException(String.format( - "HTTP status [%d], message [%s]", conn.getResponseCode(), msg)); + "HTTP status [%d], message [%s], URL [%s]", + conn.getResponseCode(), msg, conn.getURL())); } } catch (Exception ex) { toThrow = new IOException(String.format( - "HTTP status [%d], message [%s]", conn.getResponseCode(), - conn.getResponseMessage())); + "HTTP status [%d], message [%s], URL [%s], exception [%s]", + conn.getResponseCode(), conn.getResponseMessage(), conn.getURL(), + ex.toString()), ex); } finally { if (es != null) { try { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java index 776839c17f257..23388248575ac 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java @@ -28,6 +28,7 @@ import org.apache.hadoop.io.compress.zlib.ZlibFactory; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.io.nativeio.NativeIO; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -69,6 +70,7 @@ public static void main(String[] args) { boolean snappyLoaded = false; boolean isalLoaded = false; boolean zStdLoaded = false; + boolean pmdkLoaded = false; // lz4 is linked within libhadoop boolean lz4Loaded = nativeHadoopLoaded; boolean bzip2Loaded = Bzip2Factory.isNativeBzip2Loaded(conf); @@ -80,6 +82,7 @@ public static void main(String[] args) { String zlibLibraryName = ""; String snappyLibraryName = ""; String isalDetail = ""; + String pmdkDetail = ""; String zstdLibraryName = ""; String lz4LibraryName = ""; String bzip2LibraryName = ""; @@ -110,6 +113,12 @@ public static void main(String[] args) { isalLoaded = true; } + pmdkDetail = NativeIO.POSIX.getPmdkSupportStateMessage(); + pmdkLoaded = NativeIO.POSIX.isPmdkAvailable(); + if (pmdkLoaded) { + pmdkDetail = NativeIO.POSIX.Pmem.getPmdkLibPath(); + } + openSslDetail = OpensslCipher.getLoadingFailureReason(); if (openSslDetail != null) { openSslLoaded = false; @@ -148,6 +157,7 @@ public static void main(String[] args) { System.out.printf("bzip2: %b %s%n", bzip2Loaded, bzip2LibraryName); System.out.printf("openssl: %b %s%n", openSslLoaded, openSslDetail); System.out.printf("ISA-L: %b %s%n", isalLoaded, isalDetail); + System.out.printf("PMDK: %b %s%n", pmdkLoaded, pmdkDetail); if (Shell.WINDOWS) { System.out.printf("winutils: %b %s%n", winutilsExists, winutilsPath); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java index 7c46c5b7ec421..f2a5b242a8d37 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java @@ -163,6 +163,7 @@ void reportHealthStatus(HealthCheckerExitStatus status) { setHealthStatus(false, exceptionStackTrace); break; case FAILED_WITH_EXIT_CODE: + // see Javadoc above - we don't report bad health intentionally setHealthStatus(true, "", now); break; case FAILED: diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java index 556f4e0656ec0..76d90063609b2 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java @@ -92,7 +92,7 @@ public void run() { return; } long started = System.currentTimeMillis(); - int timeoutCount = executeShutdown(); + int timeoutCount = MGR.executeShutdown(); long ended = System.currentTimeMillis(); LOG.debug(String.format( "Completed shutdown in %.3f seconds; Timeouts: %d", @@ -116,9 +116,9 @@ public void run() { */ @InterfaceAudience.Private @VisibleForTesting - static int executeShutdown() { + int executeShutdown() { int timeouts = 0; - for (HookEntry entry: MGR.getShutdownHooksInOrder()) { + for (HookEntry entry: getShutdownHooksInOrder()) { Future future = EXECUTOR.submit(entry.getHook()); try { future.get(entry.getTimeout(), entry.getTimeUnit()); @@ -254,7 +254,9 @@ TimeUnit getTimeUnit() { private AtomicBoolean shutdownInProgress = new AtomicBoolean(false); //private to constructor to ensure singularity - private ShutdownHookManager() { + @VisibleForTesting + @InterfaceAudience.Private + ShutdownHookManager() { } /** @@ -267,8 +269,8 @@ private ShutdownHookManager() { @VisibleForTesting List getShutdownHooksInOrder() { List list; - synchronized (MGR.hooks) { - list = new ArrayList<>(MGR.hooks); + synchronized (hooks) { + list = new ArrayList<>(hooks); } Collections.sort(list, new Comparator() { @@ -342,7 +344,9 @@ public boolean removeShutdownHook(Runnable shutdownHook) { throw new IllegalStateException("Shutdown in progress, cannot remove a " + "shutdownHook"); } - return hooks.remove(new HookEntry(shutdownHook, 0)); + // hooks are only == by runnable + return hooks.remove(new HookEntry(shutdownHook, 0, TIMEOUT_MINIMUM, + TIME_UNIT_DEFAULT)); } /** @@ -354,7 +358,8 @@ public boolean removeShutdownHook(Runnable shutdownHook) { @InterfaceAudience.Public @InterfaceStability.Stable public boolean hasShutdownHook(Runnable shutdownHook) { - return hooks.contains(new HookEntry(shutdownHook, 0)); + return hooks.contains(new HookEntry(shutdownHook, 0, TIMEOUT_MINIMUM, + TIME_UNIT_DEFAULT)); } /** diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java index d19ced93697c8..cf7b04ab61a7e 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java @@ -424,6 +424,32 @@ public static Collection getStringCollection(String str, String delim) { return values; } + /** + * Returns a collection of strings, trimming leading and trailing whitespace + * on each value. Duplicates are not removed. + * + * @param str + * String separated by delim. + * @param delim + * Delimiter to separate the values in str. + * @return Collection of string values. + */ + public static Collection getTrimmedStringCollection(String str, + String delim) { + List values = new ArrayList(); + if (str == null) + return values; + StringTokenizer tokenizer = new StringTokenizer(str, delim); + while (tokenizer.hasMoreTokens()) { + String next = tokenizer.nextToken(); + if (next == null || next.trim().isEmpty()) { + continue; + } + values.add(next.trim()); + } + return values; + } + /** * Splits a comma separated value String, trimming leading and * trailing whitespace on each value. Duplicate and empty values are removed. diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java index d164138a39fa4..36dade27dd6f7 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java @@ -20,13 +20,13 @@ import java.io.IOException; import java.nio.charset.Charset; import java.util.ArrayList; +import java.util.LinkedList; import java.util.List; import org.apache.curator.framework.AuthInfo; import org.apache.curator.framework.CuratorFramework; import org.apache.curator.framework.CuratorFrameworkFactory; -import org.apache.curator.framework.api.transaction.CuratorTransaction; -import org.apache.curator.framework.api.transaction.CuratorTransactionFinal; +import org.apache.curator.framework.api.transaction.CuratorOp; import org.apache.curator.retry.RetryNTimes; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.conf.Configuration; @@ -387,43 +387,45 @@ public SafeTransaction createTransaction(List fencingACL, /** * Use curator transactions to ensure zk-operations are performed in an all * or nothing fashion. This is equivalent to using ZooKeeper#multi. - * - * TODO (YARN-3774): Curator 3.0 introduces CuratorOp similar to Op. We ll - * have to rewrite this inner class when we adopt that. */ public class SafeTransaction { - private CuratorTransactionFinal transactionFinal; private String fencingNodePath; + private List curatorOperations = new LinkedList<>(); SafeTransaction(List fencingACL, String fencingNodePath) throws Exception { this.fencingNodePath = fencingNodePath; - CuratorTransaction transaction = curator.inTransaction(); - transactionFinal = transaction.create() - .withMode(CreateMode.PERSISTENT).withACL(fencingACL) - .forPath(fencingNodePath, new byte[0]).and(); + curatorOperations.add(curator.transactionOp().create() + .withMode(CreateMode.PERSISTENT) + .withACL(fencingACL) + .forPath(fencingNodePath, new byte[0])); } public void commit() throws Exception { - transactionFinal = transactionFinal.delete() - .forPath(fencingNodePath).and(); - transactionFinal.commit(); + curatorOperations.add(curator.transactionOp().delete() + .forPath(fencingNodePath)); + curator.transaction().forOperations(curatorOperations); + curatorOperations.clear(); } public void create(String path, byte[] data, List acl, CreateMode mode) throws Exception { - transactionFinal = transactionFinal.create() - .withMode(mode).withACL(acl).forPath(path, data).and(); + curatorOperations.add(curator.transactionOp().create() + .withMode(mode) + .withACL(acl) + .forPath(path, data)); } public void delete(String path) throws Exception { - transactionFinal = transactionFinal.delete().forPath(path).and(); + curatorOperations.add(curator.transactionOp().delete() + .forPath(path)); } public void setData(String path, byte[] data, int version) throws Exception { - transactionFinal = transactionFinal.setData() - .withVersion(version).forPath(path, data).and(); + curatorOperations.add(curator.transactionOp().setData() + .withVersion(version) + .forPath(path, data)); } } } \ No newline at end of file diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zstd/ZStandardCompressor.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zstd/ZStandardCompressor.c index 289554b4cf140..41eb9e2c85a10 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zstd/ZStandardCompressor.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zstd/ZStandardCompressor.c @@ -78,7 +78,7 @@ static __dlsym_ZSTD_isError dlsym_ZSTD_isError; static __dlsym_ZSTD_getErrorName dlsym_ZSTD_getErrorName; #endif -// Load the libztsd.so from disk +// Load the libzstd.so from disk JNIEXPORT void JNICALL Java_org_apache_hadoop_io_compress_zstd_ZStandardCompressor_initIDs (JNIEnv *env, jclass clazz) { #ifdef UNIX // Load libzstd.so diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c index 2274d57ca9dd7..b0b5151cd1238 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c @@ -36,6 +36,10 @@ #include #include #include +#ifdef HADOOP_PMDK_LIBRARY +#include +#include "pmdk_load.h" +#endif #if !(defined(__FreeBSD__) || defined(__MACH__)) #include #endif @@ -60,6 +64,7 @@ #define NATIVE_IO_POSIX_CLASS "org/apache/hadoop/io/nativeio/NativeIO$POSIX" #define NATIVE_IO_STAT_CLASS "org/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat" +#define NATIVE_IO_POSIX_PMEMREGION_CLASS "org/apache/hadoop/io/nativeio/NativeIO$POSIX$PmemMappedRegion" #define SET_INT_OR_RETURN(E, C, F) \ { \ @@ -81,6 +86,12 @@ static jmethodID nioe_ctor; // Please see HADOOP-7156 for details. jobject pw_lock_object; +#ifdef HADOOP_PMDK_LIBRARY +// the NativeIO$POSIX$PmemMappedRegion inner class and its constructor +static jclass pmem_region_clazz = NULL; +static jmethodID pmem_region_ctor = NULL; +#endif + /* * Throw a java.IO.IOException, generating the message from errno. * NB. this is also used form windows_secure_container_executor.c @@ -269,6 +280,66 @@ static void nioe_deinit(JNIEnv *env) { nioe_ctor = NULL; } +#ifdef HADOOP_PMDK_LIBRARY +static int loadPmdkLib(JNIEnv *env) { + char errMsg[1024]; + jclass clazz = (*env)->FindClass(env, NATIVE_IO_POSIX_CLASS); + if (clazz == NULL) { + return 0; // exception has been raised + } + load_pmdk_lib(errMsg, sizeof(errMsg)); + jmethodID mid = (*env)->GetStaticMethodID(env, clazz, "setPmdkSupportState", "(I)V"); + if (mid == 0) { + return 0; + } + + if (strlen(errMsg) > 0) { + // Set PMDK support state to 1 which represents PMDK_LIB_NOT_FOUND. + (*env)->CallStaticVoidMethod(env, clazz, mid, 1); + return 0; + } + // Set PMDK support state to 0 which represents SUPPORTED. + (*env)->CallStaticVoidMethod(env, clazz, mid, 0); + return 1; +} + +static void pmem_region_init(JNIEnv *env, jclass nativeio_class) { + + jclass clazz = NULL; + // Init Stat + clazz = (*env)->FindClass(env, NATIVE_IO_POSIX_PMEMREGION_CLASS); + if (!clazz) { + THROW(env, "java/io/IOException", "Failed to get PmemMappedRegion class"); + return; // exception has been raised + } + + // Init PmemMappedRegion class + pmem_region_clazz = (*env)->NewGlobalRef(env, clazz); + if (!pmem_region_clazz) { + THROW(env, "java/io/IOException", "Failed to new global reference of PmemMappedRegion class"); + return; // exception has been raised + } + + pmem_region_ctor = (*env)->GetMethodID(env, pmem_region_clazz, "", "(JJZ)V"); + if (!pmem_region_ctor) { + THROW(env, "java/io/IOException", "Failed to get PmemMappedRegion constructor"); + return; // exception has been raised + } +} + +static void pmem_region_deinit(JNIEnv *env) { + if (pmem_region_ctor != NULL) { + (*env)->DeleteGlobalRef(env, pmem_region_ctor); + pmem_region_ctor = NULL; + } + + if (pmem_region_clazz != NULL) { + (*env)->DeleteGlobalRef(env, pmem_region_clazz); + pmem_region_clazz = NULL; + } + } +#endif + /* * private static native void initNative(); * @@ -292,6 +363,11 @@ Java_org_apache_hadoop_io_nativeio_NativeIO_initNative( #ifdef UNIX errno_enum_init(env); PASS_EXCEPTIONS_GOTO(env, error); +#ifdef HADOOP_PMDK_LIBRARY + if (loadPmdkLib(env)) { + pmem_region_init(env, clazz); + } +#endif #endif return; error: @@ -299,6 +375,9 @@ Java_org_apache_hadoop_io_nativeio_NativeIO_initNative( // class wasn't initted yet #ifdef UNIX stat_deinit(env); +#ifdef HADOOP_PMDK_LIBRARY + pmem_region_deinit(env); +#endif #endif nioe_deinit(env); fd_deinit(env); @@ -1383,3 +1462,188 @@ JNIEnv *env, jclass clazz, jstring jsrc, jstring jdst) /** * vim: sw=2: ts=2: et: */ + +#ifdef __cplusplus +extern "C" { +#endif + +/* + * Class: org_apache_hadoop_io_nativeio_NativeIO_POSIX + * Method: isPmemCheck + * Signature: (JJ)Z + */ +JNIEXPORT jboolean JNICALL Java_org_apache_hadoop_io_nativeio_NativeIO_00024POSIX_isPmemCheck( +JNIEnv *env, jclass thisClass, jlong address, jlong length) { + #if (defined UNIX) && (defined HADOOP_PMDK_LIBRARY) + jint is_pmem = pmdkLoader->pmem_is_pmem(address, length); + return (is_pmem) ? JNI_TRUE : JNI_FALSE; + #else + THROW(env, "java/lang/UnsupportedOperationException", + "The function isPmemCheck is not supported."); + return JNI_FALSE; + #endif + } + +/* + * Class: org_apache_hadoop_io_nativeio_NativeIO_POSIX + * Method: pmemCreateMapFile + * Signature: (Ljava/lang/String;J)Lorg/apache/hadoop/io/nativeio/NativeIO/POSIX/PmemMappedRegion; + */ +JNIEXPORT jobject JNICALL Java_org_apache_hadoop_io_nativeio_NativeIO_00024POSIX_pmemCreateMapFile( +JNIEnv *env, jclass thisClass, jstring filePath, jlong fileLength) { + #if (defined UNIX) && (defined HADOOP_PMDK_LIBRARY) + /* create a pmem file and memory map it */ + const char * path = NULL; + void * pmemaddr = NULL; + size_t mapped_len = 0; + int is_pmem = 1; + char msg[1000]; + + path = (*env)->GetStringUTFChars(env, filePath, NULL); + if (!path) { + THROW(env, "java/lang/IllegalArgumentException", "File Path cannot be null"); + return NULL; + } + + if (fileLength <= 0) { + (*env)->ReleaseStringUTFChars(env, filePath, path); + THROW(env, "java/lang/IllegalArgumentException", "File length should be positive"); + return NULL; + } + + pmemaddr = pmdkLoader->pmem_map_file(path, fileLength, PMEM_FILE_CREATE|PMEM_FILE_EXCL, + 0666, &mapped_len, &is_pmem); + + if (!pmemaddr) { + snprintf(msg, sizeof(msg), "Failed to create pmem file. file: %s, length: %x, error msg: %s", path, fileLength, pmem_errormsg()); + THROW(env, "java/io/IOException", msg); + (*env)->ReleaseStringUTFChars(env, filePath, path); + return NULL; + } + + if (fileLength != mapped_len) { + snprintf(msg, sizeof(msg), "Mapped length doesn't match the request length. file :%s, request length:%x, returned length:%x, error msg:%s", path, fileLength, mapped_len, pmem_errormsg()); + THROW(env, "java/io/IOException", msg); + (*env)->ReleaseStringUTFChars(env, filePath, path); + return NULL; + } + + (*env)->ReleaseStringUTFChars(env, filePath, path); + + if ((!pmem_region_clazz) || (!pmem_region_ctor)) { + THROW(env, "java/io/IOException", "PmemMappedRegion class or constructor is NULL"); + return NULL; + } + + jobject ret = (*env)->NewObject(env, pmem_region_clazz, pmem_region_ctor, pmemaddr, mapped_len, (jboolean)is_pmem); + return ret; + + #else + THROW(env, "java/lang/UnsupportedOperationException", + "The function pmemCreateMapFile is not supported."); + return NULL; + #endif + } + +/* + * Class: org_apache_hadoop_io_nativeio_NativeIO_POSIX + * Method: pmemUnMap + * Signature: (JJ)V + */ +JNIEXPORT jboolean JNICALL Java_org_apache_hadoop_io_nativeio_NativeIO_00024POSIX_pmemUnMap( +JNIEnv *env, jclass thisClass, jlong address, jlong length) { + #if (defined UNIX) && (defined HADOOP_PMDK_LIBRARY) + int succeed = 0; + char msg[1000]; + succeed = pmdkLoader->pmem_unmap(address, length); + // succeed = -1 failure; succeed = 0 success + if (succeed != 0) { + snprintf(msg, sizeof(msg), "Failed to unmap region. address: %x, length: %x, error msg: %s", address, length, pmem_errormsg()); + THROW(env, "java/io/IOException", msg); + return JNI_FALSE; + } else { + return JNI_TRUE; + } + #else + THROW(env, "java/lang/UnsupportedOperationException", + "The function pmemUnMap is not supported."); + return JNI_FALSE; + #endif + } + +/* + * Class: org_apache_hadoop_io_nativeio_NativeIO_POSIX + * Method: pmemCopy + * Signature: ([BJJ)V + */ +JNIEXPORT void JNICALL Java_org_apache_hadoop_io_nativeio_NativeIO_00024POSIX_pmemCopy( +JNIEnv *env, jclass thisClass, jbyteArray buf, jlong address, jboolean is_pmem, jlong length) { + #if (defined UNIX) && (defined HADOOP_PMDK_LIBRARY) + char msg[1000]; + jbyte* srcBuf = (*env)->GetByteArrayElements(env, buf, 0); + snprintf(msg, sizeof(msg), "Pmem copy content. dest: %x, length: %x, src: %x ", address, length, srcBuf); + if (is_pmem) { + pmdkLoader->pmem_memcpy_nodrain(address, srcBuf, length); + } else { + memcpy(address, srcBuf, length); + } + (*env)->ReleaseByteArrayElements(env, buf, srcBuf, 0); + return; + #else + THROW(env, "java/lang/UnsupportedOperationException", + "The function pmemCopy is not supported."); + #endif + } + +/* + * Class: org_apache_hadoop_io_nativeio_NativeIO + * Method: pmemDrain + * Signature: ()V + */ +JNIEXPORT void JNICALL Java_org_apache_hadoop_io_nativeio_NativeIO_00024POSIX_pmemDrain( +JNIEnv *env, jclass thisClass) { + #if (defined UNIX) && (defined HADOOP_PMDK_LIBRARY) + pmdkLoader->pmem_drain(); + #else + THROW(env, "java/lang/UnsupportedOperationException", + "The function pmemDrain is not supported."); + #endif + } + + /* + * Class: org_apache_hadoop_io_nativeio_NativeIO_POSIX + * Method: pmemSync + * Signature: (JJ)V + */ +JNIEXPORT void JNICALL Java_org_apache_hadoop_io_nativeio_NativeIO_00024POSIX_pmemSync + (JNIEnv * env, jclass thisClass, jlong address, jlong length) { + + #if (defined UNIX) && (defined HADOOP_PMDK_LIBRARY) + int succeed = 0; + char msg[1000]; + succeed = pmdkLoader->pmem_msync(address, length); + // succeed = -1 failure + if (succeed == -1) { + snprintf(msg, sizeof(msg), "Failed to msync region. address: %x, length: %x, error msg: %s", address, length, pmem_errormsg()); + THROW(env, "java/io/IOException", msg); + return; + } + #else + THROW(env, "java/lang/UnsupportedOperationException", + "The function pmemSync is not supported."); + #endif + } + +JNIEXPORT jstring JNICALL Java_org_apache_hadoop_io_nativeio_NativeIO_00024POSIX_getPmdkLibPath + (JNIEnv * env, jclass thisClass) { + jstring libpath = NULL; + + #ifdef HADOOP_PMDK_LIBRARY + libpath = (*env)->NewStringUTF(env, HADOOP_PMDK_LIBRARY); + #endif + return libpath; + } + +#ifdef __cplusplus +} +#endif diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.c new file mode 100644 index 0000000000000..502508cbf3b86 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.c @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include + +#include "org_apache_hadoop.h" +#include "pmdk_load.h" +#include "org_apache_hadoop_io_nativeio_NativeIO.h" +#include "org_apache_hadoop_io_nativeio_NativeIO_POSIX.h" + +#ifdef UNIX +#include +#include +#include +#include + +#include "config.h" +#endif + +PmdkLibLoader * pmdkLoader; + +/** + * pmdk_load.c + * Utility of loading the libpmem library and the required functions. + * Building of this codes won't rely on any libpmem source codes, but running + * into this will rely on successfully loading of the dynamic library. + * + */ + +static const char* load_functions() { +#ifdef UNIX + PMDK_LOAD_DYNAMIC_SYMBOL((pmdkLoader->pmem_map_file), "pmem_map_file"); + PMDK_LOAD_DYNAMIC_SYMBOL((pmdkLoader->pmem_unmap), "pmem_unmap"); + PMDK_LOAD_DYNAMIC_SYMBOL((pmdkLoader->pmem_is_pmem), "pmem_is_pmem"); + PMDK_LOAD_DYNAMIC_SYMBOL((pmdkLoader->pmem_drain), "pmem_drain"); + PMDK_LOAD_DYNAMIC_SYMBOL((pmdkLoader->pmem_memcpy_nodrain), "pmem_memcpy_nodrain"); + PMDK_LOAD_DYNAMIC_SYMBOL((pmdkLoader->pmem_msync), "pmem_msync"); +#endif + return NULL; +} + +void load_pmdk_lib(char* err, size_t err_len) { + const char* errMsg; + const char* library = NULL; + #ifdef UNIX + Dl_info dl_info; + #else + LPTSTR filename = NULL; + #endif + + err[0] = '\0'; + + if (pmdkLoader != NULL) { + return; + } + pmdkLoader = calloc(1, sizeof(PmdkLibLoader)); + + // Load PMDK library + #ifdef UNIX + pmdkLoader->libec = dlopen(HADOOP_PMDK_LIBRARY, RTLD_LAZY | RTLD_GLOBAL); + if (pmdkLoader->libec == NULL) { + snprintf(err, err_len, "Failed to load %s (%s)", + HADOOP_PMDK_LIBRARY, dlerror()); + return; + } + // Clear any existing error + dlerror(); + #endif + errMsg = load_functions(pmdkLoader->libec); + if (errMsg != NULL) { + snprintf(err, err_len, "Loading functions from PMDK failed: %s", errMsg); + } + + #ifdef UNIX + if (dladdr(pmdkLoader->pmem_map_file, &dl_info)) { + library = dl_info.dli_fname; + } + #else + if (GetModuleFileName(pmdkLoader->libec, filename, 256) > 0) { + library = filename; + } + #endif + + if (library == NULL) { + library = HADOOP_PMDK_LIBRARY; + } + + pmdkLoader->libname = strdup(library); +} diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.h b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.h new file mode 100644 index 0000000000000..a668377557cf4 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.h @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include + +#include "org_apache_hadoop.h" + +#ifdef UNIX +#include +#include +#include +#include +#endif + +#ifndef _PMDK_LOAD_H_ +#define _PMDK_LOAD_H_ + + +#ifdef UNIX +// For libpmem.h +typedef void * (*__d_pmem_map_file)(const char *path, size_t len, int flags, mode_t mode, + size_t *mapped_lenp, int *is_pmemp); +typedef int (* __d_pmem_unmap)(void *addr, size_t len); +typedef int (*__d_pmem_is_pmem)(const void *addr, size_t len); +typedef void (*__d_pmem_drain)(void); +typedef void * (*__d_pmem_memcpy_nodrain)(void *pmemdest, const void *src, size_t len); +typedef int (* __d_pmem_msync)(const void *addr, size_t len); + +#endif + +typedef struct __PmdkLibLoader { + // The loaded library handle + void* libec; + char* libname; + __d_pmem_map_file pmem_map_file; + __d_pmem_unmap pmem_unmap; + __d_pmem_is_pmem pmem_is_pmem; + __d_pmem_drain pmem_drain; + __d_pmem_memcpy_nodrain pmem_memcpy_nodrain; + __d_pmem_msync pmem_msync; +} PmdkLibLoader; + +extern PmdkLibLoader * pmdkLoader; + +/** + * A helper function to dlsym a 'symbol' from a given library-handle. + */ + +#ifdef UNIX + +static __attribute__ ((unused)) +void *myDlsym(void *handle, const char *symbol) { + void *func_ptr = dlsym(handle, symbol); + return func_ptr; +} + +/* A helper macro to dlsym the requisite dynamic symbol in NON-JNI env. */ +#define PMDK_LOAD_DYNAMIC_SYMBOL(func_ptr, symbol) \ + if ((func_ptr = myDlsym(pmdkLoader->libec, symbol)) == NULL) { \ + return "Failed to load symbol" symbol; \ + } + +#endif + +/** + * Initialize and load PMDK library, returning error message if any. + * + * @param err The err message buffer. + * @param err_len The length of the message buffer. + */ +void load_pmdk_lib(char* err, size_t err_len); + +#endif //_PMDK_LOAD_H_ \ No newline at end of file diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c index 402ffd5bb20a6..b463679fcdb6f 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c @@ -199,8 +199,5 @@ Java_org_apache_hadoop_security_JniBasedUnixGroupsMapping_getGroupsForUser if (ginfo) { hadoop_group_info_free(ginfo); } - if (jgroupname) { - (*env)->DeleteLocalRef(env, jgroupname); - } return jgroups; } diff --git a/hadoop-common-project/hadoop-common/src/main/proto/FSProtos.proto b/hadoop-common-project/hadoop-common/src/main/proto/FSProtos.proto index c3b768ab67ed1..c895bce757b77 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/FSProtos.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/FSProtos.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.fs"; option java_outer_classname = "FSProtos"; option java_generic_services = true; diff --git a/hadoop-common-project/hadoop-common/src/main/proto/GenericRefreshProtocol.proto b/hadoop-common-project/hadoop-common/src/main/proto/GenericRefreshProtocol.proto index fe465490b199c..6296f88da69b8 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/GenericRefreshProtocol.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/GenericRefreshProtocol.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.ipc.proto"; option java_outer_classname = "GenericRefreshProtocolProtos"; option java_generic_services = true; diff --git a/hadoop-common-project/hadoop-common/src/main/proto/GetUserMappingsProtocol.proto b/hadoop-common-project/hadoop-common/src/main/proto/GetUserMappingsProtocol.proto index 51552b879f3c4..cb91a13b04875 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/GetUserMappingsProtocol.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/GetUserMappingsProtocol.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.tools.proto"; option java_outer_classname = "GetUserMappingsProtocolProtos"; option java_generic_services = true; diff --git a/hadoop-common-project/hadoop-common/src/main/proto/HAServiceProtocol.proto b/hadoop-common-project/hadoop-common/src/main/proto/HAServiceProtocol.proto index 16ee9a2e0a5c6..5a88a7ff03f02 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/HAServiceProtocol.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/HAServiceProtocol.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.ha.proto"; option java_outer_classname = "HAServiceProtocolProtos"; option java_generic_services = true; diff --git a/hadoop-common-project/hadoop-common/src/main/proto/IpcConnectionContext.proto b/hadoop-common-project/hadoop-common/src/main/proto/IpcConnectionContext.proto index 4557e893cff82..16e2fb7c4db75 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/IpcConnectionContext.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/IpcConnectionContext.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.ipc.protobuf"; option java_outer_classname = "IpcConnectionContextProtos"; option java_generate_equals_and_hash = true; diff --git a/hadoop-common-project/hadoop-common/src/main/proto/ProtobufRpcEngine.proto b/hadoop-common-project/hadoop-common/src/main/proto/ProtobufRpcEngine.proto index a17e2078e947f..fa11313402758 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/ProtobufRpcEngine.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/ProtobufRpcEngine.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; /** * These are the messages used by Hadoop RPC for the Rpc Engine Protocol Buffer * to marshal the request and response in the RPC layer. diff --git a/hadoop-common-project/hadoop-common/src/main/proto/ProtocolInfo.proto b/hadoop-common-project/hadoop-common/src/main/proto/ProtocolInfo.proto index fdbc440d91c54..0e9d0d4baa413 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/ProtocolInfo.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/ProtocolInfo.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.ipc.protobuf"; option java_outer_classname = "ProtocolInfoProtos"; option java_generic_services = true; diff --git a/hadoop-common-project/hadoop-common/src/main/proto/RefreshAuthorizationPolicyProtocol.proto b/hadoop-common-project/hadoop-common/src/main/proto/RefreshAuthorizationPolicyProtocol.proto index 5ef1c2d0a8c97..f57c6d6303916 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/RefreshAuthorizationPolicyProtocol.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/RefreshAuthorizationPolicyProtocol.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.security.proto"; option java_outer_classname = "RefreshAuthorizationPolicyProtocolProtos"; option java_generic_services = true; diff --git a/hadoop-common-project/hadoop-common/src/main/proto/RefreshCallQueueProtocol.proto b/hadoop-common-project/hadoop-common/src/main/proto/RefreshCallQueueProtocol.proto index 67ed133251062..463b7c548fe22 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/RefreshCallQueueProtocol.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/RefreshCallQueueProtocol.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.ipc.proto"; option java_outer_classname = "RefreshCallQueueProtocolProtos"; option java_generic_services = true; diff --git a/hadoop-common-project/hadoop-common/src/main/proto/RefreshUserMappingsProtocol.proto b/hadoop-common-project/hadoop-common/src/main/proto/RefreshUserMappingsProtocol.proto index 41031ed9ea039..a1130f5c2d96d 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/RefreshUserMappingsProtocol.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/RefreshUserMappingsProtocol.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.security.proto"; option java_outer_classname = "RefreshUserMappingsProtocolProtos"; option java_generic_services = true; diff --git a/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto b/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto index e8d8cbbfe7032..4705b4276b876 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.ipc.protobuf"; option java_outer_classname = "RpcHeaderProtos"; option java_generate_equals_and_hash = true; diff --git a/hadoop-common-project/hadoop-common/src/main/proto/Security.proto b/hadoop-common-project/hadoop-common/src/main/proto/Security.proto index 4cf452023e53b..5177a86ef113e 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/Security.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/Security.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.security.proto"; option java_outer_classname = "SecurityProtos"; option java_generic_services = true; @@ -36,7 +36,6 @@ message TokenProto { required bytes password = 2; required string kind = 3; required string service = 4; - optional bytes handshakeSecret = 5; } message CredentialsKVProto { diff --git a/hadoop-common-project/hadoop-common/src/main/proto/TraceAdmin.proto b/hadoop-common-project/hadoop-common/src/main/proto/TraceAdmin.proto index 52d2a90abf496..8cf131bfb460a 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/TraceAdmin.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/TraceAdmin.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.tracing"; option java_outer_classname = "TraceAdminPB"; option java_generic_services = true; diff --git a/hadoop-common-project/hadoop-common/src/main/proto/ZKFCProtocol.proto b/hadoop-common-project/hadoop-common/src/main/proto/ZKFCProtocol.proto index a2b8dd10b302b..98bc05f4a360e 100644 --- a/hadoop-common-project/hadoop-common/src/main/proto/ZKFCProtocol.proto +++ b/hadoop-common-project/hadoop-common/src/main/proto/ZKFCProtocol.proto @@ -21,7 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *stable* .proto interface. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.ha.proto"; option java_outer_classname = "ZKFCProtocolProtos"; option java_generic_services = true; diff --git a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml index 6edcb670a443e..f0aa44dc999ca 100644 --- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml +++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml @@ -1092,8 +1092,8 @@ configuration of AWS access key ID and secret access key in environment variables named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, as documented in the AWS SDK. - * com.amazonaws.auth.InstanceProfileCredentialsProvider: supports use - of instance profile credentials if running in an EC2 VM. + * org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider: picks up + IAM credentials of any EC2 VM or AWS container in which the process is running. @@ -1213,8 +1213,12 @@ fs.s3a.connection.maximum - 15 - Controls the maximum number of simultaneous connections to S3. + 48 + Controls the maximum number of simultaneous connections to S3. + This must be bigger than the value of fs.s3a.threads.max so as to stop + threads being blocked waiting for new HTTPS connections. + Why not equal? The AWS SDK transfer manager also uses these connections. + @@ -1312,7 +1316,7 @@ fs.s3a.threads.max - 10 + 64 The total number of threads available in the filesystem for data uploads *or any other queued filesystem operation*. @@ -1326,13 +1330,30 @@ fs.s3a.max.total.tasks - 5 - The number of operations which can be queued for execution + 32 + The number of operations which can be queued for execution. + This is in addition to the number of active threads in fs.s3a.threads.max. + + + + + fs.s3a.executor.capacity + 16 + The maximum number of submitted tasks which is a single + operation (e.g. rename(), delete()) may submit simultaneously for + execution -excluding the IO-heavy block uploads, whose capacity + is set in "fs.s3a.fast.upload.active.blocks" + + All tasks are submitted to the shared thread pool whose size is + set in "fs.s3a.threads.max"; the value of capacity should be less than that + of the thread pool itself, as the goal is to stop a single operation + from overloading that thread pool. + fs.s3a.multipart.size - 100M + 64M How big (in bytes) to split upload or copy operations up into. A suffix from the set {K,M,G,T,P} may be used to scale the numeric value. @@ -1340,7 +1361,7 @@ fs.s3a.multipart.threshold - 2147483647 + 128M How big (in bytes) to split upload or copy operations up into. This also controls the partition size in renamed files, as rename() involves copying the source file(s). @@ -1502,12 +1523,10 @@ - fs.s3a.metadatastore.authoritative.dir.ttl - 3600000 + fs.s3a.metadatastore.metadata.ttl + 15m - This value sets how long a directory listing in the MS is considered as - authoritative. The value is in milliseconds. - MetadataStore should be authoritative to use this configuration knob. + This value sets how long an entry in a MetadataStore is valid. @@ -1581,23 +1600,27 @@ fs.s3a.s3guard.ddb.table.capacity.read - 500 + 0 Provisioned throughput requirements for read operations in terms of capacity - units for the DynamoDB table. This config value will only be used when - creating a new DynamoDB table, though later you can manually provision by - increasing or decreasing read capacity as needed for existing tables. - See DynamoDB documents for more information. + units for the DynamoDB table. This config value will only be used when + creating a new DynamoDB table. + If set to 0 (the default), new tables are created with "per-request" capacity. + If a positive integer is provided for this and the write capacity, then + a table with "provisioned capacity" will be created. + You can change the capacity of an existing provisioned-capacity table + through the "s3guard set-capacity" command. fs.s3a.s3guard.ddb.table.capacity.write - 100 + 0 Provisioned throughput requirements for write operations in terms of - capacity units for the DynamoDB table. Refer to related config - fs.s3a.s3guard.ddb.table.capacity.read before usage. + capacity units for the DynamoDB table. + If set to 0 (the default), new tables are created with "per-request" capacity. + Refer to related configuration option fs.s3a.s3guard.ddb.table.capacity.read @@ -1637,10 +1660,10 @@ fs.s3a.retry.limit - ${fs.s3a.attempts.maximum} + 7 Number of times to retry any repeatable S3 client request on failure, - excluding throttling requests. + excluding throttling requests and S3Guard inconsistency resolution. @@ -1648,8 +1671,8 @@ fs.s3a.retry.interval 500ms - Interval between attempts to retry operations for any reason other - than S3 throttle errors. + Initial retry interval when retrying operations for any reason other + than S3 throttle errors and S3Guard inconsistency resolution. @@ -1669,6 +1692,27 @@ + + fs.s3a.s3guard.consistency.retry.limit + 7 + + Number of times to retry attempts to read/open/copy files when + S3Guard believes a specific version of the file to be available, + but the S3 request does not find any version of a file, or a different + version. + + + + + fs.s3a.s3guard.consistency.retry.interval + 2s + + Initial interval between attempts to retry operations while waiting for S3 + to become consistent with the S3Guard data. + An exponential back-off is used here: every failure doubles the delay. + + + fs.s3a.committer.name file @@ -1724,7 +1768,7 @@ fs.s3a.committer.staging.conflict-mode - fail + append Staging committer conflict resolution policy. Supported: "fail", "append", "replace". @@ -1929,6 +1973,20 @@ + + fs.s3a.ssl.channel.mode + default_jsse + + If secure connections to S3 are enabled, configures the SSL + implementation used to encrypt connections to S3. Supported values are: + "default_jsse" and "default_jsse_with_gcm". "default_jsse" uses the Java + Secure Socket Extension package (JSSE). However, when running on Java 8, + the GCM cipher is removed from the list of enabled ciphers. This is due + to performance issues with GCM in Java 8. "default_jsse_with_gcm" uses + the JSSE with the default list of cipher suites. + + + fs.AbstractFileSystem.wasb.impl @@ -2173,7 +2231,7 @@ ipc.server.listen.queue.size - 128 + 256 Indicates the length of the listen queue for servers accepting client connections. @@ -2190,7 +2248,7 @@ ipc.maximum.data.length - 67108864 + 134217728 This indicates the maximum IPC message length (bytes) that can be accepted by the server. Messages larger than this value are rejected by the immediately to avoid possible OOMs. This setting should rarely need to be @@ -2216,6 +2274,193 @@ because the server side is stuck in TIME_WAIT state. + + + + + + + + + + + + + ipc.[port_number].backoff.enable + false + Whether or not to enable client backoff when a queue is full. + + + + + ipc.[port_number].callqueue.impl + java.util.concurrent.LinkedBlockingQueue + The fully qualified name of a class to use as the implementation + of a call queue. The default implementation is + java.util.concurrent.LinkedBlockingQueue (FIFO queue). + Use org.apache.hadoop.ipc.FairCallQueue for the Fair Call Queue. + + + + + ipc.[port_number].scheduler.impl + org.apache.hadoop.ipc.DefaultRpcScheduler + The fully qualified name of a class to use as the + implementation of the scheduler. The default implementation is + org.apache.hadoop.ipc.DefaultRpcScheduler (no-op scheduler) when not using + FairCallQueue. If using FairCallQueue, defaults to + org.apache.hadoop.ipc.DecayRpcScheduler. Use + org.apache.hadoop.ipc.DecayRpcScheduler in conjunction with the Fair Call + Queue. + + + + + ipc.[port_number].scheduler.priority.levels + 4 + How many priority levels to use within the scheduler and call + queue. This property applies to RpcScheduler and CallQueue. + + + + + ipc.[port_number].faircallqueue.multiplexer.weights + 8,4,2,1 + How much weight to give to each priority queue. This should be + a comma-separated list of length equal to the number of priority levels. + Weights descend by a factor of 2 (e.g., for 4 levels: 8,4,2,1). + This property applies to WeightedRoundRobinMultiplexer. + + + + + ipc.[port_number].identity-provider.impl + org.apache.hadoop.ipc.UserIdentityProvider + The identity provider mapping user requests to their identity. + This property applies to DecayRpcScheduler. + + + + + ipc.[port_number].cost-provider.impl + org.apache.hadoop.ipc.DefaultCostProvider + The cost provider mapping user requests to their cost. To + enable determination of cost based on processing time, use + org.apache.hadoop.ipc.WeightedTimeCostProvider. + This property applies to DecayRpcScheduler. + + + + + ipc.[port_number].decay-scheduler.period-ms + 5000 + How frequently the decay factor should be applied to the + operation counts of users. Higher values have less overhead, but respond + less quickly to changes in client behavior. + This property applies to DecayRpcScheduler. + + + + + ipc.[port_number].decay-scheduler.decay-factor + 0.5 + When decaying the operation counts of users, the multiplicative + decay factor to apply. Higher values will weight older operations more + strongly, essentially giving the scheduler a longer memory, and penalizing + heavy clients for a longer period of time. + This property applies to DecayRpcScheduler. + + + + + ipc.[port_number].decay-scheduler.thresholds + 13,25,50 + The client load threshold, as an integer percentage, for each + priority queue. Clients producing less load, as a percent of total + operations, than specified at position i will be given priority i. This + should be a comma-separated list of length equal to the number of priority + levels minus 1 (the last is implicitly 100). + Thresholds ascend by a factor of 2 (e.g., for 4 levels: 13,25,50). + This property applies to DecayRpcScheduler. + + + + + ipc.[port_number].decay-scheduler.backoff.responsetime.enable + false + Whether or not to enable the backoff by response time feature. + This property applies to DecayRpcScheduler. + + + + + ipc.[port_number].decay-scheduler.backoff.responsetime.thresholds + 10s,20s,30s,40s + The response time thresholds, as time durations, for each + priority queue. If the average response time for a queue is above this + threshold, backoff will occur in lower priority queues. This should be a + comma-separated list of length equal to the number of priority levels. + Threshold increases by 10s per level (e.g., for 4 levels: 10s,20s,30s,40s) + This property applies to DecayRpcScheduler. + + + + + ipc.[port_number].decay-scheduler.metrics.top.user.count + 10 + The number of top (i.e., heaviest) users to emit metric + information about. This property applies to DecayRpcScheduler. + + + + + ipc.[port_number].weighted-cost.lockshared + 10 + The weight multiplier to apply to the time spent in the + processing phase which holds a shared (read) lock. + This property applies to WeightedTimeCostProvider. + + + + + ipc.[port_number].weighted-cost.lockexclusive + 100 + The weight multiplier to apply to the time spent in the + processing phase which holds an exclusive (write) lock. + This property applies to WeightedTimeCostProvider. + + + + + ipc.[port_number].weighted-cost.handler + 1 + The weight multiplier to apply to the time spent in the + HANDLER phase which do not involve holding a lock. + See org.apache.hadoop.ipc.ProcessingDetails.Timing for more details on + this phase. This property applies to WeightedTimeCostProvider. + + + + + ipc.[port_number].weighted-cost.lockfree + 1 + The weight multiplier to apply to the time spent in the + LOCKFREE phase which do not involve holding a lock. + See org.apache.hadoop.ipc.ProcessingDetails.Timing for more details on + this phase. This property applies to WeightedTimeCostProvider. + + + + + ipc.[port_number].weighted-cost.response + 1 + The weight multiplier to apply to the time spent in the + RESPONSE phase which do not involve holding a lock. + See org.apache.hadoop.ipc.ProcessingDetails.Timing for more details on + this phase. This property applies to WeightedTimeCostProvider. + + + @@ -2472,6 +2717,19 @@ + + hadoop.http.authentication.kerberos.endpoint.whitelist + + + The comma-separated list of the endpoints that skips Kerberos + authentication. The endpoint must start with '/' and must not + contain special characters afterwards. This parameter is for + the monitoring tools that do not support Kerberos authentication. + Administrator must configure this parameter very carefully + because it allows unauthenticated access to the daemons. + + + hadoop.http.cross-origin.enabled @@ -2667,9 +2925,9 @@ hadoop.ssl.enabled.protocols - TLSv1.1,TLSv1.2 + TLSv1.2 - The supported SSL protocols. The parameter will only used from + The supported SSL protocols. The parameter will only be used from DatanodeHttpServer. @@ -2719,6 +2977,15 @@ + + ha.health-monitor.rpc.connect.max.retries + 1 + + The number of retries on connect error when establishing RPC proxy + connection to NameNode, used for monitorHealth() calls. + + + ha.health-monitor.rpc-timeout.ms 45000 @@ -3328,6 +3595,20 @@ + + adl.ssl.channel.mode + + + Valid inputs are OpenSSL, Default_JSE and Default (case insensitive). + If config is missing or is invalid, SSL Channel mode will be set to Default. + + When OpenSSL, SSL socket connections are created in OpenSSL mode. + When Default_JSE, SSL socket connections are created in the default JSE mode. + When Default, SSL socket connections are attempted with OpenSSL + and will fallback to Default_JSE mode if OpenSSL is not available at runtime. + + + @@ -3457,4 +3738,48 @@ with the input domain name of the services by querying the underlying DNS. + + + dfs.client.ignore.namenode.default.kms.uri + false + + Ignore KMS default URI returned from NameNode. + When set to true, kms uri is searched in the following order: + 1. If there is a mapping in Credential's secrets map for namenode uri. + 2. Fallback to local conf. (i.e hadoop.security.key.provider.path) + If client choose to ignore KMS uri provided by NameNode then client + should set KMS URI using 'hadoop.security.key.provider.path' to access + the right KMS for encrypted files. + + + + + hadoop.prometheus.endpoint.enabled + false + + If set to true, prometheus compatible metric page on the HTTP servers + is enabled via '/prom' endpoint. + + + + + fs.getspaceused.classname + + + The class that can tell estimate much space is used in a directory. + There are four impl classes that being supported: + org.apache.hadoop.fs.DU(default), org.apache.hadoop.fs.WindowsGetSpaceUsed + org.apache.hadoop.fs.DFCachingGetSpaceUsed and + org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed. + And the ReplicaCachingGetSpaceUsed impl class only used in HDFS module. + + + + + fs.getspaceused.jitterMillis + 60000 + + fs space usage statistics refresh jitter in msec. + + diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md b/hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md index f39a92d9e31af..0bda253fc8b54 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md @@ -125,6 +125,7 @@ Usage: `hadoop credential [options]` | create *alias* [-provider *provider-path*] [-strict] [-value *credential-value*] | Prompts the user for a credential to be stored as the given alias. The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. The `-strict` flag will cause the command to fail if the provider uses a default password. Use `-value` flag to supply the credential value (a.k.a. the alias password) instead of being prompted. | | delete *alias* [-provider *provider-path*] [-strict] [-f] | Deletes the credential with the provided alias. The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. The `-strict` flag will cause the command to fail if the provider uses a default password. The command asks for confirmation unless `-f` is specified | | list [-provider *provider-path*] [-strict] | Lists all of the credential aliases The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. The `-strict` flag will cause the command to fail if the provider uses a default password. | +| check *alias* [-provider *provider-path*] [-strict] | Check the password for the given alias. The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. The `-strict` flag will cause the command to fail if the provider uses a default password. | Command to manage credentials, passwords and secrets within credential providers. @@ -221,6 +222,8 @@ Usage: `hadoop key [options]` | roll *keyname* [-provider *provider*] [-strict] [-help] | Creates a new version for the specified key within the provider indicated using the `-provider` argument. The `-strict` flag will cause the command to fail if the provider uses a default password. | | delete *keyname* [-provider *provider*] [-strict] [-f] [-help] | Deletes all versions of the key specified by the *keyname* argument from within the provider specified by `-provider`. The `-strict` flag will cause the command to fail if the provider uses a default password. The command asks for user confirmation unless `-f` is specified. | | list [-provider *provider*] [-strict] [-metadata] [-help] | Displays the keynames contained within a particular provider as configured in core-site.xml or specified with the `-provider` argument. The `-strict` flag will cause the command to fail if the provider uses a default password. `-metadata` displays the metadata. | +| check *keyname* [-provider *provider*] [-strict] [-help] | Check password of the *keyname* contained within a particular provider as configured in core-site.xml or specified with the `-provider` argument. The `-strict` flag will cause the command to fail if the provider uses a default password. | + | -help | Prints usage of this command | Manage keys via the KeyProvider. For details on KeyProviders, see the [Transparent Encryption Guide](../hadoop-hdfs/TransparentEncryption.html). diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md b/hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md index 8ec7c14343865..281e42dad887f 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md @@ -65,7 +65,6 @@ The following table lists the configuration property names that are deprecated i | fs.s3a.server-side-encryption-key | fs.s3a.server-side-encryption.key | | hadoop.configured.node.mapping | net.topology.configured.node.mapping | | hadoop.native.lib | io.native.lib.available | -| hadoop.net.static.resolutions | mapreduce.tasktracker.net.static.resolutions | | hadoop.pipes.command-file.keep | mapreduce.pipes.commandfile.preserve | | hadoop.pipes.executable.interpretor | mapreduce.pipes.executable.interpretor | | hadoop.pipes.executable | mapreduce.pipes.executable | @@ -75,6 +74,10 @@ The following table lists the configuration property names that are deprecated i | hadoop.pipes.java.reducer | mapreduce.pipes.isjavareducer | | hadoop.pipes.partitioner | mapreduce.pipes.partitioner | | heartbeat.recheck.interval | dfs.namenode.heartbeat.recheck-interval | +| httpfs.authentication.kerberos.keytab | hadoop.http.authentication.kerberos.keytab | +| httpfs.authentication.kerberos.principal | hadoop.http.authentication.kerberos.principal | +| httpfs.authentication.signature.secret.file | hadoop.http.authentication.signature.secret.file | +| httpfs.authentication.type | hadoop.http.authentication.type | | io.bytes.per.checksum | dfs.bytes-per-checksum | | io.sort.factor | mapreduce.task.io.sort.factor | | io.sort.mb | mapreduce.task.io.sort.mb | @@ -89,7 +92,6 @@ The following table lists the configuration property names that are deprecated i | keep.failed.task.files | mapreduce.task.files.preserve.failedtasks | | keep.task.files.pattern | mapreduce.task.files.preserve.filepattern | | key.value.separator.in.input.line | mapreduce.input.keyvaluelinerecordreader.key.value.separator | -| local.cache.size | mapreduce.tasktracker.cache.local.size | | map.input.file | mapreduce.map.input.file | | map.input.length | mapreduce.map.input.length | | map.input.start | mapreduce.map.input.start | @@ -113,10 +115,6 @@ The following table lists the configuration property names that are deprecated i | mapred.compress.map.output | mapreduce.map.output.compress | | mapred.data.field.separator | mapreduce.fieldsel.data.field.separator | | mapred.debug.out.lines | mapreduce.task.debugout.lines | -| mapred.healthChecker.interval | mapreduce.tasktracker.healthchecker.interval | -| mapred.healthChecker.script.args | mapreduce.tasktracker.healthchecker.script.args | -| mapred.healthChecker.script.path | mapreduce.tasktracker.healthchecker.script.path | -| mapred.healthChecker.script.timeout | mapreduce.tasktracker.healthchecker.script.timeout | | mapred.inmem.merge.threshold | mapreduce.reduce.merge.inmem.threshold | | mapred.input.dir.formats | mapreduce.input.multipleinputs.dir.formats | | mapred.input.dir.mappers | mapreduce.input.multipleinputs.dir.mappers | @@ -146,8 +144,6 @@ The following table lists the configuration property names that are deprecated i | mapred.line.input.format.linespermap | mapreduce.input.lineinputformat.linespermap | | mapred.linerecordreader.maxlength | mapreduce.input.linerecordreader.line.maxlength | | mapred.local.dir | mapreduce.cluster.local.dir | -| mapred.local.dir.minspacekill | mapreduce.tasktracker.local.dir.minspacekill | -| mapred.local.dir.minspacestart | mapreduce.tasktracker.local.dir.minspacestart | | mapred.map.child.env | mapreduce.map.env | | mapred.map.child.java.opts | mapreduce.map.java.opts | | mapred.map.child.log.level | mapreduce.map.log.level | @@ -212,19 +208,10 @@ The following table lists the configuration property names that are deprecated i | mapred.task.profile.params | mapreduce.task.profile.params | | mapred.task.profile.reduces | mapreduce.task.profile.reduces | | mapred.task.timeout | mapreduce.task.timeout | -| mapred.tasktracker.dns.interface | mapreduce.tasktracker.dns.interface | -| mapred.tasktracker.dns.nameserver | mapreduce.tasktracker.dns.nameserver | -| mapred.tasktracker.events.batchsize | mapreduce.tasktracker.events.batchsize | -| mapred.task.tracker.http.address | mapreduce.tasktracker.http.address | | mapred.tasktracker.indexcache.mb | mapreduce.tasktracker.indexcache.mb | -| mapred.tasktracker.instrumentation | mapreduce.tasktracker.instrumentation | | mapred.tasktracker.map.tasks.maximum | mapreduce.tasktracker.map.tasks.maximum | | mapred.tasktracker.memory\_calculator\_plugin | mapreduce.tasktracker.resourcecalculatorplugin | | mapred.tasktracker.memorycalculatorplugin | mapreduce.tasktracker.resourcecalculatorplugin | -| mapred.tasktracker.reduce.tasks.maximum | mapreduce.tasktracker.reduce.tasks.maximum | -| mapred.task.tracker.report.address | mapreduce.tasktracker.report.address | -| mapred.task.tracker.task-controller | mapreduce.tasktracker.taskcontroller | -| mapred.tasktracker.tasks.sleeptime-before-sigkill | mapreduce.tasktracker.tasks.sleeptimebeforesigkill | | mapred.temp.dir | mapreduce.cluster.temp.dir | | mapred.text.key.comparator.options | mapreduce.partition.keycomparator.options | | mapred.text.key.partitioner.options | mapreduce.partition.keypartitioner.options | @@ -239,7 +226,6 @@ The following table lists the configuration property names that are deprecated i | mapreduce.reduce.class | mapreduce.job.reduce.class | | mapred.used.genericoptionsparser | mapreduce.client.genericoptionsparser.used | | mapred.userlog.limit.kb | mapreduce.task.userlog.limit.kb | -| mapred.userlog.retain.hours | mapreduce.job.userlog.retain.hours | | mapred.working.dir | mapreduce.job.working.dir | | mapred.work.output.dir | mapreduce.task.output.dir | | min.num.spills.for.combine | mapreduce.map.combine.minspills | @@ -251,9 +237,6 @@ The following table lists the configuration property names that are deprecated i | sequencefile.filter.regex | mapreduce.input.sequencefileinputfilter.regex | | session.id | dfs.metrics.session-id | | slave.host.name | dfs.datanode.hostname | -| slave.host.name | mapreduce.tasktracker.host.name | -| tasktracker.contention.tracking | mapreduce.tasktracker.contention.tracking | -| tasktracker.http.threads | mapreduce.tasktracker.http.threads | | topology.node.switch.mapping.impl | net.topology.node.switch.mapping.impl | | topology.script.file.name | net.topology.script.file.name | | topology.script.number.args | net.topology.script.number.args | diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/FairCallQueue.md b/hadoop-common-project/hadoop-common/src/site/markdown/FairCallQueue.md index e62c7ad42efda..22ac05a53b951 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/FairCallQueue.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/FairCallQueue.md @@ -91,6 +91,21 @@ This is configurable via the **identity provider**, which defaults to the **User provider simply uses the username of the client submitting the request. However, a custom identity provider can be used to performing throttling based on other groupings, or using an external identity provider. +### Cost-based Fair Call Queue + +Though the fair call queue itself does a good job of mitigating the impact from users who submit a very high _number_ +of requests, it does not take account into how expensive each request is to process. Thus, when considering the +HDFS NameNode, a user who submits 1000 "getFileInfo" requests would be prioritized the same as a user who submits 1000 +"listStatus" requests on some very large directory, or a user who submits 1000 "mkdir" requests, which are more +expensive as they require an exclusive lock on the namesystem. To account for the _cost_ of an operation when +considering the prioritization of user requests, there is a "cost-based" extension to the Fair Call Queue which uses +the aggregate processing time of a user's operations to determine how that user should be prioritized. By default, +queue time (time spent waiting to be processed) and lock wait time (time spent waiting to acquire a lock) is not +considered in the cost, time spent processing without a lock is neutrally (1x) weighted, time spent processing with a +shared lock is weighted 10x higher, and time spent processing with an exclusive lock is weighted 100x higher. +This attempts to prioritize users based on the actual load they place on the server. To enable this feature, set the +`costprovder.impl` configuration to `org.apache.hadoop.ipc.WeightedTimeCostProvider` as described below. + Configuration ------------- @@ -115,12 +130,16 @@ omitted. | scheduler.priority.levels | RpcScheduler, CallQueue | How many priority levels to use within the scheduler and call queue. | 4 | | faircallqueue.multiplexer.weights | WeightedRoundRobinMultiplexer | How much weight to give to each priority queue. This should be a comma-separated list of length equal to the number of priority levels. | Weights descend by a factor of 2 (e.g., for 4 levels: `8,4,2,1`) | | identity-provider.impl | DecayRpcScheduler | The identity provider mapping user requests to their identity. | org.apache.hadoop.ipc.UserIdentityProvider | +| cost-provider.impl | DecayRpcScheduler | The cost provider mapping user requests to their cost. To enable determination of cost based on processing time, use `org.apache.hadoop.ipc.WeightedTimeCostProvider`. | org.apache.hadoop.ipc.DefaultCostProvider | | decay-scheduler.period-ms | DecayRpcScheduler | How frequently the decay factor should be applied to the operation counts of users. Higher values have less overhead, but respond less quickly to changes in client behavior. | 5000 | | decay-scheduler.decay-factor | DecayRpcScheduler | When decaying the operation counts of users, the multiplicative decay factor to apply. Higher values will weight older operations more strongly, essentially giving the scheduler a longer memory, and penalizing heavy clients for a longer period of time. | 0.5 | | decay-scheduler.thresholds | DecayRpcScheduler | The client load threshold, as an integer percentage, for each priority queue. Clients producing less load, as a percent of total operations, than specified at position _i_ will be given priority _i_. This should be a comma-separated list of length equal to the number of priority levels minus 1 (the last is implicitly 100). | Thresholds ascend by a factor of 2 (e.g., for 4 levels: `13,25,50`) | | decay-scheduler.backoff.responsetime.enable | DecayRpcScheduler | Whether or not to enable the backoff by response time feature. | false | | decay-scheduler.backoff.responsetime.thresholds | DecayRpcScheduler | The response time thresholds, as time durations, for each priority queue. If the average response time for a queue is above this threshold, backoff will occur in lower priority queues. This should be a comma-separated list of length equal to the number of priority levels. | Threshold increases by 10s per level (e.g., for 4 levels: `10s,20s,30s,40s`) | | decay-scheduler.metrics.top.user.count | DecayRpcScheduler | The number of top (i.e., heaviest) users to emit metric information about. | 10 | +| weighted-cost.lockshared | WeightedTimeCostProvider | The weight multiplier to apply to the time spent in the processing phase which holds a shared (read) lock. | 10 | +| weighted-cost.lockexclusive | WeightedTimeCostProvider | The weight multiplier to apply to the time spent in the processing phase which holds an exclusive (write) lock. | 100 | +| weighted-cost.{handler,lockfree,response} | WeightedTimeCostProvider | The weight multiplier to apply to the time spent in the processing phases which do not involve holding a lock. See `org.apache.hadoop.ipc.ProcessingDetails.Timing` for more details on each phase. | 1 | ### Example Configuration diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md b/hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md index f050e30832c75..7df2cce574b68 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md @@ -268,7 +268,7 @@ Displays a summary of file lengths. expunge ------- -Usage: `hadoop fs -expunge [-immediate]` +Usage: `hadoop fs -expunge [-immediate] [-fs ]` Permanently delete files in checkpoints older than the retention threshold from trash directory, and create new checkpoint. @@ -286,6 +286,15 @@ This value should be smaller or equal to `fs.trash.interval`. If the `-immediate` option is passed, all files in the trash for the current user are immediately deleted, ignoring the `fs.trash.interval` setting. +If the `-fs` option is passed, the supplied filesystem will be expunged, +rather than the default filesystem and checkpoint is created. + +For example + +``` +hadoop fs -expunge --immediate -fs s3a://landsat-pds/ +``` + Refer to the [HDFS Architecture guide](../hadoop-hdfs/HdfsDesign.html#File_Deletes_and_Undeletes) for more information about trash feature of HDFS. @@ -629,7 +638,7 @@ Options: * -R: Apply operations to all files and directories recursively. * -m: Modify ACL. New entries are added to the ACL, and existing entries are retained. * -x: Remove specified ACL entries. Other ACL entries are retained. -* ``--set``: Fully replace the ACL, discarding all existing entries. The *acl\_spec* must include entries for user, group, and others for compatibility with permission bits. +* ``--set``: Fully replace the ACL, discarding all existing entries. The *acl\_spec* must include entries for user, group, and others for compatibility with permission bits. If the ACL spec contains only access entries, then the existing default entries are retained. If the ACL spec contains only default entries, then the existing access entries are retained. If the ACL spec contains both access and default entries, then both are replaced. * *acl\_spec*: Comma separated list of ACL entries. * *path*: File or directory to modify. @@ -724,16 +733,16 @@ Exit Code: Returns 0 on success and -1 on error. test ---- -Usage: `hadoop fs -test -[defsz] URI` +Usage: `hadoop fs -test -[defswrz] URI` Options: -* -d: f the path is a directory, return 0. +* -d: if the path is a directory, return 0. * -e: if the path exists, return 0. * -f: if the path is a file, return 0. * -s: if the path is not empty, return 0. -* -r: if the path exists and read permission is granted, return 0. * -w: if the path exists and write permission is granted, return 0. +* -r: if the path exists and read permission is granted, return 0. * -z: if the file is zero length, return 0. diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/HttpAuthentication.md b/hadoop-common-project/hadoop-common/src/site/markdown/HttpAuthentication.md index 721abea93b764..ca5ce4898aa71 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/HttpAuthentication.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/HttpAuthentication.md @@ -64,3 +64,13 @@ Add org.apache.hadoop.security.HttpCrossOriginFilterInitializer to hadoop.http.f | hadoop.http.cross-origin.allowed-methods | `GET,POST,HEAD` | Comma separated list of methods that are allowed | | hadoop.http.cross-origin.allowed-headers | `X-Requested-With,Content-Type,Accept,Origin` | Comma separated list of headers that are allowed | | hadoop.http.cross-origin.max-age | `1800` | Number of seconds a pre-flighted request can be cached | + + +Trusted Proxy +------------- +Trusted Proxy adds support to perform operations using end user instead of proxy user. It fetches the end user from +doAs query parameter. To enable Trusted Proxy, please set the following configuration parameter: + +Add org.apache.hadoop.security.authentication.server.ProxyUserAuthenticationFilterInitializer to hadoop.http.filter.initializers in core-site.xml +instead of org.apache.hadoop.security.AuthenticationFilterInitializer. + diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md b/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md index 1ef2b44b6ecb3..2d0f23293bfa3 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md @@ -71,6 +71,8 @@ Each metrics record contains tags such as Hostname and port (number to which ser | `SentBytes` | Total number of sent bytes | | `RpcQueueTimeNumOps` | Total number of RPC calls | | `RpcQueueTimeAvgTime` | Average queue time in milliseconds | +| `RpcLockWaitTimeNumOps` | Total number of RPC call (same as RpcQueueTimeNumOps) | +| `RpcLockWaitTimeAvgTime` | Average time waiting for lock acquisition in milliseconds | | `RpcProcessingTimeNumOps` | Total number of RPC calls (same to RpcQueueTimeNumOps) | | `RpcProcessingAvgTime` | Average Processing time in milliseconds | | `RpcAuthenticationFailures` | Total number of authentication failures | @@ -92,6 +94,12 @@ Each metrics record contains tags such as Hostname and port (number to which ser | `rpcProcessingTime`*num*`s90thPercentileLatency` | Shows the 90th percentile of RPC processing time in milliseconds (*num* seconds granularity) if `rpc.metrics.quantile.enable` is set to true. *num* is specified by `rpc.metrics.percentiles.intervals`. | | `rpcProcessingTime`*num*`s95thPercentileLatency` | Shows the 95th percentile of RPC processing time in milliseconds (*num* seconds granularity) if `rpc.metrics.quantile.enable` is set to true. *num* is specified by `rpc.metrics.percentiles.intervals`. | | `rpcProcessingTime`*num*`s99thPercentileLatency` | Shows the 99th percentile of RPC processing time in milliseconds (*num* seconds granularity) if `rpc.metrics.quantile.enable` is set to true. *num* is specified by `rpc.metrics.percentiles.intervals`. | +| `rpcLockWaitTime`*num*`sNumOps` | Shows total number of RPC calls (*num* seconds granularity) if `rpc.metrics.quantile.enable` is set to true. *num* is specified by `rpc.metrics.percentiles.intervals`. | +| `rpcLockWaitTime`*num*`s50thPercentileLatency` | Shows the 50th percentile of RPC lock wait time in milliseconds (*num* seconds granularity) if `rpc.metrics.quantile.enable` is set to true. *num* is specified by `rpc.metrics.percentiles.intervals`. | +| `rpcLockWaitTime`*num*`s75thPercentileLatency` | Shows the 75th percentile of RPC lock wait time in milliseconds (*num* seconds granularity) if `rpc.metrics.quantile.enable` is set to true. *num* is specified by `rpc.metrics.percentiles.intervals`. | +| `rpcLockWaitTime`*num*`s90thPercentileLatency` | Shows the 90th percentile of RPC lock wait time in milliseconds (*num* seconds granularity) if `rpc.metrics.quantile.enable` is set to true. *num* is specified by `rpc.metrics.percentiles.intervals`. | +| `rpcLockWaitTime`*num*`s95thPercentileLatency` | Shows the 95th percentile of RPC lock wait time in milliseconds (*num* seconds granularity) if `rpc.metrics.quantile.enable` is set to true. *num* is specified by `rpc.metrics.percentiles.intervals`. | +| `rpcLockWaitTime`*num*`s99thPercentileLatency` | Shows the 99th percentile of RPC lock wait time in milliseconds (*num* seconds granularity) if `rpc.metrics.quantile.enable` is set to true. *num* is specified by `rpc.metrics.percentiles.intervals`. | RetryCache/NameNodeRetryCache ----------------------------- @@ -118,6 +126,7 @@ rpcdetailed context =================== Metrics of rpcdetailed context are exposed in unified manner by RPC layer. Two metrics are exposed for each RPC based on its name. Metrics named "(RPC method name)NumOps" indicates total number of method calls, and metrics named "(RPC method name)AvgTime" shows average turn around time for method calls in milliseconds. +Please note that the AvgTime metrics do not include time spent waiting to acquire locks on data structures (see RpcLockWaitTimeAvgTime). rpcdetailed ----------- @@ -469,6 +478,40 @@ contains tags such as Hostname as additional information along with metrics. | `FileIoErrorRateNumOps` | The number of file io error operations within an interval time of metric | | `FileIoErrorRateAvgTime` | It measures the mean time in milliseconds from the start of an operation to hitting a failure | +RBFMetrics +---------------- +RBFMetrics shows the metrics which are the aggregated values of sub-clusters' information in the Router-based federation. + +| Name | Description | +|:---- |:---- | +| `NumFiles` | Current number of files and directories | +| `NumBlocks` | Current number of allocated blocks | +| `NumOfBlocksPendingReplication` | Current number of blocks pending to be replicated | +| `NumOfBlocksUnderReplicated` | Current number of blocks under replicated | +| `NumOfBlocksPendingDeletion` | Current number of blocks pending deletion | +| `ProvidedSpace` | The total remote storage capacity mounted in the federated cluster | +| `NumInMaintenanceLiveDataNodes` | Number of live Datanodes which are in maintenance state | +| `NumInMaintenanceDeadDataNodes` | Number of dead Datanodes which are in maintenance state | +| `NumEnteringMaintenanceDataNodes` | Number of Datanodes that are entering the maintenance state | +| `TotalCapacity` | Current raw capacity of DataNodes in bytes | +| `UsedCapacity` | Current used capacity across all DataNodes in bytes | +| `RemainingCapacity` | Current remaining capacity in bytes | +| `NumOfMissingBlocks` | Current number of missing blocks | +| `NumLiveNodes` | Number of datanodes which are currently live | +| `NumDeadNodes` | Number of datanodes which are currently dead | +| `NumStaleNodes` | Current number of DataNodes marked stale due to delayed heartbeat | +| `NumDecomLiveNodes` | Number of datanodes which have been decommissioned and are now live | +| `NumDecomDeadNodes` | Number of datanodes which have been decommissioned and are now dead | +| `NumDecommissioningNodes` | Number of datanodes in decommissioning state | +| `Namenodes` | Current information about all the namenodes | +| `Nameservices` | Current information for each registered nameservice | +| `MountTable` | The mount table for the federated filesystem | +| `Routers` | Current information about all routers | +| `NumNameservices` | Number of nameservices | +| `NumNamenodes` | Number of namenodes | +| `NumExpiredNamenodes` | Number of expired namenodes | +| `NodeUsage` | Max, Median, Min and Standard Deviation of DataNodes usage | + RouterRPCMetrics ---------------- RouterRPCMetrics shows the statistics of the Router component in Router-based federation. diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md index 9440b4e050a72..a2458ee891448 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md @@ -220,21 +220,21 @@ directory contains many thousands of files. Consider a directory `"/d"` with the contents: - a - part-0000001 - part-0000002 - ... - part-9999999 + a + part-0000001 + part-0000002 + ... + part-9999999 If the number of files is such that HDFS returns a partial listing in each response, then, if a listing `listStatus("/d")` takes place concurrently with the operation `rename("/d/a","/d/z"))`, the result may be one of: - [a, part-0000001, ... , part-9999999] - [part-0000001, ... , part-9999999, z] - [a, part-0000001, ... , part-9999999, z] - [part-0000001, ... , part-9999999] + [a, part-0000001, ... , part-9999999] + [part-0000001, ... , part-9999999, z] + [a, part-0000001, ... , part-9999999, z] + [part-0000001, ... , part-9999999] While this situation is likely to be a rare occurrence, it MAY happen. In HDFS these inconsistent views are only likely when listing a directory with many children. @@ -526,7 +526,7 @@ on the filesystem. `getFileStatus(P).getBlockSize()`. 1. By inference, it MUST be > 0 for any file of length > 0. -## State Changing Operations +## State Changing Operations ### `boolean mkdirs(Path p, FsPermission permission)` @@ -964,7 +964,7 @@ A path referring to a file is removed, return value: `True` Deleting an empty root does not change the filesystem state and may return true or false. - if isDir(FS, p) and isRoot(p) and children(FS, p) == {} : + if isRoot(p) and children(FS, p) == {} : FS ' = FS result = (undetermined) @@ -973,6 +973,9 @@ There is no consistent return code from an attempt to delete the root directory. Implementations SHOULD return true; this avoids code which checks for a false return value from overreacting. +*Object Stores*: see [Object Stores: root directory deletion](#object-stores-rm-root). + + ##### Empty (non-root) directory `recursive == False` Deleting an empty directory that is not root will remove the path from the FS and @@ -986,7 +989,7 @@ return true. ##### Recursive delete of non-empty root directory Deleting a root path with children and `recursive==True` - can do one of two things. +can generally have three outcomes: 1. The POSIX model assumes that if the user has the correct permissions to delete everything, @@ -1004,6 +1007,8 @@ filesystem is desired. FS' = FS result = False +1. Object Stores: see [Object Stores: root directory deletion](#object-stores-rm-root). + HDFS has the notion of *Protected Directories*, which are declared in the option `fs.protected.directories`. Any attempt to delete such a directory or a parent thereof raises an `AccessControlException`. Accordingly, any @@ -1019,6 +1024,23 @@ Any filesystem client which interacts with a remote filesystem which lacks such a security model, MAY reject calls to `delete("/", true)` on the basis that it makes it too easy to lose data. + +### Object Stores: root directory deletion + +Some of the object store based filesystem implementations always return +false when deleting the root, leaving the state of the store unchanged. + + if isRoot(p) : + FS ' = FS + result = False + +This is irrespective of the recursive flag status or the state of the directory. + +This is a simplification which avoids the inevitably non-atomic scan and delete +of the contents of the store. It also avoids any confusion about whether +the operation actually deletes that specific store/container itself, and +adverse consequences of the simpler permissions models of stores. + ##### Recursive delete of non-root directory Deleting a non-root path with children `recursive==true` @@ -1457,7 +1479,7 @@ public interface StreamCapabilities { ### `boolean hasCapability(capability)` -Return true if the `OutputStream`, `InputStream`, or other FileSystem class +Return true iff the `OutputStream`, `InputStream`, or other FileSystem class has the desired capability. The caller can query the capabilities of a stream using a string value. @@ -1470,3 +1492,4 @@ hsync | HSYNC | Syncable | Flush out the data in client's us in:readahead | READAHEAD | CanSetReadahead | Set the readahead on the input stream. dropbehind | DROPBEHIND | CanSetDropBehind | Drop the cache. in:unbuffer | UNBUFFER | CanUnbuffer | Reduce the buffering on the input stream. + diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstreambuilder.md b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstreambuilder.md index f1beed862cdbf..a7c393d9a41c1 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstreambuilder.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstreambuilder.md @@ -54,7 +54,7 @@ of `FileSystem`. ```java out = fs.openFile(path) - .opt("fs.s3a.experimental.fadvise", "random") + .opt("fs.s3a.experimental.input.fadvise", "random") .must("fs.s3a.readahead.range", 256 * 1024) .build() .get(); diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/index.md b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/index.md index 6b4399ea2123c..df538ee6cf96b 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/index.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/index.md @@ -33,6 +33,7 @@ HDFS as these are commonly expected by Hadoop client applications. 1. [Model](model.html) 1. [FileSystem class](filesystem.html) 1. [FSDataInputStream class](fsdatainputstream.html) +1. [PathCapabilities interface](pathcapabilities.html) 1. [FSDataOutputStreamBuilder class](fsdataoutputstreambuilder.html) 2. [Testing with the Filesystem specification](testing.html) 2. [Extending the specification and its tests](extending.html) diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/pathcapabilities.md b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/pathcapabilities.md new file mode 100644 index 0000000000000..e053bfbaede9b --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/pathcapabilities.md @@ -0,0 +1,158 @@ + + +# interface `PathCapabilities` + +The `PathCapabilities` interface provides a way to programmatically query the +operations offered under a given path by an instance of `FileSystem`, `FileContext` +or other implementing class. + +```java +public interface PathCapabilities { + boolean hasPathCapability(Path path, String capability) + throws IOException; +} +``` + +There are a number of goals here: + +1. Allow callers to probe for optional filesystem operations without actually +having to invoke them. +1. Allow filesystems with their own optional per-instance features to declare +whether or not they are active for the specific instance. +1. Allow for fileystem connectors which work with object stores to expose the +fundamental difference in semantics of these stores (e.g: files not visible +until closed, file rename being `O(data)`), directory rename being non-atomic, +etc. + +### Available Capabilities + +Capabilities are defined as strings and split into "Common Capabilites" +and non-standard ones for a specific store. + +The common capabilities are all defined under the prefix `fs.capability.` + +Consult the javadocs for `org.apache.hadoop.fs.CommonPathCapabilities` for these. + + +Individual filesystems MAY offer their own set of capabilities which +can be probed for. These MUST begin with `fs.` + the filesystem scheme + + `.capability`. For example `fs.s3a.capability.select.sql`; + +### `boolean hasPathCapability(path, capability)` + +Probe for the instance offering a specific capability under the +given path. + +#### Postconditions + +```python +if fs_supports_the_feature(path, capability): + return True +else: + return False +``` + +Return: `True`, iff the specific capability is available. + +A filesystem instance *MUST NOT* return `True` for any capability unless it is +known to be supported by that specific instance. As a result, if a caller +probes for a capability then it can assume that the specific feature/semantics +are available. + +If the probe returns `False` then it can mean one of: + +1. The capability is unknown. +1. The capability is known, and known to be unavailable on this instance. +1. The capability is known but this local class does not know if it is supported + under the supplied path. + +This predicate is intended to be low cost. If it requires remote calls other +than path/link resolution, it SHOULD conclude that the availability +of the feature is unknown and return `False`. + +The predicate MUST also be side-effect free. + +*Validity of paths* +There is no requirement that the existence of the path must be checked; +the parameter exists so that any filesystem which relays operations to other +filesystems (e.g `viewfs`) can resolve and relay it to the nested filesystem. +Consider the call to be *relatively* lightweight. + +Because of this, it may be that while the filesystem declares that +it supports a capability under a path, the actual invocation of the operation +may fail for other reasons. + +As an example, while a filesystem may support `append()` under a path, +if invoked on a directory, the call may fail. + +That is for a path `root = new Path("/")`: the capabilities call may succeed + +```java +fs.hasCapabilities(root, "fs.capability.append") == true +``` + +But a subsequent call to the operation on that specific path may fail, +because the root path is a directory: + +```java +fs.append(root) +``` + + +Similarly, there is no checking that the caller has the permission to +perform a specific operation: just because a feature is available on that +path does not mean that the caller can execute the operation. + +The `hasCapabilities(path, capability)` probe is therefore declaring that +the operation will not be rejected as unsupported, not that a specific invocation +will be permitted on that path by the caller. + +*Duration of availability* + +As the state of a remote store changes,so may path capabilities. This +may be due to changes in the local state of the fileystem (e.g. symbolic links +or mount points changing), or changes in its functionality (e.g. a feature +becoming availaible/unavailable due to operational changes, system upgrades, etc.) + +*Capabilities which must be invoked to determine availablity* + +Some operations may be known by the client connector, and believed to be available, +but may actually fail when invoked due to the state and permissons of the remote +store —state which is cannot be determined except by attempting +side-effecting operations. + +A key example of this is symbolic links and the local filesystem. +The filesystem declares that it supports this unless symbolic links are explicitly +disabled —when invoked they may actually fail. + +### Implementors Notes + +Implementors *MUST NOT* return `true` for any capability which is not guaranteed +to be supported. To return `true` indicates that the implementation/deployment +of the filesystem does, to the best of the knowledge of the filesystem client, +offer the desired operations *and semantics* queried for. + +For performance reasons, implementations *SHOULD NOT* check the path for +existence, unless it needs to resolve symbolic links in parts of the path +to determine whether a feature is present. This is required of `FileContext` +and `viewfs`. + +Individual filesystems *MUST NOT* unilaterally define new `fs.capability`-prefixed +capabilities. Instead they *MUST* do one of the following: + +* Define and stabilize new cross-filesystem capability flags (preferred), +and so formally add a new `fs.capability` value. +* Use the scheme of the filesystem to as a prefix for their own options, +e.g `fs.hdfs.` diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/2.10.0/CHANGES.2.10.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/2.10.0/CHANGES.2.10.0.md new file mode 100644 index 0000000000000..d8dd2daca09cc --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/2.10.0/CHANGES.2.10.0.md @@ -0,0 +1,788 @@ + + +# "Apache Hadoop" Changelog + +## Release 2.10.0 - 2019-10-22 + +### INCOMPATIBLE CHANGES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HDFS-12883](https://issues.apache.org/jira/browse/HDFS-12883) | RBF: Document Router and State Store metrics | Major | documentation | Yiqun Lin | Yiqun Lin | +| [HDFS-12895](https://issues.apache.org/jira/browse/HDFS-12895) | RBF: Add ACL support for mount table | Major | . | Yiqun Lin | Yiqun Lin | +| [HDFS-13099](https://issues.apache.org/jira/browse/HDFS-13099) | RBF: Use the ZooKeeper as the default State Store | Minor | documentation | Yiqun Lin | Yiqun Lin | +| [HADOOP-16055](https://issues.apache.org/jira/browse/HADOOP-16055) | Upgrade AWS SDK to 1.11.271 in branch-2 | Blocker | fs/s3 | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-16053](https://issues.apache.org/jira/browse/HADOOP-16053) | Backport HADOOP-14816 to branch-2 | Major | build | Akira Ajisaka | Akira Ajisaka | + + +### IMPORTANT ISSUES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HDFS-13083](https://issues.apache.org/jira/browse/HDFS-13083) | RBF: Fix doc error setting up client | Major | federation | tartarus | tartarus | + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HDFS-13283](https://issues.apache.org/jira/browse/HDFS-13283) | Percentage based Reserved Space Calculation for DataNode | Major | datanode, hdfs | Lukas Majercak | Lukas Majercak | +| [HDFS-13553](https://issues.apache.org/jira/browse/HDFS-13553) | RBF: Support global quota | Major | . | Íñigo Goiri | Yiqun Lin | +| [HADOOP-15950](https://issues.apache.org/jira/browse/HADOOP-15950) | Failover for LdapGroupsMapping | Major | common, security | Lukas Majercak | Lukas Majercak | +| [YARN-9761](https://issues.apache.org/jira/browse/YARN-9761) | Allow overriding application submissions based on server side configs | Major | . | Jonathan Hung | pralabhkumar | +| [YARN-9760](https://issues.apache.org/jira/browse/YARN-9760) | Support configuring application priorities on a workflow level | Major | . | Jonathan Hung | Varun Saxena | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-14987](https://issues.apache.org/jira/browse/HADOOP-14987) | Improve KMSClientProvider log around delegation token checking | Major | . | Xiaoyu Yao | Xiaoyu Yao | +| [HADOOP-14872](https://issues.apache.org/jira/browse/HADOOP-14872) | CryptoInputStream should implement unbuffer | Major | fs, security | John Zhuge | John Zhuge | +| [HADOOP-14960](https://issues.apache.org/jira/browse/HADOOP-14960) | Add GC time percentage monitor/alerter | Major | . | Misha Dmitriev | Misha Dmitriev | +| [HADOOP-15023](https://issues.apache.org/jira/browse/HADOOP-15023) | ValueQueue should also validate (lowWatermark \* numValues) \> 0 on construction | Minor | . | Xiao Chen | Xiao Chen | +| [YARN-6851](https://issues.apache.org/jira/browse/YARN-6851) | Capacity Scheduler: document configs for controlling # containers allowed to be allocated per node heartbeat | Minor | . | Wei Yan | Wei Yan | +| [YARN-7495](https://issues.apache.org/jira/browse/YARN-7495) | Improve robustness of the AggregatedLogDeletionService | Major | log-aggregation | Jonathan Turner Eagles | Jonathan Turner Eagles | +| [HADOOP-15056](https://issues.apache.org/jira/browse/HADOOP-15056) | Fix TestUnbuffer#testUnbufferException failure | Minor | test | Jack Bearden | Jack Bearden | +| [HADOOP-15012](https://issues.apache.org/jira/browse/HADOOP-15012) | Add readahead, dropbehind, and unbuffer to StreamCapabilities | Major | fs | John Zhuge | John Zhuge | +| [HADOOP-15104](https://issues.apache.org/jira/browse/HADOOP-15104) | AliyunOSS: change the default value of max error retry | Major | fs/oss | wujinhu | wujinhu | +| [YARN-7274](https://issues.apache.org/jira/browse/YARN-7274) | Ability to disable elasticity at leaf queue level | Major | capacityscheduler | Scott Brokaw | Zian Chen | +| [YARN-7642](https://issues.apache.org/jira/browse/YARN-7642) | Add test case to verify context update after container promotion or demotion with or without auto update | Minor | nodemanager | Weiwei Yang | Weiwei Yang | +| [HADOOP-15111](https://issues.apache.org/jira/browse/HADOOP-15111) | AliyunOSS: backport HADOOP-14993 to branch-2 | Major | fs/oss | Genmao Yu | Genmao Yu | +| [HDFS-12818](https://issues.apache.org/jira/browse/HDFS-12818) | Support multiple storages in DataNodeCluster / SimulatedFSDataset | Minor | datanode, test | Erik Krogen | Erik Krogen | +| [HDFS-9023](https://issues.apache.org/jira/browse/HDFS-9023) | When NN is not able to identify DN for replication, reason behind it can be logged | Critical | hdfs-client, namenode | Surendra Singh Lilhore | Xiao Chen | +| [YARN-7678](https://issues.apache.org/jira/browse/YARN-7678) | Ability to enable logging of container memory stats | Major | nodemanager | Jim Brennan | Jim Brennan | +| [HDFS-12945](https://issues.apache.org/jira/browse/HDFS-12945) | Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethods | Minor | . | Wei Yan | Wei Yan | +| [YARN-7622](https://issues.apache.org/jira/browse/YARN-7622) | Allow fair-scheduler configuration on HDFS | Minor | fairscheduler, resourcemanager | Greg Phillips | Greg Phillips | +| [YARN-7590](https://issues.apache.org/jira/browse/YARN-7590) | Improve container-executor validation check | Major | security, yarn | Eric Yang | Eric Yang | +| [MAPREDUCE-7029](https://issues.apache.org/jira/browse/MAPREDUCE-7029) | FileOutputCommitter is slow on filesystems lacking recursive delete | Minor | . | Karthik Palaniappan | Karthik Palaniappan | +| [MAPREDUCE-6984](https://issues.apache.org/jira/browse/MAPREDUCE-6984) | MR AM to clean up temporary files from previous attempt in case of no recovery | Major | applicationmaster | Gergo Repas | Gergo Repas | +| [HADOOP-15189](https://issues.apache.org/jira/browse/HADOOP-15189) | backport HADOOP-15039 to branch-2 and branch-3 | Blocker | . | Genmao Yu | Genmao Yu | +| [HADOOP-15212](https://issues.apache.org/jira/browse/HADOOP-15212) | Add independent secret manager method for logging expired tokens | Major | security | Daryn Sharp | Daryn Sharp | +| [YARN-7728](https://issues.apache.org/jira/browse/YARN-7728) | Expose container preemptions related information in Capacity Scheduler queue metrics | Major | . | Eric Payne | Eric Payne | +| [MAPREDUCE-7048](https://issues.apache.org/jira/browse/MAPREDUCE-7048) | Uber AM can crash due to unknown task in statusUpdate | Major | mr-am | Peter Bacsko | Peter Bacsko | +| [HADOOP-13972](https://issues.apache.org/jira/browse/HADOOP-13972) | ADLS to support per-store configuration | Major | fs/adl | John Zhuge | Sharad Sonker | +| [YARN-7813](https://issues.apache.org/jira/browse/YARN-7813) | Capacity Scheduler Intra-queue Preemption should be configurable for each queue | Major | capacity scheduler, scheduler preemption | Eric Payne | Eric Payne | +| [HADOOP-15235](https://issues.apache.org/jira/browse/HADOOP-15235) | Authentication Tokens should use HMAC instead of MAC | Major | security | Robert Kanter | Robert Kanter | +| [HDFS-11187](https://issues.apache.org/jira/browse/HDFS-11187) | Optimize disk access for last partial chunk checksum of Finalized replica | Major | datanode | Wei-Chiu Chuang | Gabor Bota | +| [HADOOP-15266](https://issues.apache.org/jira/browse/HADOOP-15266) | [branch-2] Upper/Lower case conversion support for group names in LdapGroupsMapping | Major | . | Nanda kumar | Nanda kumar | +| [HADOOP-15279](https://issues.apache.org/jira/browse/HADOOP-15279) | increase maven heap size recommendations | Minor | build, documentation, test | Allen Wittenauer | Allen Wittenauer | +| [HDFS-12884](https://issues.apache.org/jira/browse/HDFS-12884) | BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo | Major | namenode | Konstantin Shvachko | chencan | +| [HADOOP-15334](https://issues.apache.org/jira/browse/HADOOP-15334) | Upgrade Maven surefire plugin | Major | build | Arpit Agarwal | Arpit Agarwal | +| [HADOOP-15312](https://issues.apache.org/jira/browse/HADOOP-15312) | Undocumented KeyProvider configuration keys | Major | . | Wei-Chiu Chuang | LiXin Ge | +| [YARN-7623](https://issues.apache.org/jira/browse/YARN-7623) | Fix the CapacityScheduler Queue configuration documentation | Major | . | Arun Suresh | Jonathan Hung | +| [HDFS-13314](https://issues.apache.org/jira/browse/HDFS-13314) | NameNode should optionally exit if it detects FsImage corruption | Major | namenode | Arpit Agarwal | Arpit Agarwal | +| [HDFS-13418](https://issues.apache.org/jira/browse/HDFS-13418) | NetworkTopology should be configurable when enable DFSNetworkTopology | Major | . | Tao Jie | Tao Jie | +| [HADOOP-15394](https://issues.apache.org/jira/browse/HADOOP-15394) | Backport PowerShell NodeFencer HADOOP-14309 to branch-2 | Minor | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13462](https://issues.apache.org/jira/browse/HDFS-13462) | Add BIND\_HOST configuration for JournalNode's HTTP and RPC Servers | Major | hdfs, journal-node | Lukas Majercak | Lukas Majercak | +| [HDFS-13492](https://issues.apache.org/jira/browse/HDFS-13492) | Limit httpfs binds to certain IP addresses in branch-2 | Major | httpfs | Wei-Chiu Chuang | Wei-Chiu Chuang | +| [HDFS-12981](https://issues.apache.org/jira/browse/HDFS-12981) | renameSnapshot a Non-Existent snapshot to itself should throw error | Minor | hdfs | Sailesh Patel | Kitti Nanasi | +| [HADOOP-15441](https://issues.apache.org/jira/browse/HADOOP-15441) | Log kms url and token service at debug level. | Minor | . | Wei-Chiu Chuang | Gabor Bota | +| [HDFS-13544](https://issues.apache.org/jira/browse/HDFS-13544) | Improve logging for JournalNode in federated cluster | Major | federation, hdfs | Hanisha Koneru | Hanisha Koneru | +| [YARN-8249](https://issues.apache.org/jira/browse/YARN-8249) | Few REST api's in RMWebServices are missing static user check | Critical | webapp, yarn | Sunil G | Sunil G | +| [HADOOP-15486](https://issues.apache.org/jira/browse/HADOOP-15486) | Make NetworkTopology#netLock fair | Major | net | Nanda kumar | Nanda kumar | +| [HADOOP-15449](https://issues.apache.org/jira/browse/HADOOP-15449) | Increase default timeout of ZK session to avoid frequent NameNode failover | Critical | common | Karthik Palanisamy | Karthik Palanisamy | +| [HDFS-13602](https://issues.apache.org/jira/browse/HDFS-13602) | Add checkOperation(WRITE) checks in FSNamesystem | Major | ha, namenode | Erik Krogen | Chao Sun | +| [HDFS-13644](https://issues.apache.org/jira/browse/HDFS-13644) | Backport HDFS-10376 to branch-2 | Major | . | Yiqun Lin | Zsolt Venczel | +| [HDFS-13653](https://issues.apache.org/jira/browse/HDFS-13653) | Make dfs.client.failover.random.order a per nameservice configuration | Major | federation | Ekanth Sethuramalingam | Ekanth Sethuramalingam | +| [HDFS-13686](https://issues.apache.org/jira/browse/HDFS-13686) | Add overall metrics for FSNamesystemLock | Major | hdfs, namenode | Lukas Majercak | Lukas Majercak | +| [HDFS-13714](https://issues.apache.org/jira/browse/HDFS-13714) | Fix TestNameNodePrunesMissingStorages test failures on Windows | Major | hdfs, namenode, test | Lukas Majercak | Lukas Majercak | +| [HDFS-13719](https://issues.apache.org/jira/browse/HDFS-13719) | Docs around dfs.image.transfer.timeout are misleading | Major | documentation | Kitti Nanasi | Kitti Nanasi | +| [HDFS-11060](https://issues.apache.org/jira/browse/HDFS-11060) | make DEFAULT\_MAX\_CORRUPT\_FILEBLOCKS\_RETURNED configurable | Minor | hdfs | Lantao Jin | Lantao Jin | +| [YARN-8155](https://issues.apache.org/jira/browse/YARN-8155) | Improve ATSv2 client logging in RM and NM publisher | Major | . | Rohith Sharma K S | Abhishek Modi | +| [HDFS-13814](https://issues.apache.org/jira/browse/HDFS-13814) | Remove super user privilege requirement for NameNode.getServiceStatus | Minor | namenode | Chao Sun | Chao Sun | +| [YARN-8559](https://issues.apache.org/jira/browse/YARN-8559) | Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint | Major | resourcemanager | Anna Savarin | Weiwei Yang | +| [HDFS-13813](https://issues.apache.org/jira/browse/HDFS-13813) | Exit NameNode if dangling child inode is detected when saving FsImage | Major | hdfs, namenode | Siyao Meng | Siyao Meng | +| [HDFS-13821](https://issues.apache.org/jira/browse/HDFS-13821) | RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache | Major | hdfs | Fei Hui | Fei Hui | +| [HADOOP-15689](https://issues.apache.org/jira/browse/HADOOP-15689) | Add "\*.patch" into .gitignore file of branch-2 | Major | . | Rui Gao | Rui Gao | +| [HDFS-13831](https://issues.apache.org/jira/browse/HDFS-13831) | Make block increment deletion number configurable | Major | . | Yiqun Lin | Ryan Wu | +| [YARN-8051](https://issues.apache.org/jira/browse/YARN-8051) | TestRMEmbeddedElector#testCallbackSynchronization is flakey | Major | test | Robert Kanter | Robert Kanter | +| [HADOOP-15547](https://issues.apache.org/jira/browse/HADOOP-15547) | WASB: improve listStatus performance | Major | fs/azure | Thomas Marqardt | Thomas Marqardt | +| [HDFS-13857](https://issues.apache.org/jira/browse/HDFS-13857) | RBF: Choose to enable the default nameservice to read/write files | Major | federation, hdfs | yanghuafeng | yanghuafeng | +| [HDFS-13812](https://issues.apache.org/jira/browse/HDFS-13812) | Fix the inconsistent default refresh interval on Caching documentation | Trivial | documentation | David Mollitor | Hrishikesh Gadre | +| [HADOOP-15657](https://issues.apache.org/jira/browse/HADOOP-15657) | Registering MutableQuantiles via Metric annotation | Major | metrics | Sushil Ks | Sushil Ks | +| [HDFS-13902](https://issues.apache.org/jira/browse/HDFS-13902) | Add JMX, conf and stacks menus to the datanode page | Minor | datanode | fengchuang | fengchuang | +| [HADOOP-15726](https://issues.apache.org/jira/browse/HADOOP-15726) | Create utility to limit frequency of log statements | Major | common, util | Erik Krogen | Erik Krogen | +| [YARN-7974](https://issues.apache.org/jira/browse/YARN-7974) | Allow updating application tracking url after registration | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-8750](https://issues.apache.org/jira/browse/YARN-8750) | Refactor TestQueueMetrics | Minor | resourcemanager | Szilard Nemeth | Szilard Nemeth | +| [YARN-8896](https://issues.apache.org/jira/browse/YARN-8896) | Limit the maximum number of container assignments per heartbeat | Major | . | Weiwei Yang | Zhankun Tang | +| [HADOOP-15804](https://issues.apache.org/jira/browse/HADOOP-15804) | upgrade to commons-compress 1.18 | Major | . | PJ Fanning | Akira Ajisaka | +| [YARN-8915](https://issues.apache.org/jira/browse/YARN-8915) | Update the doc about the default value of "maximum-container-assignments" for capacity scheduler | Minor | . | Zhankun Tang | Zhankun Tang | +| [YARN-7225](https://issues.apache.org/jira/browse/YARN-7225) | Add queue and partition info to RM audit log | Major | resourcemanager | Jonathan Hung | Eric Payne | +| [HADOOP-15919](https://issues.apache.org/jira/browse/HADOOP-15919) | AliyunOSS: Enable Yarn to use OSS | Major | fs/oss | wujinhu | wujinhu | +| [HADOOP-15943](https://issues.apache.org/jira/browse/HADOOP-15943) | AliyunOSS: add missing owner & group attributes for oss FileStatus | Major | fs/oss | wujinhu | wujinhu | +| [YARN-9036](https://issues.apache.org/jira/browse/YARN-9036) | Escape newlines in health report in YARN UI | Major | . | Jonathan Hung | Keqiu Hu | +| [YARN-9085](https://issues.apache.org/jira/browse/YARN-9085) | Add Guaranteed and MaxCapacity to CSQueueMetrics | Major | . | Jonathan Hung | Jonathan Hung | +| [HDFS-14171](https://issues.apache.org/jira/browse/HDFS-14171) | Performance improvement in Tailing EditLog | Major | namenode | Kenneth Yang | Kenneth Yang | +| [HADOOP-15481](https://issues.apache.org/jira/browse/HADOOP-15481) | Emit FairCallQueue stats as metrics | Major | metrics, rpc-server | Erik Krogen | Christopher Gregorian | +| [HADOOP-15617](https://issues.apache.org/jira/browse/HADOOP-15617) | Node.js and npm package loading in the Dockerfile failing on branch-2 | Major | build | Allen Wittenauer | Akhil PB | +| [HADOOP-16089](https://issues.apache.org/jira/browse/HADOOP-16089) | AliyunOSS: update oss-sdk version to 3.4.1 | Major | fs/oss | wujinhu | wujinhu | +| [YARN-7171](https://issues.apache.org/jira/browse/YARN-7171) | RM UI should sort memory / cores numerically | Major | . | Eric Maynard | Ahmed Hussein | +| [YARN-9282](https://issues.apache.org/jira/browse/YARN-9282) | Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be 'authentication' | Trivial | . | Szilard Nemeth | Charan Hebri | +| [HADOOP-16126](https://issues.apache.org/jira/browse/HADOOP-16126) | ipc.Client.stop() may sleep too long to wait for all connections | Major | ipc | Tsz-wo Sze | Tsz-wo Sze | +| [HDFS-14247](https://issues.apache.org/jira/browse/HDFS-14247) | Repeat adding node description into network topology | Minor | datanode | HuangTao | HuangTao | +| [YARN-9150](https://issues.apache.org/jira/browse/YARN-9150) | Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2 | Major | ATSv2 | Sushil Ks | Sushil Ks | +| [HDFS-14366](https://issues.apache.org/jira/browse/HDFS-14366) | Improve HDFS append performance | Major | hdfs | Chao Sun | Chao Sun | +| [HDFS-14205](https://issues.apache.org/jira/browse/HDFS-14205) | Backport HDFS-6440 to branch-2 | Major | . | Chen Liang | Chao Sun | +| [HDFS-14391](https://issues.apache.org/jira/browse/HDFS-14391) | Backport HDFS-9659 to branch-2 | Minor | . | Chao Sun | Chao Sun | +| [HDFS-14415](https://issues.apache.org/jira/browse/HDFS-14415) | Backport HDFS-13799 to branch-2 | Trivial | . | Chao Sun | Chao Sun | +| [HDFS-14432](https://issues.apache.org/jira/browse/HDFS-14432) | dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml | Minor | hdfs | puleya7 | puleya7 | +| [YARN-9529](https://issues.apache.org/jira/browse/YARN-9529) | Log correct cpu controller path on error while initializing CGroups. | Major | nodemanager | Jonathan Hung | Jonathan Hung | +| [HDFS-14502](https://issues.apache.org/jira/browse/HDFS-14502) | keepResults option in NNThroughputBenchmark should call saveNamespace() | Major | benchmarks, hdfs | Konstantin Shvachko | Konstantin Shvachko | +| [HADOOP-16323](https://issues.apache.org/jira/browse/HADOOP-16323) | https everywhere in Maven settings | Minor | build | Akira Ajisaka | Akira Ajisaka | +| [YARN-9563](https://issues.apache.org/jira/browse/YARN-9563) | Resource report REST API could return NaN or Inf | Minor | . | Ahmed Hussein | Ahmed Hussein | +| [HDFS-14513](https://issues.apache.org/jira/browse/HDFS-14513) | FSImage which is saving should be clean while NameNode shutdown | Major | namenode | Xiaoqiao He | Xiaoqiao He | +| [HADOOP-16369](https://issues.apache.org/jira/browse/HADOOP-16369) | Fix zstandard shortname misspelled as zts | Major | . | Jonathan Turner Eagles | Jonathan Turner Eagles | +| [HADOOP-16359](https://issues.apache.org/jira/browse/HADOOP-16359) | Bundle ZSTD native in branch-2 | Major | native | Chao Sun | Chao Sun | +| [HADOOP-15253](https://issues.apache.org/jira/browse/HADOOP-15253) | Should update maxQueueSize when refresh call queue | Minor | . | Tao Jie | Tao Jie | +| [HADOOP-16266](https://issues.apache.org/jira/browse/HADOOP-16266) | Add more fine-grained processing time metrics to the RPC layer | Minor | ipc | Christopher Gregorian | Erik Krogen | +| [HDFS-13694](https://issues.apache.org/jira/browse/HDFS-13694) | Making md5 computing being in parallel with image loading | Major | . | zhouyingchao | Lisheng Sun | +| [HDFS-14632](https://issues.apache.org/jira/browse/HDFS-14632) | Reduce useless #getNumLiveDataNodes call in SafeModeMonitor | Major | namenode | Xiaoqiao He | Xiaoqiao He | +| [HDFS-14547](https://issues.apache.org/jira/browse/HDFS-14547) | DirectoryWithQuotaFeature.quota costs additional memory even the storage type quota is not set. | Major | . | Jinglun | Jinglun | +| [HDFS-14697](https://issues.apache.org/jira/browse/HDFS-14697) | Backport HDFS-14513 to branch-2 | Minor | namenode | Xiaoqiao He | Xiaoqiao He | +| [YARN-8045](https://issues.apache.org/jira/browse/YARN-8045) | Reduce log output from container status calls | Major | . | Shane Kumpf | Craig Condit | +| [HDFS-14313](https://issues.apache.org/jira/browse/HDFS-14313) | Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du | Major | datanode, performance | Lisheng Sun | Lisheng Sun | +| [HDFS-14696](https://issues.apache.org/jira/browse/HDFS-14696) | Backport HDFS-11273 to branch-2 (Move TransferFsImage#doGetUrl function to a Util class) | Major | . | Siyao Meng | Siyao Meng | +| [HDFS-14370](https://issues.apache.org/jira/browse/HDFS-14370) | Edit log tailing fast-path should allow for backoff | Major | namenode, qjm | Erik Krogen | Erik Krogen | +| [YARN-9442](https://issues.apache.org/jira/browse/YARN-9442) | container working directory has group read permissions | Minor | yarn | Jim Brennan | Jim Brennan | +| [HADOOP-16459](https://issues.apache.org/jira/browse/HADOOP-16459) | Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2 | Major | . | Erik Krogen | Erik Krogen | +| [HDFS-14707](https://issues.apache.org/jira/browse/HDFS-14707) | Add JAVA\_LIBRARY\_PATH to HTTPFS startup options in branch-2 | Major | httpfs | Masatake Iwasaki | Masatake Iwasaki | +| [HDFS-14723](https://issues.apache.org/jira/browse/HDFS-14723) | Add helper method FSNamesystem#setBlockManagerForTesting() in branch-2 | Blocker | namenode | Wei-Chiu Chuang | Wei-Chiu Chuang | +| [HDFS-14276](https://issues.apache.org/jira/browse/HDFS-14276) | [SBN read] Reduce tailing overhead | Major | ha, namenode | Wei-Chiu Chuang | Ayush Saxena | +| [HDFS-14617](https://issues.apache.org/jira/browse/HDFS-14617) | Improve fsimage load time by writing sub-sections to the fsimage index | Major | namenode | Stephen O'Donnell | Stephen O'Donnell | +| [YARN-9756](https://issues.apache.org/jira/browse/YARN-9756) | Create metric that sums total memory/vcores preempted per round | Major | capacity scheduler | Eric Payne | Manikandan R | +| [HDFS-14633](https://issues.apache.org/jira/browse/HDFS-14633) | The StorageType quota and consume in QuotaFeature is not handled for rename | Major | . | Jinglun | Jinglun | +| [HADOOP-16439](https://issues.apache.org/jira/browse/HADOOP-16439) | Upgrade bundled Tomcat in branch-2 | Major | httpfs, kms | Masatake Iwasaki | Masatake Iwasaki | +| [YARN-9810](https://issues.apache.org/jira/browse/YARN-9810) | Add queue capacity/maxcapacity percentage metrics | Major | . | Jonathan Hung | Shubham Gupta | +| [YARN-9763](https://issues.apache.org/jira/browse/YARN-9763) | Print application tags in application summary | Major | . | Jonathan Hung | Manoj Kumar | +| [YARN-9764](https://issues.apache.org/jira/browse/YARN-9764) | Print application submission context label in application summary | Major | . | Jonathan Hung | Manoj Kumar | +| [HADOOP-16530](https://issues.apache.org/jira/browse/HADOOP-16530) | Update xercesImpl in branch-2 | Major | . | Wei-Chiu Chuang | Wei-Chiu Chuang | +| [YARN-9824](https://issues.apache.org/jira/browse/YARN-9824) | Fall back to configured queue ordering policy class name | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9825](https://issues.apache.org/jira/browse/YARN-9825) | Changes for initializing placement rules with ResourceScheduler in branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9762](https://issues.apache.org/jira/browse/YARN-9762) | Add submission context label to audit logs | Major | . | Jonathan Hung | Manoj Kumar | +| [HDFS-14667](https://issues.apache.org/jira/browse/HDFS-14667) | Backport [HDFS-14403] "Cost-based FairCallQueue" to branch-2 | Major | . | Erik Krogen | Erik Krogen | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [MAPREDUCE-6896](https://issues.apache.org/jira/browse/MAPREDUCE-6896) | Document wrong spelling in usage of MapredTestDriver tools. | Major | documentation | LiXin Ge | LiXin Ge | +| [HDFS-12052](https://issues.apache.org/jira/browse/HDFS-12052) | Set SWEBHDFS delegation token kind when ssl is enabled in HttpFS | Major | httpfs, webhdfs | Zoran Dimitrijevic | Zoran Dimitrijevic | +| [HDFS-12318](https://issues.apache.org/jira/browse/HDFS-12318) | Fix IOException condition for openInfo in DFSInputStream | Major | . | legend | legend | +| [HDFS-12614](https://issues.apache.org/jira/browse/HDFS-12614) | FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured | Major | . | Manoj Govindassamy | Manoj Govindassamy | +| [YARN-7396](https://issues.apache.org/jira/browse/YARN-7396) | NPE when accessing container logs due to null dirsHandler | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-7370](https://issues.apache.org/jira/browse/YARN-7370) | Preemption properties should be refreshable | Major | capacity scheduler, scheduler preemption | Eric Payne | Gergely Novák | +| [YARN-7428](https://issues.apache.org/jira/browse/YARN-7428) | Add containerId to Localizer failed logs | Minor | nodemanager | Prabhu Joseph | Prabhu Joseph | +| [YARN-7410](https://issues.apache.org/jira/browse/YARN-7410) | Cleanup FixedValueResource to avoid dependency to ResourceUtils | Major | resourcemanager | Sunil G | Wangda Tan | +| [HDFS-12783](https://issues.apache.org/jira/browse/HDFS-12783) | [branch-2] "dfsrouter" should use hdfsScript | Major | . | Brahma Reddy Battula | Brahma Reddy Battula | +| [HDFS-12788](https://issues.apache.org/jira/browse/HDFS-12788) | Reset the upload button when file upload fails | Critical | ui, webhdfs | Brahma Reddy Battula | Brahma Reddy Battula | +| [YARN-7469](https://issues.apache.org/jira/browse/YARN-7469) | Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit | Major | capacity scheduler, yarn | Eric Payne | Eric Payne | +| [HADOOP-14982](https://issues.apache.org/jira/browse/HADOOP-14982) | Clients using FailoverOnNetworkExceptionRetry can go into a loop if they're used without authenticating with kerberos in HA env | Major | common | Peter Bacsko | Peter Bacsko | +| [YARN-7489](https://issues.apache.org/jira/browse/YARN-7489) | ConcurrentModificationException in RMAppImpl#getRMAppMetrics | Major | capacityscheduler | Tao Yang | Tao Yang | +| [YARN-7525](https://issues.apache.org/jira/browse/YARN-7525) | Incorrect query parameters in cluster nodes REST API document | Minor | documentation | Tao Yang | Tao Yang | +| [HDFS-12813](https://issues.apache.org/jira/browse/HDFS-12813) | RequestHedgingProxyProvider can hide Exception thrown from the Namenode for proxy size of 1 | Major | ha | Mukul Kumar Singh | Mukul Kumar Singh | +| [HADOOP-15045](https://issues.apache.org/jira/browse/HADOOP-15045) | ISA-L build options are documented in branch-2 | Major | build, documentation | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-15067](https://issues.apache.org/jira/browse/HADOOP-15067) | GC time percentage reported in JvmMetrics should be a gauge, not counter | Major | . | Misha Dmitriev | Misha Dmitriev | +| [YARN-7363](https://issues.apache.org/jira/browse/YARN-7363) | ContainerLocalizer doesn't have a valid log4j config when using LinuxContainerExecutor | Major | nodemanager | Yufei Gu | Yufei Gu | +| [HDFS-12754](https://issues.apache.org/jira/browse/HDFS-12754) | Lease renewal can hit a deadlock | Major | . | Kuhu Shukla | Kuhu Shukla | +| [HDFS-12832](https://issues.apache.org/jira/browse/HDFS-12832) | INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit | Critical | namenode | DENG FEI | Konstantin Shvachko | +| [HDFS-11754](https://issues.apache.org/jira/browse/HDFS-11754) | Make FsServerDefaults cache configurable. | Minor | . | Rushabh Shah | Mikhail Erofeev | +| [HADOOP-15042](https://issues.apache.org/jira/browse/HADOOP-15042) | Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0 | Minor | fs/azure | Rajesh Balamohan | Rajesh Balamohan | +| [HDFS-12638](https://issues.apache.org/jira/browse/HDFS-12638) | Delete copy-on-truncate block along with the original block, when deleting a file being truncated | Blocker | hdfs | Jiandan Yang | Konstantin Shvachko | +| [YARN-4813](https://issues.apache.org/jira/browse/YARN-4813) | TestRMWebServicesDelegationTokenAuthentication.testDoAs fails intermittently | Major | resourcemanager | Daniel Templeton | Gergo Repas | +| [MAPREDUCE-5124](https://issues.apache.org/jira/browse/MAPREDUCE-5124) | AM lacks flow control for task events | Major | mr-am | Jason Darrell Lowe | Peter Bacsko | +| [YARN-7455](https://issues.apache.org/jira/browse/YARN-7455) | quote\_and\_append\_arg can overflow buffer | Major | nodemanager | Jason Darrell Lowe | Jim Brennan | +| [YARN-7594](https://issues.apache.org/jira/browse/YARN-7594) | TestNMWebServices#testGetNMResourceInfo fails on trunk | Major | nodemanager, webapp | Gergely Novák | Gergely Novák | +| [YARN-5594](https://issues.apache.org/jira/browse/YARN-5594) | Handle old RMDelegationToken format when recovering RM | Major | resourcemanager | Tatyana But | Robert Kanter | +| [HADOOP-14985](https://issues.apache.org/jira/browse/HADOOP-14985) | Remove subversion related code from VersionInfoMojo.java | Minor | build | Akira Ajisaka | Ajay Kumar | +| [HDFS-11751](https://issues.apache.org/jira/browse/HDFS-11751) | DFSZKFailoverController daemon exits with wrong status code | Major | auto-failover | Doris Gu | Bharat Viswanadham | +| [HDFS-12889](https://issues.apache.org/jira/browse/HDFS-12889) | Router UI is missing robots.txt file | Major | . | Bharat Viswanadham | Bharat Viswanadham | +| [YARN-7607](https://issues.apache.org/jira/browse/YARN-7607) | Remove the trailing duplicated timestamp in container diagnostics message | Minor | nodemanager | Weiwei Yang | Weiwei Yang | +| [HADOOP-15080](https://issues.apache.org/jira/browse/HADOOP-15080) | Aliyun OSS: update oss sdk from 2.8.1 to 2.8.3 to remove its dependency on Cat-x "json-lib" | Blocker | fs/oss | Christopher Douglas | Sammi Chen | +| [YARN-7608](https://issues.apache.org/jira/browse/YARN-7608) | Incorrect sTarget column causing DataTable warning on RM application and scheduler web page | Major | resourcemanager, webapp | Weiwei Yang | Gergely Novák | +| [HDFS-12833](https://issues.apache.org/jira/browse/HDFS-12833) | Distcp : Update the usage of delete option for dependency with update and overwrite option | Minor | distcp, hdfs | Harshakiran Reddy | usharani | +| [YARN-7647](https://issues.apache.org/jira/browse/YARN-7647) | NM print inappropriate error log when node-labels is enabled | Minor | . | Yang Wang | Yang Wang | +| [HDFS-12907](https://issues.apache.org/jira/browse/HDFS-12907) | Allow read-only access to reserved raw for non-superusers | Major | namenode | Daryn Sharp | Rushabh Shah | +| [HDFS-12881](https://issues.apache.org/jira/browse/HDFS-12881) | Output streams closed with IOUtils suppressing write errors | Major | . | Jason Darrell Lowe | Ajay Kumar | +| [YARN-7595](https://issues.apache.org/jira/browse/YARN-7595) | Container launching code suppresses close exceptions after writes | Major | nodemanager | Jason Darrell Lowe | Jim Brennan | +| [HADOOP-15085](https://issues.apache.org/jira/browse/HADOOP-15085) | Output streams closed with IOUtils suppressing write errors | Major | . | Jason Darrell Lowe | Jim Brennan | +| [HADOOP-15123](https://issues.apache.org/jira/browse/HADOOP-15123) | KDiag tries to load krb5.conf from KRB5CCNAME instead of KRB5\_CONFIG | Minor | security | Vipin Rathor | Vipin Rathor | +| [YARN-7661](https://issues.apache.org/jira/browse/YARN-7661) | NodeManager metrics return wrong value after update node resource | Major | . | Yang Wang | Yang Wang | +| [HDFS-12347](https://issues.apache.org/jira/browse/HDFS-12347) | TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently | Critical | test | Xiao Chen | Bharat Viswanadham | +| [YARN-7662](https://issues.apache.org/jira/browse/YARN-7662) | [Atsv2] Define new set of configurations for reader and collectors to bind. | Major | . | Rohith Sharma K S | Rohith Sharma K S | +| [YARN-7674](https://issues.apache.org/jira/browse/YARN-7674) | Update Timeline Reader web app address in UI2 | Major | . | Rohith Sharma K S | Sunil G | +| [YARN-7542](https://issues.apache.org/jira/browse/YARN-7542) | Fix issue that causes some Running Opportunistic Containers to be recovered as PAUSED | Major | . | Arun Suresh | Sampada Dehankar | +| [HADOOP-15143](https://issues.apache.org/jira/browse/HADOOP-15143) | NPE due to Invalid KerberosTicket in UGI | Major | . | Jitendra Nath Pandey | Mukul Kumar Singh | +| [YARN-7692](https://issues.apache.org/jira/browse/YARN-7692) | Skip validating priority acls while recovering applications | Blocker | resourcemanager | Charan Hebri | Sunil G | +| [MAPREDUCE-7028](https://issues.apache.org/jira/browse/MAPREDUCE-7028) | Concurrent task progress updates causing NPE in Application Master | Blocker | mr-am | Gergo Repas | Gergo Repas | +| [YARN-7619](https://issues.apache.org/jira/browse/YARN-7619) | Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user | Major | capacity scheduler, yarn | Eric Payne | Eric Payne | +| [YARN-7699](https://issues.apache.org/jira/browse/YARN-7699) | queueUsagePercentage is coming as INF for getApp REST api call | Major | webapp | Sunil G | Sunil G | +| [HDFS-12985](https://issues.apache.org/jira/browse/HDFS-12985) | NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deleted | Major | hdfs | Manoj Govindassamy | Manoj Govindassamy | +| [YARN-4227](https://issues.apache.org/jira/browse/YARN-4227) | Ignore expired containers from removed nodes in FairScheduler | Critical | fairscheduler | Wilfred Spiegelenburg | Wilfred Spiegelenburg | +| [YARN-7508](https://issues.apache.org/jira/browse/YARN-7508) | NPE in FiCaSchedulerApp when debug log enabled in async-scheduling mode | Major | capacityscheduler | Tao Yang | Tao Yang | +| [YARN-7663](https://issues.apache.org/jira/browse/YARN-7663) | RMAppImpl:Invalid event: START at KILLED | Major | resourcemanager | lujie | lujie | +| [YARN-6948](https://issues.apache.org/jira/browse/YARN-6948) | Invalid event: ATTEMPT\_ADDED at FINAL\_SAVING | Major | yarn | lujie | lujie | +| [HADOOP-15060](https://issues.apache.org/jira/browse/HADOOP-15060) | TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime flaky | Major | . | Miklos Szegedi | Miklos Szegedi | +| [YARN-7735](https://issues.apache.org/jira/browse/YARN-7735) | Fix typo in YARN documentation | Minor | documentation | Takanobu Asanuma | Takanobu Asanuma | +| [YARN-7727](https://issues.apache.org/jira/browse/YARN-7727) | Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector | Minor | yarn | Prabhu Joseph | Prabhu Joseph | +| [HDFS-11915](https://issues.apache.org/jira/browse/HDFS-11915) | Sync rbw dir on the first hsync() to avoid file lost on power failure | Critical | . | Kanaka Kumar Avvaru | Vinayakumar B | +| [YARN-7705](https://issues.apache.org/jira/browse/YARN-7705) | Create the container log directory with correct sticky bit in C code | Major | nodemanager | Yufei Gu | Yufei Gu | +| [HDFS-9049](https://issues.apache.org/jira/browse/HDFS-9049) | Make Datanode Netty reverse proxy port to be configurable | Major | datanode | Vinayakumar B | Vinayakumar B | +| [YARN-7758](https://issues.apache.org/jira/browse/YARN-7758) | Add an additional check to the validity of container and application ids passed to container-executor | Major | nodemanager | Miklos Szegedi | Yufei Gu | +| [HADOOP-15150](https://issues.apache.org/jira/browse/HADOOP-15150) | in FsShell, UGI params should be overidden through env vars(-D arg) | Major | . | Brahma Reddy Battula | Brahma Reddy Battula | +| [HADOOP-15181](https://issues.apache.org/jira/browse/HADOOP-15181) | Typo in SecureMode.md | Trivial | documentation | Masahiro Tanaka | Masahiro Tanaka | +| [YARN-7806](https://issues.apache.org/jira/browse/YARN-7806) | Distributed Shell should use timeline async api's | Major | distributed-shell | Sumana Sathish | Rohith Sharma K S | +| [HADOOP-15121](https://issues.apache.org/jira/browse/HADOOP-15121) | Encounter NullPointerException when using DecayRpcScheduler | Major | . | Tao Jie | Tao Jie | +| [MAPREDUCE-7015](https://issues.apache.org/jira/browse/MAPREDUCE-7015) | Possible race condition in JHS if the job is not loaded | Major | jobhistoryserver | Peter Bacsko | Peter Bacsko | +| [YARN-7737](https://issues.apache.org/jira/browse/YARN-7737) | prelaunch.err file not found exception on container failure | Major | . | Jonathan Hung | Keqiu Hu | +| [HDFS-13063](https://issues.apache.org/jira/browse/HDFS-13063) | Fix the incorrect spelling in HDFSHighAvailabilityWithQJM.md | Trivial | documentation | Jianfei Jiang | Jianfei Jiang | +| [YARN-7102](https://issues.apache.org/jira/browse/YARN-7102) | NM heartbeat stuck when responseId overflows MAX\_INT | Critical | . | Botong Huang | Botong Huang | +| [MAPREDUCE-7041](https://issues.apache.org/jira/browse/MAPREDUCE-7041) | MR should not try to clean up at first job attempt | Major | . | Takanobu Asanuma | Gergo Repas | +| [MAPREDUCE-7020](https://issues.apache.org/jira/browse/MAPREDUCE-7020) | Task timeout in uber mode can crash AM | Major | mr-am | Akira Ajisaka | Peter Bacsko | +| [YARN-7765](https://issues.apache.org/jira/browse/YARN-7765) | [Atsv2] GSSException: No valid credentials provided - Failed to find any Kerberos tgt thrown by Timelinev2Client & HBaseClient in NM | Blocker | . | Sumana Sathish | Rohith Sharma K S | +| [HDFS-12974](https://issues.apache.org/jira/browse/HDFS-12974) | Exception message is not printed when creating an encryption zone fails with AuthorizationException | Minor | encryption | fang zhenyi | fang zhenyi | +| [YARN-7698](https://issues.apache.org/jira/browse/YARN-7698) | A misleading variable's name in ApplicationAttemptEventDispatcher | Minor | resourcemanager | Jinjiang Ling | Jinjiang Ling | +| [HDFS-12528](https://issues.apache.org/jira/browse/HDFS-12528) | Add an option to not disable short-circuit reads on failures | Major | hdfs-client, performance | Andre Araujo | Xiao Chen | +| [HDFS-13100](https://issues.apache.org/jira/browse/HDFS-13100) | Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs. | Critical | hdfs, webhdfs | Yongjun Zhang | Yongjun Zhang | +| [YARN-7849](https://issues.apache.org/jira/browse/YARN-7849) | TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error | Major | test | Jason Darrell Lowe | Botong Huang | +| [YARN-7801](https://issues.apache.org/jira/browse/YARN-7801) | AmFilterInitializer should addFilter after fill all parameters | Critical | . | Sumana Sathish | Wangda Tan | +| [YARN-7890](https://issues.apache.org/jira/browse/YARN-7890) | NPE during container relaunch | Major | . | Billie Rinaldi | Jason Darrell Lowe | +| [HDFS-13115](https://issues.apache.org/jira/browse/HDFS-13115) | In getNumUnderConstructionBlocks(), ignore the inodeIds for which the inodes have been deleted | Major | . | Yongjun Zhang | Yongjun Zhang | +| [HDFS-12935](https://issues.apache.org/jira/browse/HDFS-12935) | Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up | Major | tools | Jianfei Jiang | Jianfei Jiang | +| [HDFS-13120](https://issues.apache.org/jira/browse/HDFS-13120) | Snapshot diff could be corrupted after concat | Major | namenode, snapshots | Xiaoyu Yao | Xiaoyu Yao | +| [HDFS-10453](https://issues.apache.org/jira/browse/HDFS-10453) | ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster. | Major | namenode | Xiaoqiao He | Xiaoqiao He | +| [HDFS-8693](https://issues.apache.org/jira/browse/HDFS-8693) | refreshNamenodes does not support adding a new standby to a running DN | Critical | datanode, ha | Jian Fang | Ajith S | +| [MAPREDUCE-7052](https://issues.apache.org/jira/browse/MAPREDUCE-7052) | TestFixedLengthInputFormat#testFormatCompressedIn is flaky | Major | client, test | Peter Bacsko | Peter Bacsko | +| [HDFS-13112](https://issues.apache.org/jira/browse/HDFS-13112) | Token expiration edits may cause log corruption or deadlock | Critical | namenode | Daryn Sharp | Daryn Sharp | +| [MAPREDUCE-7053](https://issues.apache.org/jira/browse/MAPREDUCE-7053) | Timed out tasks can fail to produce thread dump | Major | . | Jason Darrell Lowe | Jason Darrell Lowe | +| [HADOOP-15206](https://issues.apache.org/jira/browse/HADOOP-15206) | BZip2 drops and duplicates records when input split size is small | Major | . | Aki Tanaka | Aki Tanaka | +| [YARN-7937](https://issues.apache.org/jira/browse/YARN-7937) | Fix http method name in Cluster Application Timeout Update API example request | Minor | docs, documentation | Charan Hebri | Charan Hebri | +| [YARN-7947](https://issues.apache.org/jira/browse/YARN-7947) | Capacity Scheduler intra-queue preemption can NPE for non-schedulable apps | Major | capacity scheduler, scheduler preemption | Eric Payne | Eric Payne | +| [YARN-7945](https://issues.apache.org/jira/browse/YARN-7945) | Java Doc error in UnmanagedAMPoolManager for branch-2 | Major | . | Rohith Sharma K S | Botong Huang | +| [HADOOP-14903](https://issues.apache.org/jira/browse/HADOOP-14903) | Add json-smart explicitly to pom.xml | Major | common | Ray Chiang | Ray Chiang | +| [HADOOP-15236](https://issues.apache.org/jira/browse/HADOOP-15236) | Fix typo in RequestHedgingProxyProvider and RequestHedgingRMFailoverProxyProvider | Trivial | documentation | Akira Ajisaka | Gabor Bota | +| [MAPREDUCE-7027](https://issues.apache.org/jira/browse/MAPREDUCE-7027) | HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails | Critical | harchive | Gergely Novák | Gergely Novák | +| [HDFS-12781](https://issues.apache.org/jira/browse/HDFS-12781) | After Datanode down, In Namenode UI Datanode tab is throwing warning message. | Major | datanode | Harshakiran Reddy | Brahma Reddy Battula | +| [HDFS-12070](https://issues.apache.org/jira/browse/HDFS-12070) | Failed block recovery leaves files open indefinitely and at risk for data loss | Major | . | Daryn Sharp | Kihwal Lee | +| [HADOOP-15251](https://issues.apache.org/jira/browse/HADOOP-15251) | Backport HADOOP-13514 (surefire upgrade) to branch-2 | Major | test | Christopher Douglas | Christopher Douglas | +| [HDFS-13194](https://issues.apache.org/jira/browse/HDFS-13194) | CachePool permissions incorrectly checked | Major | . | Yiqun Lin | Jianfei Jiang | +| [HADOOP-15276](https://issues.apache.org/jira/browse/HADOOP-15276) | branch-2 site not building after ADL troubleshooting doc added | Major | documentation | Steve Loughran | Steve Loughran | +| [YARN-7835](https://issues.apache.org/jira/browse/YARN-7835) | [Atsv2] Race condition in NM while publishing events if second attempt is launched on the same node | Critical | . | Rohith Sharma K S | Rohith Sharma K S | +| [HADOOP-15275](https://issues.apache.org/jira/browse/HADOOP-15275) | Incorrect javadoc for return type of RetryPolicy#shouldRetry | Minor | documentation | Nanda kumar | Nanda kumar | +| [YARN-7511](https://issues.apache.org/jira/browse/YARN-7511) | NPE in ContainerLocalizer when localization failed for running container | Major | nodemanager | Tao Yang | Tao Yang | +| [MAPREDUCE-7023](https://issues.apache.org/jira/browse/MAPREDUCE-7023) | TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun | Minor | test | Gergely Novák | Gergely Novák | +| [HADOOP-15283](https://issues.apache.org/jira/browse/HADOOP-15283) | Upgrade from findbugs 3.0.1 to spotbugs 3.1.2 in branch-2 to fix docker image build | Major | . | Xiao Chen | Akira Ajisaka | +| [HADOOP-15286](https://issues.apache.org/jira/browse/HADOOP-15286) | Remove unused imports from TestKMSWithZK.java | Minor | test | Akira Ajisaka | Ajay Kumar | +| [HDFS-13040](https://issues.apache.org/jira/browse/HDFS-13040) | Kerberized inotify client fails despite kinit properly | Major | namenode | Wei-Chiu Chuang | Xiao Chen | +| [YARN-7736](https://issues.apache.org/jira/browse/YARN-7736) | Fix itemization in YARN federation document | Minor | documentation | Akira Ajisaka | Sen Zhao | +| [HDFS-13164](https://issues.apache.org/jira/browse/HDFS-13164) | File not closed if streamer fail with DSQuotaExceededException | Major | hdfs-client | Xiao Chen | Xiao Chen | +| [HDFS-13109](https://issues.apache.org/jira/browse/HDFS-13109) | Support fully qualified hdfs path in EZ commands | Major | hdfs | Hanisha Koneru | Hanisha Koneru | +| [MAPREDUCE-6930](https://issues.apache.org/jira/browse/MAPREDUCE-6930) | mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml | Major | mrv2 | Daniel Templeton | Sen Zhao | +| [HDFS-10618](https://issues.apache.org/jira/browse/HDFS-10618) | TestPendingReconstruction#testPendingAndInvalidate is flaky due to race condition | Major | . | Eric Badger | Eric Badger | +| [HDFS-10803](https://issues.apache.org/jira/browse/HDFS-10803) | TestBalancerWithMultipleNameNodes#testBalancing2OutOf3Blockpools fails intermittently due to no free space available | Major | . | Yiqun Lin | Yiqun Lin | +| [HDFS-12156](https://issues.apache.org/jira/browse/HDFS-12156) | TestFSImage fails without -Pnative | Major | test | Akira Ajisaka | Akira Ajisaka | +| [HDFS-13261](https://issues.apache.org/jira/browse/HDFS-13261) | Fix incorrect null value check | Minor | hdfs | Jianfei Jiang | Jianfei Jiang | +| [HDFS-12886](https://issues.apache.org/jira/browse/HDFS-12886) | Ignore minReplication for block recovery | Major | hdfs, namenode | Lukas Majercak | Lukas Majercak | +| [YARN-8039](https://issues.apache.org/jira/browse/YARN-8039) | Clean up log dir configuration in TestLinuxContainerExecutorWithMocks.testStartLocalizer | Minor | . | Miklos Szegedi | Miklos Szegedi | +| [HDFS-13296](https://issues.apache.org/jira/browse/HDFS-13296) | GenericTestUtils generates paths with drive letter in Windows and fail webhdfs related test cases | Major | . | Xiao Liang | Xiao Liang | +| [HDFS-13268](https://issues.apache.org/jira/browse/HDFS-13268) | TestWebHdfsFileContextMainOperations fails on Windows | Major | . | Íñigo Goiri | Xiao Liang | +| [YARN-8054](https://issues.apache.org/jira/browse/YARN-8054) | Improve robustness of the LocalDirsHandlerService MonitoringTimerTask thread | Major | . | Jonathan Turner Eagles | Jonathan Turner Eagles | +| [YARN-7873](https://issues.apache.org/jira/browse/YARN-7873) | Revert YARN-6078 | Blocker | . | Billie Rinaldi | Billie Rinaldi | +| [HDFS-13195](https://issues.apache.org/jira/browse/HDFS-13195) | DataNode conf page cannot display the current value after reconfig | Minor | datanode | maobaolong | maobaolong | +| [YARN-8063](https://issues.apache.org/jira/browse/YARN-8063) | DistributedShellTimelinePlugin wrongly check for entityId instead of entityType | Major | . | Rohith Sharma K S | Rohith Sharma K S | +| [YARN-8068](https://issues.apache.org/jira/browse/YARN-8068) | Application Priority field causes NPE in app timeline publish when Hadoop 2.7 based clients to 2.8+ | Blocker | yarn | Sunil G | Sunil G | +| [HADOOP-12862](https://issues.apache.org/jira/browse/HADOOP-12862) | LDAP Group Mapping over SSL can not specify trust store | Major | . | Wei-Chiu Chuang | Wei-Chiu Chuang | +| [HADOOP-15317](https://issues.apache.org/jira/browse/HADOOP-15317) | Improve NetworkTopology chooseRandom's loop | Major | . | Xiao Chen | Xiao Chen | +| [HADOOP-15355](https://issues.apache.org/jira/browse/HADOOP-15355) | TestCommonConfigurationFields is broken by HADOOP-15312 | Major | test | Konstantin Shvachko | LiXin Ge | +| [HDFS-13176](https://issues.apache.org/jira/browse/HDFS-13176) | WebHdfs file path gets truncated when having semicolon (;) inside | Major | webhdfs | Zsolt Venczel | Zsolt Venczel | +| [HADOOP-15375](https://issues.apache.org/jira/browse/HADOOP-15375) | Branch-2 pre-commit failed to build docker image | Major | . | Xiao Chen | Xiao Chen | +| [HADOOP-15357](https://issues.apache.org/jira/browse/HADOOP-15357) | Configuration.getPropsWithPrefix no longer does variable substitution | Major | . | Jim Brennan | Jim Brennan | +| [MAPREDUCE-7062](https://issues.apache.org/jira/browse/MAPREDUCE-7062) | Update mapreduce.job.tags description for making use for ATSv2 purpose. | Major | . | Charan Hebri | Charan Hebri | +| [YARN-8073](https://issues.apache.org/jira/browse/YARN-8073) | TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration | Major | . | Rohith Sharma K S | Rohith Sharma K S | +| [YARN-6629](https://issues.apache.org/jira/browse/YARN-6629) | NPE occurred when container allocation proposal is applied but its resource requests are removed before | Critical | . | Tao Yang | Tao Yang | +| [HDFS-13427](https://issues.apache.org/jira/browse/HDFS-13427) | Fix the section titles of transparent encryption document | Minor | documentation | Akira Ajisaka | Akira Ajisaka | +| [YARN-7527](https://issues.apache.org/jira/browse/YARN-7527) | Over-allocate node resource in async-scheduling mode of CapacityScheduler | Major | capacityscheduler | Tao Yang | Tao Yang | +| [HDFS-7101](https://issues.apache.org/jira/browse/HDFS-7101) | Potential null dereference in DFSck#doWork() | Minor | . | Ted Yu | skrho | +| [YARN-8120](https://issues.apache.org/jira/browse/YARN-8120) | JVM can crash with SIGSEGV when exiting due to custom leveldb logger | Major | nodemanager, resourcemanager | Jason Darrell Lowe | Jason Darrell Lowe | +| [YARN-8147](https://issues.apache.org/jira/browse/YARN-8147) | TestClientRMService#testGetApplications sporadically fails | Major | test | Jason Darrell Lowe | Jason Darrell Lowe | +| [HADOOP-14970](https://issues.apache.org/jira/browse/HADOOP-14970) | MiniHadoopClusterManager doesn't respect lack of format option | Minor | . | Erik Krogen | Erik Krogen | +| [YARN-8156](https://issues.apache.org/jira/browse/YARN-8156) | Increase the default value of yarn.timeline-service.app-collector.linger-period.ms | Major | . | Rohith Sharma K S | Charan Hebri | +| [YARN-8165](https://issues.apache.org/jira/browse/YARN-8165) | Incorrect queue name logging in AbstractContainerAllocator | Trivial | capacityscheduler | Weiwei Yang | Weiwei Yang | +| [YARN-8164](https://issues.apache.org/jira/browse/YARN-8164) | Fix a potential NPE in AbstractSchedulerPlanFollower | Major | . | lujie | lujie | +| [HDFS-12828](https://issues.apache.org/jira/browse/HDFS-12828) | OIV ReverseXML Processor fails with escaped characters | Critical | hdfs | Erik Krogen | Erik Krogen | +| [HADOOP-15180](https://issues.apache.org/jira/browse/HADOOP-15180) | branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out file | Minor | scripts | Ranith Sardar | Ranith Sardar | +| [HADOOP-15396](https://issues.apache.org/jira/browse/HADOOP-15396) | Some java source files are executable | Minor | . | Akira Ajisaka | Shashikant Banerjee | +| [YARN-6827](https://issues.apache.org/jira/browse/YARN-6827) | [ATS1/1.5] NPE exception while publishing recovering applications into ATS during RM restart. | Major | resourcemanager | Rohith Sharma K S | Rohith Sharma K S | +| [YARN-7786](https://issues.apache.org/jira/browse/YARN-7786) | NullPointerException while launching ApplicationMaster | Major | . | lujie | lujie | +| [HDFS-13408](https://issues.apache.org/jira/browse/HDFS-13408) | MiniDFSCluster to support being built on randomized base directory | Major | test | Xiao Liang | Xiao Liang | +| [HADOOP-15390](https://issues.apache.org/jira/browse/HADOOP-15390) | Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens | Critical | . | Xiao Chen | Xiao Chen | +| [HDFS-13336](https://issues.apache.org/jira/browse/HDFS-13336) | Test cases of TestWriteToReplica failed in windows | Major | . | Xiao Liang | Xiao Liang | +| [YARN-7598](https://issues.apache.org/jira/browse/YARN-7598) | Document how to use classpath isolation for aux-services in YARN | Major | . | Xuan Gong | Xuan Gong | +| [YARN-8183](https://issues.apache.org/jira/browse/YARN-8183) | Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap | Critical | yarn | Sumana Sathish | Suma Shivaprasad | +| [HADOOP-15385](https://issues.apache.org/jira/browse/HADOOP-15385) | Many tests are failing in hadoop-distcp project in branch-2 | Critical | tools/distcp | Rushabh Shah | Jason Darrell Lowe | +| [MAPREDUCE-7042](https://issues.apache.org/jira/browse/MAPREDUCE-7042) | Killed MR job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled | Major | . | Yesha Vora | Xuan Gong | +| [YARN-8205](https://issues.apache.org/jira/browse/YARN-8205) | Application State is not updated to ATS if AM launching is delayed. | Critical | . | Sumana Sathish | Rohith Sharma K S | +| [MAPREDUCE-7072](https://issues.apache.org/jira/browse/MAPREDUCE-7072) | mapred job -history prints duplicate counter in human output | Major | client | Wilfred Spiegelenburg | Wilfred Spiegelenburg | +| [YARN-8221](https://issues.apache.org/jira/browse/YARN-8221) | RMWebServices also need to honor yarn.resourcemanager.display.per-user-apps | Major | webapp | Sunil G | Sunil G | +| [HDFS-13509](https://issues.apache.org/jira/browse/HDFS-13509) | Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows | Major | . | Xiao Liang | Xiao Liang | +| [MAPREDUCE-7073](https://issues.apache.org/jira/browse/MAPREDUCE-7073) | Optimize TokenCache#obtainTokensForNamenodesInternal | Major | . | Bibin Chundatt | Bibin Chundatt | +| [YARN-8025](https://issues.apache.org/jira/browse/YARN-8025) | UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty | Major | yarn | Jiandan Yang | Tao Yang | +| [YARN-8232](https://issues.apache.org/jira/browse/YARN-8232) | RMContainer lost queue name when RM HA happens | Major | resourcemanager | Hu Ziqian | Hu Ziqian | +| [HDFS-13537](https://issues.apache.org/jira/browse/HDFS-13537) | TestHdfsHelper does not generate jceks path properly for relative path in Windows | Major | . | Xiao Liang | Xiao Liang | +| [HADOOP-15446](https://issues.apache.org/jira/browse/HADOOP-15446) | WASB: PageBlobInputStream.skip breaks HBASE replication | Major | fs/azure | Thomas Marqardt | Thomas Marqardt | +| [YARN-8244](https://issues.apache.org/jira/browse/YARN-8244) | TestContainerSchedulerQueuing.testStartMultipleContainers failed | Major | . | Miklos Szegedi | Jim Brennan | +| [HDFS-13586](https://issues.apache.org/jira/browse/HDFS-13586) | Fsync fails on directories on Windows | Critical | datanode, hdfs | Lukas Majercak | Lukas Majercak | +| [HDFS-13590](https://issues.apache.org/jira/browse/HDFS-13590) | Backport HDFS-12378 to branch-2 | Major | datanode, hdfs, test | Lukas Majercak | Lukas Majercak | +| [HADOOP-15478](https://issues.apache.org/jira/browse/HADOOP-15478) | WASB: hflush() and hsync() regression | Major | fs/azure | Thomas Marqardt | Thomas Marqardt | +| [HADOOP-15450](https://issues.apache.org/jira/browse/HADOOP-15450) | Avoid fsync storm triggered by DiskChecker and handle disk full situation | Blocker | . | Kihwal Lee | Arpit Agarwal | +| [HDFS-13601](https://issues.apache.org/jira/browse/HDFS-13601) | Optimize ByteString conversions in PBHelper | Major | . | Andrew Wang | Andrew Wang | +| [HDFS-13588](https://issues.apache.org/jira/browse/HDFS-13588) | Fix TestFsDatasetImpl test failures on Windows | Major | . | Xiao Liang | Xiao Liang | +| [YARN-8310](https://issues.apache.org/jira/browse/YARN-8310) | Handle old NMTokenIdentifier, AMRMTokenIdentifier, and ContainerTokenIdentifier formats | Major | . | Robert Kanter | Robert Kanter | +| [YARN-8344](https://issues.apache.org/jira/browse/YARN-8344) | Missing nm.stop() in TestNodeManagerResync to fix testKillContainersOnResync | Major | . | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola | +| [YARN-8327](https://issues.apache.org/jira/browse/YARN-8327) | Fix TestAggregatedLogFormat#testReadAcontainerLogs1 on Windows | Major | log-aggregation | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola | +| [YARN-8346](https://issues.apache.org/jira/browse/YARN-8346) | Upgrading to 3.1 kills running containers with error "Opportunistic container queue is full" | Blocker | . | Rohith Sharma K S | Jason Darrell Lowe | +| [HDFS-13611](https://issues.apache.org/jira/browse/HDFS-13611) | Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient | Major | . | Andrew Wang | Andrew Wang | +| [HDFS-13618](https://issues.apache.org/jira/browse/HDFS-13618) | Fix TestDataNodeFaultInjector test failures on Windows | Major | test | Xiao Liang | Xiao Liang | +| [HADOOP-15473](https://issues.apache.org/jira/browse/HADOOP-15473) | Configure serialFilter in KeyProvider to avoid UnrecoverableKeyException caused by JDK-8189997 | Critical | kms | Gabor Bota | Gabor Bota | +| [HDFS-13626](https://issues.apache.org/jira/browse/HDFS-13626) | Fix incorrect username when deny the setOwner operation | Minor | namenode | luhuachao | Zsolt Venczel | +| [MAPREDUCE-7103](https://issues.apache.org/jira/browse/MAPREDUCE-7103) | Fix TestHistoryViewerPrinter on windows due to a mismatch line separator | Minor | . | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola | +| [YARN-8359](https://issues.apache.org/jira/browse/YARN-8359) | Exclude containermanager.linux test classes on Windows | Major | . | Giovanni Matteo Fumarola | Jason Darrell Lowe | +| [HDFS-13664](https://issues.apache.org/jira/browse/HDFS-13664) | Refactor ConfiguredFailoverProxyProvider to make inheritance easier | Minor | hdfs-client | Chao Sun | Chao Sun | +| [YARN-8405](https://issues.apache.org/jira/browse/YARN-8405) | RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773 | Major | . | Rohith Sharma K S | Íñigo Goiri | +| [MAPREDUCE-7108](https://issues.apache.org/jira/browse/MAPREDUCE-7108) | TestFileOutputCommitter fails on Windows | Minor | test | Zuoming Zhang | Zuoming Zhang | +| [MAPREDUCE-7101](https://issues.apache.org/jira/browse/MAPREDUCE-7101) | Add config parameter to allow JHS to alway scan user dir irrespective of modTime | Critical | . | Wangda Tan | Thomas Marqardt | +| [YARN-8404](https://issues.apache.org/jira/browse/YARN-8404) | Timeline event publish need to be async to avoid Dispatcher thread leak in case ATS is down | Blocker | . | Rohith Sharma K S | Rohith Sharma K S | +| [HDFS-13675](https://issues.apache.org/jira/browse/HDFS-13675) | Speed up TestDFSAdminWithHA | Major | hdfs, namenode | Lukas Majercak | Lukas Majercak | +| [HDFS-13673](https://issues.apache.org/jira/browse/HDFS-13673) | TestNameNodeMetrics fails on Windows | Minor | test | Zuoming Zhang | Zuoming Zhang | +| [HDFS-13676](https://issues.apache.org/jira/browse/HDFS-13676) | TestEditLogRace fails on Windows | Minor | test | Zuoming Zhang | Zuoming Zhang | +| [HADOOP-15523](https://issues.apache.org/jira/browse/HADOOP-15523) | Shell command timeout given is in seconds whereas it is taken as millisec while scheduling | Major | . | Bilwa S T | Bilwa S T | +| [YARN-8444](https://issues.apache.org/jira/browse/YARN-8444) | NodeResourceMonitor crashes on bad swapFree value | Major | . | Jim Brennan | Jim Brennan | +| [YARN-8457](https://issues.apache.org/jira/browse/YARN-8457) | Compilation is broken with -Pyarn-ui | Major | webapp | Sunil G | Sunil G | +| [YARN-8401](https://issues.apache.org/jira/browse/YARN-8401) | [UI2] new ui is not accessible with out internet connection | Blocker | . | Bibin Chundatt | Bibin Chundatt | +| [YARN-8451](https://issues.apache.org/jira/browse/YARN-8451) | Multiple NM heartbeat thread created when a slow NM resync with RM | Major | nodemanager | Botong Huang | Botong Huang | +| [HADOOP-15548](https://issues.apache.org/jira/browse/HADOOP-15548) | Randomize local dirs | Minor | . | Jim Brennan | Jim Brennan | +| [YARN-8473](https://issues.apache.org/jira/browse/YARN-8473) | Containers being launched as app tears down can leave containers in NEW state | Major | nodemanager | Jason Darrell Lowe | Jason Darrell Lowe | +| [HDFS-13729](https://issues.apache.org/jira/browse/HDFS-13729) | Fix broken links to RBF documentation | Minor | documentation | jwhitter | Gabor Bota | +| [YARN-8515](https://issues.apache.org/jira/browse/YARN-8515) | container-executor can crash with SIGPIPE after nodemanager restart | Major | . | Jim Brennan | Jim Brennan | +| [YARN-8421](https://issues.apache.org/jira/browse/YARN-8421) | when moving app, activeUsers is increased, even though app does not have outstanding request | Major | . | kyungwan nam | | +| [HADOOP-15614](https://issues.apache.org/jira/browse/HADOOP-15614) | TestGroupsCaching.testExceptionOnBackgroundRefreshHandled reliably fails | Major | . | Kihwal Lee | Weiwei Yang | +| [YARN-4606](https://issues.apache.org/jira/browse/YARN-4606) | CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps | Critical | capacity scheduler, capacityscheduler | Karam Singh | Manikandan R | +| [HADOOP-15637](https://issues.apache.org/jira/browse/HADOOP-15637) | LocalFs#listLocatedStatus does not filter out hidden .crc files | Minor | fs | Erik Krogen | Erik Krogen | +| [HADOOP-15644](https://issues.apache.org/jira/browse/HADOOP-15644) | Hadoop Docker Image Pip Install Fails on branch-2 | Critical | build | Haibo Chen | Haibo Chen | +| [YARN-6966](https://issues.apache.org/jira/browse/YARN-6966) | NodeManager metrics may return wrong negative values when NM restart | Major | . | Yang Wang | Szilard Nemeth | +| [YARN-8331](https://issues.apache.org/jira/browse/YARN-8331) | Race condition in NM container launched after done | Major | . | Yang Wang | Pradeep Ambati | +| [HDFS-13758](https://issues.apache.org/jira/browse/HDFS-13758) | DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction | Major | namenode | Wei-Chiu Chuang | chencan | +| [YARN-8612](https://issues.apache.org/jira/browse/YARN-8612) | Fix NM Collector Service Port issue in YarnConfiguration | Major | ATSv2 | Prabha Manepalli | Prabha Manepalli | +| [HADOOP-15674](https://issues.apache.org/jira/browse/HADOOP-15674) | Test failure TestSSLHttpServer.testExcludedCiphers with TLS\_ECDHE\_RSA\_WITH\_AES\_128\_CBC\_SHA256 cipher suite | Major | common | Gabor Bota | Szilard Nemeth | +| [YARN-8640](https://issues.apache.org/jira/browse/YARN-8640) | Restore previous state in container-executor after failure | Major | . | Jim Brennan | Jim Brennan | +| [YARN-8679](https://issues.apache.org/jira/browse/YARN-8679) | [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked | Major | . | Rohith Sharma K S | Wangda Tan | +| [HADOOP-14314](https://issues.apache.org/jira/browse/HADOOP-14314) | The OpenSolaris taxonomy link is dead in InterfaceClassification.md | Major | documentation | Daniel Templeton | Rui Gao | +| [YARN-8649](https://issues.apache.org/jira/browse/YARN-8649) | NPE in localizer hearbeat processing if a container is killed while localizing | Major | . | lujie | lujie | +| [HADOOP-10219](https://issues.apache.org/jira/browse/HADOOP-10219) | ipc.Client.setupIOstreams() needs to check for ClientCache.stopClient requested shutdowns | Major | ipc | Steve Loughran | Kihwal Lee | +| [MAPREDUCE-7131](https://issues.apache.org/jira/browse/MAPREDUCE-7131) | Job History Server has race condition where it moves files from intermediate to finished but thinks file is in intermediate | Major | . | Anthony Hsu | Anthony Hsu | +| [HDFS-13836](https://issues.apache.org/jira/browse/HDFS-13836) | RBF: Handle mount table znode with null value | Major | federation, hdfs | yanghuafeng | yanghuafeng | +| [HDFS-12716](https://issues.apache.org/jira/browse/HDFS-12716) | 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available | Major | datanode | usharani | Ranith Sardar | +| [YARN-8709](https://issues.apache.org/jira/browse/YARN-8709) | CS preemption monitor always fails since one under-served queue was deleted | Major | capacityscheduler, scheduler preemption | Tao Yang | Tao Yang | +| [HDFS-13051](https://issues.apache.org/jira/browse/HDFS-13051) | Fix dead lock during async editlog rolling if edit queue is full | Major | namenode | zhangwei | Daryn Sharp | +| [HDFS-13914](https://issues.apache.org/jira/browse/HDFS-13914) | Fix DN UI logs link broken when https is enabled after HDFS-13902 | Minor | datanode | Jianfei Jiang | Jianfei Jiang | +| [MAPREDUCE-7133](https://issues.apache.org/jira/browse/MAPREDUCE-7133) | History Server task attempts REST API returns invalid data | Major | jobhistoryserver | Oleksandr Shevchenko | Oleksandr Shevchenko | +| [YARN-8720](https://issues.apache.org/jira/browse/YARN-8720) | CapacityScheduler does not enforce max resource allocation check at queue level | Major | capacity scheduler, capacityscheduler, resourcemanager | Tarun Parimi | Tarun Parimi | +| [HDFS-13844](https://issues.apache.org/jira/browse/HDFS-13844) | Fix the fmt\_bytes function in the dfs-dust.js | Minor | hdfs, ui | yanghuafeng | yanghuafeng | +| [HADOOP-15755](https://issues.apache.org/jira/browse/HADOOP-15755) | StringUtils#createStartupShutdownMessage throws NPE when args is null | Major | . | Lokesh Jain | Dinesh Chitlangia | +| [MAPREDUCE-3801](https://issues.apache.org/jira/browse/MAPREDUCE-3801) | org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently | Major | mrv2 | Robert Joseph Evans | Jason Darrell Lowe | +| [MAPREDUCE-7137](https://issues.apache.org/jira/browse/MAPREDUCE-7137) | MRAppBenchmark.benchmark1() fails with NullPointerException | Minor | test | Oleksandr Shevchenko | Oleksandr Shevchenko | +| [MAPREDUCE-7138](https://issues.apache.org/jira/browse/MAPREDUCE-7138) | ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler | Minor | test | Oleksandr Shevchenko | Oleksandr Shevchenko | +| [HDFS-13908](https://issues.apache.org/jira/browse/HDFS-13908) | TestDataNodeMultipleRegistrations is flaky | Major | . | Íñigo Goiri | Ayush Saxena | +| [HADOOP-15772](https://issues.apache.org/jira/browse/HADOOP-15772) | Remove the 'Path ... should be specified as a URI' warnings on startup | Major | conf | Arpit Agarwal | Ayush Saxena | +| [YARN-8804](https://issues.apache.org/jira/browse/YARN-8804) | resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues | Critical | capacityscheduler | Tao Yang | Tao Yang | +| [YARN-8774](https://issues.apache.org/jira/browse/YARN-8774) | Memory leak when CapacityScheduler allocates from reserved container with non-default label | Critical | capacityscheduler | Tao Yang | Tao Yang | +| [YARN-8840](https://issues.apache.org/jira/browse/YARN-8840) | Add missing cleanupSSLConfig() call for TestTimelineClient test | Minor | test, timelineclient | Aki Tanaka | Aki Tanaka | +| [HADOOP-15817](https://issues.apache.org/jira/browse/HADOOP-15817) | Reuse Object Mapper in KMSJSONReader | Major | kms | Jonathan Turner Eagles | Jonathan Turner Eagles | +| [HADOOP-15820](https://issues.apache.org/jira/browse/HADOOP-15820) | ZStandardDecompressor native code sets an integer field as a long | Blocker | . | Jason Darrell Lowe | Jason Darrell Lowe | +| [HDFS-13964](https://issues.apache.org/jira/browse/HDFS-13964) | RBF: TestRouterWebHDFSContractAppend fails with No Active Namenode under nameservice | Major | . | Ayush Saxena | Ayush Saxena | +| [HDFS-13768](https://issues.apache.org/jira/browse/HDFS-13768) | Adding replicas to volume map makes DataNode start slowly | Major | . | Yiqun Lin | Surendra Singh Lilhore | +| [HADOOP-15818](https://issues.apache.org/jira/browse/HADOOP-15818) | Fix deprecated maven-surefire-plugin configuration in hadoop-kms module | Minor | kms | Akira Ajisaka | Vidura Bhathiya Mudalige | +| [HDFS-13802](https://issues.apache.org/jira/browse/HDFS-13802) | RBF: Remove FSCK from Router Web UI | Major | . | Fei Hui | Fei Hui | +| [HADOOP-15859](https://issues.apache.org/jira/browse/HADOOP-15859) | ZStandardDecompressor.c mistakes a class for an instance | Blocker | . | Ben Lau | Jason Darrell Lowe | +| [HADOOP-15850](https://issues.apache.org/jira/browse/HADOOP-15850) | CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0 | Critical | tools/distcp | Ted Yu | Ted Yu | +| [YARN-7502](https://issues.apache.org/jira/browse/YARN-7502) | Nodemanager restart docs should describe nodemanager supervised property | Major | documentation | Jason Darrell Lowe | Suma Shivaprasad | +| [HADOOP-15866](https://issues.apache.org/jira/browse/HADOOP-15866) | Renamed HADOOP\_SECURITY\_GROUP\_SHELL\_COMMAND\_TIMEOUT keys break compatibility | Blocker | . | Wei-Chiu Chuang | Wei-Chiu Chuang | +| [YARN-8826](https://issues.apache.org/jira/browse/YARN-8826) | Fix lingering timeline collector after serviceStop in TimelineCollectorManager | Trivial | ATSv2 | Prabha Manepalli | Prabha Manepalli | +| [HADOOP-15822](https://issues.apache.org/jira/browse/HADOOP-15822) | zstd compressor can fail with a small output buffer | Major | . | Jason Darrell Lowe | Jason Darrell Lowe | +| [HDFS-13959](https://issues.apache.org/jira/browse/HDFS-13959) | TestUpgradeDomainBlockPlacementPolicy is flaky | Major | . | Ayush Saxena | Ayush Saxena | +| [HADOOP-15899](https://issues.apache.org/jira/browse/HADOOP-15899) | Update AWS Java SDK versions in NOTICE.txt | Major | . | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-15900](https://issues.apache.org/jira/browse/HADOOP-15900) | Update JSch versions in LICENSE.txt | Major | . | Akira Ajisaka | Akira Ajisaka | +| [HDFS-14043](https://issues.apache.org/jira/browse/HDFS-14043) | Tolerate corrupted seen\_txid file | Major | hdfs, namenode | Lukas Majercak | Lukas Majercak | +| [YARN-8858](https://issues.apache.org/jira/browse/YARN-8858) | CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. | Major | . | Sumana Sathish | Wangda Tan | +| [HDFS-14048](https://issues.apache.org/jira/browse/HDFS-14048) | DFSOutputStream close() throws exception on subsequent call after DataNode restart | Major | hdfs-client | Erik Krogen | Erik Krogen | +| [MAPREDUCE-7156](https://issues.apache.org/jira/browse/MAPREDUCE-7156) | NullPointerException when reaching max shuffle connections | Major | mrv2 | Peter Bacsko | Peter Bacsko | +| [YARN-8233](https://issues.apache.org/jira/browse/YARN-8233) | NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null | Critical | capacityscheduler | Tao Yang | Tao Yang | +| [HADOOP-15923](https://issues.apache.org/jira/browse/HADOOP-15923) | create-release script should set max-cache-ttl as well as default-cache-ttl for gpg-agent | Blocker | build | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-15930](https://issues.apache.org/jira/browse/HADOOP-15930) | Exclude MD5 checksum files from release artifact | Critical | build | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-15925](https://issues.apache.org/jira/browse/HADOOP-15925) | The config and log of gpg-agent are removed in create-release script | Major | build | Akira Ajisaka | Dinesh Chitlangia | +| [HDFS-14056](https://issues.apache.org/jira/browse/HDFS-14056) | Fix error messages in HDFS-12716 | Minor | hdfs | Adam Antal | Ayush Saxena | +| [YARN-7794](https://issues.apache.org/jira/browse/YARN-7794) | SLSRunner is not loading timeline service jars causing failure | Blocker | scheduler-load-simulator | Sunil G | Yufei Gu | +| [HADOOP-16008](https://issues.apache.org/jira/browse/HADOOP-16008) | Fix typo in CommandsManual.md | Minor | . | Akira Ajisaka | Shweta | +| [HADOOP-15973](https://issues.apache.org/jira/browse/HADOOP-15973) | Configuration: Included properties are not cached if resource is a stream | Critical | . | Eric Payne | Eric Payne | +| [YARN-9175](https://issues.apache.org/jira/browse/YARN-9175) | Null resources check in ResourceInfo for branch-3.0 | Major | . | Jonathan Hung | Jonathan Hung | +| [HADOOP-15992](https://issues.apache.org/jira/browse/HADOOP-15992) | JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0 | Blocker | . | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-16030](https://issues.apache.org/jira/browse/HADOOP-16030) | AliyunOSS: bring fixes back from HADOOP-15671 | Blocker | fs/oss | wujinhu | wujinhu | +| [YARN-9162](https://issues.apache.org/jira/browse/YARN-9162) | Fix TestRMAdminCLI#testHelp | Major | resourcemanager, test | Ayush Saxena | Ayush Saxena | +| [YARN-8833](https://issues.apache.org/jira/browse/YARN-8833) | Avoid potential integer overflow when computing fair shares | Major | fairscheduler | liyakun | liyakun | +| [HADOOP-16016](https://issues.apache.org/jira/browse/HADOOP-16016) | TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds | Major | security, test | Jason Darrell Lowe | Akira Ajisaka | +| [HADOOP-16013](https://issues.apache.org/jira/browse/HADOOP-16013) | DecayRpcScheduler decay thread should run as a daemon | Major | ipc | Erik Krogen | Erik Krogen | +| [YARN-8747](https://issues.apache.org/jira/browse/YARN-8747) | [UI2] YARN UI2 page loading failed due to js error under some time zone configuration | Critical | webapp | collinma | collinma | +| [YARN-9210](https://issues.apache.org/jira/browse/YARN-9210) | RM nodes web page can not display node info | Blocker | yarn | Jiandan Yang | Jiandan Yang | +| [YARN-7088](https://issues.apache.org/jira/browse/YARN-7088) | Add application launch time to Resource Manager REST API | Major | . | Abdullah Yousufi | Kanwaljeet Sachdev | +| [YARN-9222](https://issues.apache.org/jira/browse/YARN-9222) | Print launchTime in ApplicationSummary | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-6616](https://issues.apache.org/jira/browse/YARN-6616) | YARN AHS shows submitTime for jobs same as startTime | Minor | . | Prabhu Joseph | Prabhu Joseph | +| [MAPREDUCE-7177](https://issues.apache.org/jira/browse/MAPREDUCE-7177) | Disable speculative execution in TestDFSIO | Major | . | Kihwal Lee | Zhaohui Xin | +| [YARN-9206](https://issues.apache.org/jira/browse/YARN-9206) | RMServerUtils does not count SHUTDOWN as an accepted state | Major | . | Kuhu Shukla | Kuhu Shukla | +| [YARN-9283](https://issues.apache.org/jira/browse/YARN-9283) | Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference | Minor | documentation | Szilard Nemeth | Adam Antal | +| [HADOOP-15813](https://issues.apache.org/jira/browse/HADOOP-15813) | Enable more reliable SSL connection reuse | Major | common | Daryn Sharp | Daryn Sharp | +| [HDFS-14314](https://issues.apache.org/jira/browse/HDFS-14314) | fullBlockReportLeaseId should be reset after registering to NN | Critical | datanode | star | star | +| [YARN-5714](https://issues.apache.org/jira/browse/YARN-5714) | ContainerExecutor does not order environment map | Trivial | nodemanager | Remi Catherinot | Remi Catherinot | +| [HADOOP-16192](https://issues.apache.org/jira/browse/HADOOP-16192) | CallQueue backoff bug fixes: doesn't perform backoff when add() is used, and doesn't update backoff when refreshed | Major | ipc | Erik Krogen | Erik Krogen | +| [HDFS-14399](https://issues.apache.org/jira/browse/HDFS-14399) | Backport HDFS-10536 to branch-2 | Critical | . | Chao Sun | Chao Sun | +| [HDFS-14407](https://issues.apache.org/jira/browse/HDFS-14407) | Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes | Minor | . | Wanqiang Ji | Wanqiang Ji | +| [HDFS-14414](https://issues.apache.org/jira/browse/HDFS-14414) | Clean up findbugs warning in branch-2 | Major | . | Wei-Chiu Chuang | Dinesh Chitlangia | +| [HADOOP-14544](https://issues.apache.org/jira/browse/HADOOP-14544) | DistCp documentation for command line options is misaligned. | Minor | documentation | Chris Nauroth | Masatake Iwasaki | +| [HDFS-10477](https://issues.apache.org/jira/browse/HDFS-10477) | Stop decommission a rack of DataNodes caused NameNode fail over to standby | Major | namenode | yunjiong zhao | yunjiong zhao | +| [HDFS-13677](https://issues.apache.org/jira/browse/HDFS-13677) | Dynamic refresh Disk configuration results in overwriting VolumeMap | Blocker | . | xuzq | xuzq | +| [YARN-9285](https://issues.apache.org/jira/browse/YARN-9285) | RM UI progress column is of wrong type | Minor | yarn | Ahmed Hussein | Ahmed Hussein | +| [HDFS-14500](https://issues.apache.org/jira/browse/HDFS-14500) | NameNode StartupProgress continues to report edit log segments after the LOADING\_EDITS phase is finished | Major | namenode | Erik Krogen | Erik Krogen | +| [HADOOP-16331](https://issues.apache.org/jira/browse/HADOOP-16331) | Fix ASF License check in pom.xml | Major | . | Wanqiang Ji | Akira Ajisaka | +| [HDFS-14514](https://issues.apache.org/jira/browse/HDFS-14514) | Actual read size of open file in encryption zone still larger than listing size even after enabling HDFS-11402 in Hadoop 2 | Major | encryption, hdfs, snapshots | Siyao Meng | Siyao Meng | +| [HDFS-14512](https://issues.apache.org/jira/browse/HDFS-14512) | ONE\_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes) | Major | . | Shen Yinjie | Ayush Saxena | +| [HDFS-14521](https://issues.apache.org/jira/browse/HDFS-14521) | Suppress setReplication logging. | Major | . | Kihwal Lee | Kihwal Lee | +| [YARN-8625](https://issues.apache.org/jira/browse/YARN-8625) | Aggregate Resource Allocation for each job is not present in ATS | Major | ATSv2 | Prabhu Joseph | Prabhu Joseph | +| [HADOOP-16345](https://issues.apache.org/jira/browse/HADOOP-16345) | Potential NPE when instantiating FairCallQueue metrics | Major | ipc | Erik Krogen | Erik Krogen | +| [HDFS-14494](https://issues.apache.org/jira/browse/HDFS-14494) | Move Server logging of StatedId inside receiveRequestState() | Major | . | Konstantin Shvachko | Shweta | +| [HDFS-14535](https://issues.apache.org/jira/browse/HDFS-14535) | The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read | Major | hdfs-client | Zheng Hu | Zheng Hu | +| [HDFS-13730](https://issues.apache.org/jira/browse/HDFS-13730) | BlockReaderRemote.sendReadResult throws NPE | Major | hdfs-client | Wei-Chiu Chuang | Yuanbo Liu | +| [HDFS-13770](https://issues.apache.org/jira/browse/HDFS-13770) | dfsadmin -report does not always decrease "missing blocks (with replication factor 1)" metrics when file is deleted | Major | hdfs | Kitti Nanasi | Kitti Nanasi | +| [HDFS-14101](https://issues.apache.org/jira/browse/HDFS-14101) | Random failure of testListCorruptFilesCorruptedBlock | Major | test | Kihwal Lee | Zsolt Venczel | +| [HDFS-14465](https://issues.apache.org/jira/browse/HDFS-14465) | When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. | Major | . | Yicong Cai | Yicong Cai | +| [HDFS-14541](https://issues.apache.org/jira/browse/HDFS-14541) | When evictableMmapped or evictable size is zero, do not throw NoSuchElementException | Major | hdfs-client, performance | Zheng Hu | Lisheng Sun | +| [HDFS-14629](https://issues.apache.org/jira/browse/HDFS-14629) | Property value Hard Coded in DNConf.java | Trivial | . | hemanthboyina | hemanthboyina | +| [HDFS-12703](https://issues.apache.org/jira/browse/HDFS-12703) | Exceptions are fatal to decommissioning monitor | Critical | namenode | Daryn Sharp | Xiaoqiao He | +| [HDFS-12748](https://issues.apache.org/jira/browse/HDFS-12748) | NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY | Major | hdfs | Jiandan Yang | Weiwei Yang | +| [HADOOP-16386](https://issues.apache.org/jira/browse/HADOOP-16386) | FindBugs warning in branch-2: GlobalStorageStatistics defines non-transient non-serializable instance field map | Major | fs | Wei-Chiu Chuang | Masatake Iwasaki | +| [MAPREDUCE-6521](https://issues.apache.org/jira/browse/MAPREDUCE-6521) | MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test | Major | test | Masatake Iwasaki | Masatake Iwasaki | +| [MAPREDUCE-7076](https://issues.apache.org/jira/browse/MAPREDUCE-7076) | TestNNBench#testNNBenchCreateReadAndDelete failing in our internal build | Minor | test | Rushabh Shah | kevin su | +| [YARN-9668](https://issues.apache.org/jira/browse/YARN-9668) | UGI conf doesn't read user overridden configurations on RM and NM startup | Major | . | Jonathan Hung | Jonathan Hung | +| [HADOOP-9844](https://issues.apache.org/jira/browse/HADOOP-9844) | NPE when trying to create an error message response of SASL RPC | Major | ipc | Steve Loughran | Steve Loughran | +| [HADOOP-16245](https://issues.apache.org/jira/browse/HADOOP-16245) | Enabling SSL within LdapGroupsMapping can break system SSL configs | Major | common, security | Erik Krogen | Erik Krogen | +| [HDFS-14660](https://issues.apache.org/jira/browse/HDFS-14660) | [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider | Major | . | Chao Sun | Chao Sun | +| [HDFS-14429](https://issues.apache.org/jira/browse/HDFS-14429) | Block remain in COMMITTED but not COMPLETE caused by Decommission | Major | . | Yicong Cai | Yicong Cai | +| [HDFS-14672](https://issues.apache.org/jira/browse/HDFS-14672) | Backport HDFS-12703 to branch-2 | Major | namenode | Xiaoqiao He | Xiaoqiao He | +| [HADOOP-16435](https://issues.apache.org/jira/browse/HADOOP-16435) | RpcMetrics should not be retained forever | Critical | rpc-server | Zoltan Haindrich | Zoltan Haindrich | +| [HDFS-14464](https://issues.apache.org/jira/browse/HDFS-14464) | Remove unnecessary log message from DFSInputStream | Trivial | . | Kihwal Lee | Chao Sun | +| [HDFS-14569](https://issues.apache.org/jira/browse/HDFS-14569) | Result of crypto -listZones is not formatted properly | Major | . | hemanthboyina | hemanthboyina | +| [YARN-9596](https://issues.apache.org/jira/browse/YARN-9596) | QueueMetrics has incorrect metrics when labelled partitions are involved | Major | capacity scheduler | Muhammad Samir Khan | Muhammad Samir Khan | +| [HADOOP-15237](https://issues.apache.org/jira/browse/HADOOP-15237) | In KMS docs there should be one space between KMS\_LOG and NOTE | Minor | kms | Snigdhanjali Mishra | Snigdhanjali Mishra | +| [HDFS-14462](https://issues.apache.org/jira/browse/HDFS-14462) | WebHDFS throws "Error writing request body to server" instead of DSQuotaExceededException | Major | webhdfs | Erik Krogen | Simbarashe Dzinamarira | +| [HDFS-14631](https://issues.apache.org/jira/browse/HDFS-14631) | The DirectoryScanner doesn't fix the wrongly placed replica. | Major | . | Jinglun | Jinglun | +| [HDFS-14724](https://issues.apache.org/jira/browse/HDFS-14724) | Fix JDK7 compatibility in branch-2 | Blocker | . | Wei-Chiu Chuang | Chen Liang | +| [HDFS-14423](https://issues.apache.org/jira/browse/HDFS-14423) | Percent (%) and plus (+) characters no longer work in WebHDFS | Major | webhdfs | Jing Wang | Masatake Iwasaki | +| [HDFS-13101](https://issues.apache.org/jira/browse/HDFS-13101) | Yet another fsimage corruption related to snapshot | Major | snapshots | Yongjun Zhang | Shashikant Banerjee | +| [HDFS-14311](https://issues.apache.org/jira/browse/HDFS-14311) | Multi-threading conflict at layoutVersion when loading block pool storage | Major | rolling upgrades | Yicong Cai | Yicong Cai | +| [HADOOP-16494](https://issues.apache.org/jira/browse/HADOOP-16494) | Add SHA-256 or SHA-512 checksum to release artifacts to comply with the release distribution policy | Blocker | build | Akira Ajisaka | Akira Ajisaka | +| [HDFS-13977](https://issues.apache.org/jira/browse/HDFS-13977) | NameNode can kill itself if it tries to send too many txns to a QJM simultaneously | Major | namenode, qjm | Erik Krogen | Erik Krogen | +| [YARN-9438](https://issues.apache.org/jira/browse/YARN-9438) | launchTime not written to state store for running applications | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-7585](https://issues.apache.org/jira/browse/YARN-7585) | NodeManager should go unhealthy when state store throws DBException | Major | nodemanager | Wilfred Spiegelenburg | Wilfred Spiegelenburg | +| [HDFS-14726](https://issues.apache.org/jira/browse/HDFS-14726) | Fix JN incompatibility issue in branch-2 due to backport of HDFS-10519 | Blocker | journal-node | Chen Liang | Chen Liang | +| [YARN-9806](https://issues.apache.org/jira/browse/YARN-9806) | TestNMSimulator#testNMSimulator fails in branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9820](https://issues.apache.org/jira/browse/YARN-9820) | RM logs InvalidStateTransitionException when app is submitted | Critical | . | Rohith Sharma K S | Prabhu Joseph | +| [HDFS-14303](https://issues.apache.org/jira/browse/HDFS-14303) | check block directory logic not correct when there is only meta file, print no meaning warn log | Minor | datanode, hdfs | qiang Liu | qiang Liu | +| [HADOOP-16582](https://issues.apache.org/jira/browse/HADOOP-16582) | LocalFileSystem's mkdirs() does not work as expected under viewfs. | Major | . | Kihwal Lee | Kihwal Lee | +| [HADOOP-16581](https://issues.apache.org/jira/browse/HADOOP-16581) | ValueQueue does not trigger an async refill when number of values falls below watermark | Major | common, kms | Yuval Degani | Yuval Degani | +| [HDFS-14853](https://issues.apache.org/jira/browse/HDFS-14853) | NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present | Major | . | Ranith Sardar | Ranith Sardar | +| [YARN-9858](https://issues.apache.org/jira/browse/YARN-9858) | Optimize RMContext getExclusiveEnforcedPartitions | Major | . | Jonathan Hung | Jonathan Hung | +| [HDFS-14655](https://issues.apache.org/jira/browse/HDFS-14655) | [SBN Read] Namenode crashes if one of The JN is down | Critical | . | Harshakiran Reddy | Ayush Saxena | +| [HDFS-14245](https://issues.apache.org/jira/browse/HDFS-14245) | Class cast error in GetGroups with ObserverReadProxyProvider | Major | . | Shen Yinjie | Erik Krogen | +| [HDFS-14509](https://issues.apache.org/jira/browse/HDFS-14509) | DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x | Blocker | . | Yuxuan Wang | Yuxuan Wang | +| [HADOOP-16655](https://issues.apache.org/jira/browse/HADOOP-16655) | Change cipher suite when fetching tomcat tarball for branch-2 | Major | . | Jonathan Hung | Jonathan Hung | + + +### TESTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HDFS-13337](https://issues.apache.org/jira/browse/HDFS-13337) | Backport HDFS-4275 to branch-2.9 | Minor | . | Íñigo Goiri | Xiao Liang | +| [HDFS-13503](https://issues.apache.org/jira/browse/HDFS-13503) | Fix TestFsck test failures on Windows | Major | hdfs | Xiao Liang | Xiao Liang | +| [HDFS-13542](https://issues.apache.org/jira/browse/HDFS-13542) | TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13551](https://issues.apache.org/jira/browse/HDFS-13551) | TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-11700](https://issues.apache.org/jira/browse/HDFS-11700) | TestHDFSServerPorts#testBackupNodePorts doesn't pass on Windows | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13548](https://issues.apache.org/jira/browse/HDFS-13548) | TestResolveHdfsSymlink#testFcResolveAfs fails on Windows | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13567](https://issues.apache.org/jira/browse/HDFS-13567) | TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13557](https://issues.apache.org/jira/browse/HDFS-13557) | TestDFSAdmin#testListOpenFiles fails on Windows | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13550](https://issues.apache.org/jira/browse/HDFS-13550) | TestDebugAdmin#testComputeMetaCommand fails on Windows | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13559](https://issues.apache.org/jira/browse/HDFS-13559) | TestBlockScanner does not close TestContext properly | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13570](https://issues.apache.org/jira/browse/HDFS-13570) | TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13558](https://issues.apache.org/jira/browse/HDFS-13558) | TestDatanodeHttpXFrame does not shut down cluster | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13554](https://issues.apache.org/jira/browse/HDFS-13554) | TestDatanodeRegistration#testForcedRegistration does not shut down cluster | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13556](https://issues.apache.org/jira/browse/HDFS-13556) | TestNestedEncryptionZones does not shut down cluster | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13560](https://issues.apache.org/jira/browse/HDFS-13560) | Insufficient system resources exist to complete the requested service for some tests on Windows | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13592](https://issues.apache.org/jira/browse/HDFS-13592) | TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13593](https://issues.apache.org/jira/browse/HDFS-13593) | TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows | Minor | test | Anbang Hu | Anbang Hu | +| [HDFS-13587](https://issues.apache.org/jira/browse/HDFS-13587) | TestQuorumJournalManager fails on Windows | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13620](https://issues.apache.org/jira/browse/HDFS-13620) | Randomize the test directory path for TestHDFSFileSystemContract | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13591](https://issues.apache.org/jira/browse/HDFS-13591) | TestDFSShell#testSetrepLow fails on Windows | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13632](https://issues.apache.org/jira/browse/HDFS-13632) | Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA | Minor | . | Anbang Hu | Anbang Hu | +| [MAPREDUCE-7102](https://issues.apache.org/jira/browse/MAPREDUCE-7102) | Fix TestJavaSerialization for Windows due a mismatch line separator | Minor | . | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola | +| [HDFS-13652](https://issues.apache.org/jira/browse/HDFS-13652) | Randomize baseDir for MiniDFSCluster in TestBlockScanner | Minor | . | Anbang Hu | Anbang Hu | +| [YARN-8370](https://issues.apache.org/jira/browse/YARN-8370) | Some Node Manager tests fail on Windows due to improper path/file separator | Minor | . | Anbang Hu | Anbang Hu | +| [YARN-8422](https://issues.apache.org/jira/browse/YARN-8422) | TestAMSimulator failing with NPE | Minor | . | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola | +| [HADOOP-15532](https://issues.apache.org/jira/browse/HADOOP-15532) | TestBasicDiskValidator fails with NoSuchFileException | Minor | . | Íñigo Goiri | Giovanni Matteo Fumarola | +| [HDFS-13563](https://issues.apache.org/jira/browse/HDFS-13563) | TestDFSAdminWithHA times out on Windows | Minor | . | Anbang Hu | Lukas Majercak | +| [HDFS-13681](https://issues.apache.org/jira/browse/HDFS-13681) | Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows | Major | test | Xiao Liang | Xiao Liang | +| [YARN-8944](https://issues.apache.org/jira/browse/YARN-8944) | TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896 | Minor | capacity scheduler | Wilfred Spiegelenburg | Wilfred Spiegelenburg | +| [HDFS-11950](https://issues.apache.org/jira/browse/HDFS-11950) | Disable libhdfs zerocopy test on Mac | Minor | libhdfs | John Zhuge | Akira Ajisaka | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [YARN-4081](https://issues.apache.org/jira/browse/YARN-4081) | Add support for multiple resource types in the Resource class | Major | resourcemanager | Varun Vasudev | Varun Vasudev | +| [YARN-4172](https://issues.apache.org/jira/browse/YARN-4172) | Extend DominantResourceCalculator to account for all resources | Major | resourcemanager | Varun Vasudev | Varun Vasudev | +| [YARN-4715](https://issues.apache.org/jira/browse/YARN-4715) | Add support to read resource types from a config file | Major | nodemanager, resourcemanager | Varun Vasudev | Varun Vasudev | +| [YARN-4829](https://issues.apache.org/jira/browse/YARN-4829) | Add support for binary units | Major | nodemanager, resourcemanager | Varun Vasudev | Varun Vasudev | +| [YARN-4830](https://issues.apache.org/jira/browse/YARN-4830) | Add support for resource types in the nodemanager | Major | nodemanager | Varun Vasudev | Varun Vasudev | +| [YARN-5242](https://issues.apache.org/jira/browse/YARN-5242) | Update DominantResourceCalculator to consider all resource types in calculations | Major | resourcemanager | Varun Vasudev | Varun Vasudev | +| [YARN-5586](https://issues.apache.org/jira/browse/YARN-5586) | Update the Resources class to consider all resource types | Major | nodemanager, resourcemanager | Varun Vasudev | Varun Vasudev | +| [YARN-6761](https://issues.apache.org/jira/browse/YARN-6761) | Fix build for YARN-3926 branch | Major | nodemanager, resourcemanager | Varun Vasudev | Varun Vasudev | +| [YARN-6786](https://issues.apache.org/jira/browse/YARN-6786) | ResourcePBImpl imports cleanup | Trivial | resourcemanager | Daniel Templeton | Yeliang Cang | +| [YARN-6788](https://issues.apache.org/jira/browse/YARN-6788) | Improve performance of resource profile branch | Blocker | nodemanager, resourcemanager | Sunil G | Sunil G | +| [YARN-6994](https://issues.apache.org/jira/browse/YARN-6994) | Remove last uses of Long from resource types code | Minor | resourcemanager | Daniel Templeton | Daniel Templeton | +| [YARN-6892](https://issues.apache.org/jira/browse/YARN-6892) | Improve API implementation in Resources and DominantResourceCalculator class | Major | nodemanager, resourcemanager | Sunil G | Sunil G | +| [YARN-6610](https://issues.apache.org/jira/browse/YARN-6610) | DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources | Critical | resourcemanager | Daniel Templeton | Daniel Templeton | +| [YARN-7030](https://issues.apache.org/jira/browse/YARN-7030) | Performance optimizations in Resource and ResourceUtils class | Critical | nodemanager, resourcemanager | Wangda Tan | Wangda Tan | +| [YARN-7042](https://issues.apache.org/jira/browse/YARN-7042) | Clean up unit tests after YARN-6610 | Major | test | Daniel Templeton | Daniel Templeton | +| [YARN-6789](https://issues.apache.org/jira/browse/YARN-6789) | Add Client API to get all supported resource types from RM | Major | nodemanager, resourcemanager | Sunil G | Sunil G | +| [YARN-6781](https://issues.apache.org/jira/browse/YARN-6781) | ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter | Minor | resourcemanager | Daniel Templeton | Yu-Tang Lin | +| [YARN-7067](https://issues.apache.org/jira/browse/YARN-7067) | Optimize ResourceType information display in UI | Critical | nodemanager, resourcemanager | Wangda Tan | Wangda Tan | +| [YARN-7039](https://issues.apache.org/jira/browse/YARN-7039) | Fix javac and javadoc errors in YARN-3926 branch | Major | nodemanager, resourcemanager | Sunil G | Sunil G | +| [YARN-7093](https://issues.apache.org/jira/browse/YARN-7093) | Improve log message in ResourceUtils | Trivial | nodemanager, resourcemanager | Sunil G | Sunil G | +| [YARN-6933](https://issues.apache.org/jira/browse/YARN-6933) | ResourceUtils.DISALLOWED\_NAMES check is duplicated | Major | resourcemanager | Daniel Templeton | Manikandan R | +| [YARN-7137](https://issues.apache.org/jira/browse/YARN-7137) | Move newly added APIs to unstable in YARN-3926 branch | Blocker | nodemanager, resourcemanager | Wangda Tan | Wangda Tan | +| [HADOOP-14799](https://issues.apache.org/jira/browse/HADOOP-14799) | Update nimbus-jose-jwt to 4.41.1 | Major | . | Ray Chiang | Ray Chiang | +| [YARN-7345](https://issues.apache.org/jira/browse/YARN-7345) | GPU Isolation: Incorrect minor device numbers written to devices.deny file | Major | . | Jonathan Hung | Jonathan Hung | +| [HADOOP-14997](https://issues.apache.org/jira/browse/HADOOP-14997) | Add hadoop-aliyun as dependency of hadoop-cloud-storage | Minor | fs/oss | Genmao Yu | Genmao Yu | +| [YARN-7143](https://issues.apache.org/jira/browse/YARN-7143) | FileNotFound handling in ResourceUtils is inconsistent | Major | resourcemanager | Daniel Templeton | Daniel Templeton | +| [YARN-6909](https://issues.apache.org/jira/browse/YARN-6909) | Use LightWeightedResource when number of resource types more than two | Critical | resourcemanager | Daniel Templeton | Sunil G | +| [HDFS-12801](https://issues.apache.org/jira/browse/HDFS-12801) | RBF: Set MountTableResolver as default file resolver | Minor | . | Íñigo Goiri | Íñigo Goiri | +| [YARN-7430](https://issues.apache.org/jira/browse/YARN-7430) | Enable user re-mapping for Docker containers by default | Blocker | security, yarn | Eric Yang | Eric Yang | +| [HADOOP-15024](https://issues.apache.org/jira/browse/HADOOP-15024) | AliyunOSS: support user agent configuration and include that & Hadoop version information to oss server | Major | fs, fs/oss | Sammi Chen | Sammi Chen | +| [HDFS-12858](https://issues.apache.org/jira/browse/HDFS-12858) | RBF: Add router admin commands usage in HDFS commands reference doc | Minor | documentation | Yiqun Lin | Yiqun Lin | +| [HDFS-12835](https://issues.apache.org/jira/browse/HDFS-12835) | RBF: Fix Javadoc parameter errors | Minor | . | Wei Yan | Wei Yan | +| [YARN-7573](https://issues.apache.org/jira/browse/YARN-7573) | Gpu Information page could be empty for nodes without GPU | Major | webapp, yarn-ui-v2 | Sunil G | Sunil G | +| [HDFS-12396](https://issues.apache.org/jira/browse/HDFS-12396) | Webhdfs file system should get delegation token from kms provider. | Major | encryption, kms, webhdfs | Rushabh Shah | Rushabh Shah | +| [YARN-6704](https://issues.apache.org/jira/browse/YARN-6704) | Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyService | Major | . | Botong Huang | Botong Huang | +| [HDFS-12875](https://issues.apache.org/jira/browse/HDFS-12875) | RBF: Complete logic for -readonly option of dfsrouteradmin add command | Major | . | Yiqun Lin | Íñigo Goiri | +| [YARN-7383](https://issues.apache.org/jira/browse/YARN-7383) | Node resource is not parsed correctly for resource names containing dot | Major | nodemanager, resourcemanager | Jonathan Hung | Gergely Novák | +| [YARN-7630](https://issues.apache.org/jira/browse/YARN-7630) | Fix AMRMToken rollover handling in AMRMProxy | Minor | . | Botong Huang | Botong Huang | +| [HDFS-12937](https://issues.apache.org/jira/browse/HDFS-12937) | RBF: Add more unit tests for router admin commands | Major | test | Yiqun Lin | Yiqun Lin | +| [YARN-7032](https://issues.apache.org/jira/browse/YARN-7032) | [ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled. | Critical | . | Rohith Sharma K S | Rohith Sharma K S | +| [HDFS-12988](https://issues.apache.org/jira/browse/HDFS-12988) | RBF: Mount table entries not properly updated in the local cache | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-12802](https://issues.apache.org/jira/browse/HDFS-12802) | RBF: Control MountTableResolver cache size | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-12934](https://issues.apache.org/jira/browse/HDFS-12934) | RBF: Federation supports global quota | Major | . | Yiqun Lin | Yiqun Lin | +| [HDFS-12972](https://issues.apache.org/jira/browse/HDFS-12972) | RBF: Display mount table quota info in Web UI and admin command | Major | . | Yiqun Lin | Yiqun Lin | +| [YARN-6736](https://issues.apache.org/jira/browse/YARN-6736) | Consider writing to both ats v1 & v2 from RM for smoother upgrades | Major | timelineserver | Vrushali C | Aaron Gresch | +| [HADOOP-15027](https://issues.apache.org/jira/browse/HADOOP-15027) | AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance | Major | fs/oss | wujinhu | wujinhu | +| [HDFS-12973](https://issues.apache.org/jira/browse/HDFS-12973) | RBF: Document global quota supporting in federation | Major | . | Yiqun Lin | Yiqun Lin | +| [HDFS-13028](https://issues.apache.org/jira/browse/HDFS-13028) | RBF: Fix spurious TestRouterRpc#testProxyGetStats | Minor | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-12772](https://issues.apache.org/jira/browse/HDFS-12772) | RBF: Federation Router State State Store internal API | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13042](https://issues.apache.org/jira/browse/HDFS-13042) | RBF: Heartbeat Router State | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13049](https://issues.apache.org/jira/browse/HDFS-13049) | RBF: Inconsistent Router OPTS config in branch-2 and branch-3 | Minor | . | Wei Yan | Wei Yan | +| [YARN-7817](https://issues.apache.org/jira/browse/YARN-7817) | Add Resource reference to RM's NodeInfo object so REST API can get non memory/vcore resource usages. | Major | . | Sumana Sathish | Sunil G | +| [HDFS-12574](https://issues.apache.org/jira/browse/HDFS-12574) | Add CryptoInputStream to WebHdfsFileSystem read call. | Major | encryption, kms, webhdfs | Rushabh Shah | Rushabh Shah | +| [HDFS-13044](https://issues.apache.org/jira/browse/HDFS-13044) | RBF: Add a safe mode for the Router | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13043](https://issues.apache.org/jira/browse/HDFS-13043) | RBF: Expose the state of the Routers in the federation | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13068](https://issues.apache.org/jira/browse/HDFS-13068) | RBF: Add router admin option to manage safe mode | Major | . | Íñigo Goiri | Yiqun Lin | +| [YARN-7860](https://issues.apache.org/jira/browse/YARN-7860) | Fix UT failure TestRMWebServiceAppsNodelabel#testAppsRunning | Major | . | Weiwei Yang | Sunil G | +| [HDFS-13119](https://issues.apache.org/jira/browse/HDFS-13119) | RBF: Manage unavailable clusters | Major | . | Íñigo Goiri | Yiqun Lin | +| [YARN-7223](https://issues.apache.org/jira/browse/YARN-7223) | Document GPU isolation feature | Blocker | . | Wangda Tan | Wangda Tan | +| [HDFS-13187](https://issues.apache.org/jira/browse/HDFS-13187) | RBF: Fix Routers information shown in the web UI | Minor | . | Wei Yan | Wei Yan | +| [HDFS-13184](https://issues.apache.org/jira/browse/HDFS-13184) | RBF: Improve the unit test TestRouterRPCClientRetries | Minor | test | Yiqun Lin | Yiqun Lin | +| [HDFS-13199](https://issues.apache.org/jira/browse/HDFS-13199) | RBF: Fix the hdfs router page missing label icon issue | Major | federation, hdfs | maobaolong | maobaolong | +| [HADOOP-15090](https://issues.apache.org/jira/browse/HADOOP-15090) | Add ADL troubleshooting doc | Major | documentation, fs/adl | Steve Loughran | Steve Loughran | +| [YARN-7919](https://issues.apache.org/jira/browse/YARN-7919) | Refactor timelineservice-hbase module into submodules | Major | timelineservice | Haibo Chen | Haibo Chen | +| [YARN-8003](https://issues.apache.org/jira/browse/YARN-8003) | Backport the code structure changes in YARN-7346 to branch-2 | Major | . | Haibo Chen | Haibo Chen | +| [HDFS-13214](https://issues.apache.org/jira/browse/HDFS-13214) | RBF: Complete document of Router configuration | Major | . | Tao Jie | Yiqun Lin | +| [HADOOP-15267](https://issues.apache.org/jira/browse/HADOOP-15267) | S3A multipart upload fails when SSE-C encryption is enabled | Critical | fs/s3 | Anis Elleuch | Anis Elleuch | +| [HDFS-13230](https://issues.apache.org/jira/browse/HDFS-13230) | RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns | Minor | . | Wei Yan | Chao Sun | +| [HDFS-13233](https://issues.apache.org/jira/browse/HDFS-13233) | RBF: MountTableResolver doesn't return the correct mount point of the given path | Major | hdfs | wangzhiyuan | wangzhiyuan | +| [HDFS-13212](https://issues.apache.org/jira/browse/HDFS-13212) | RBF: Fix router location cache issue | Major | federation, hdfs | Wu Weiwei | Wu Weiwei | +| [HDFS-13232](https://issues.apache.org/jira/browse/HDFS-13232) | RBF: ConnectionPool should return first usable connection | Minor | . | Wei Yan | Ekanth Sethuramalingam | +| [HDFS-13240](https://issues.apache.org/jira/browse/HDFS-13240) | RBF: Update some inaccurate document descriptions | Minor | . | Yiqun Lin | Yiqun Lin | +| [HDFS-11399](https://issues.apache.org/jira/browse/HDFS-11399) | Many tests fails in Windows due to injecting disk failures | Major | . | Yiqun Lin | Yiqun Lin | +| [HDFS-13241](https://issues.apache.org/jira/browse/HDFS-13241) | RBF: TestRouterSafemode failed if the port 8888 is in use | Major | hdfs, test | maobaolong | maobaolong | +| [HDFS-13253](https://issues.apache.org/jira/browse/HDFS-13253) | RBF: Quota management incorrect parent-child relationship judgement | Major | . | Yiqun Lin | Yiqun Lin | +| [HDFS-13226](https://issues.apache.org/jira/browse/HDFS-13226) | RBF: Throw the exception if mount table entry validated failed | Major | hdfs | maobaolong | maobaolong | +| [HADOOP-15308](https://issues.apache.org/jira/browse/HADOOP-15308) | TestConfiguration fails on Windows because of paths | Major | test | Íñigo Goiri | Xiao Liang | +| [HDFS-12773](https://issues.apache.org/jira/browse/HDFS-12773) | RBF: Improve State Store FS implementation | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13198](https://issues.apache.org/jira/browse/HDFS-13198) | RBF: RouterHeartbeatService throws out CachedStateStore related exceptions when starting router | Minor | . | Wei Yan | Wei Yan | +| [HDFS-13224](https://issues.apache.org/jira/browse/HDFS-13224) | RBF: Resolvers to support mount points across multiple subclusters | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13299](https://issues.apache.org/jira/browse/HDFS-13299) | RBF : Fix compilation error in branch-2 (TestMultipleDestinationResolver) | Blocker | . | Brahma Reddy Battula | Brahma Reddy Battula | +| [HADOOP-15262](https://issues.apache.org/jira/browse/HADOOP-15262) | AliyunOSS: move files under a directory in parallel when rename a directory | Major | fs/oss | wujinhu | wujinhu | +| [HDFS-13215](https://issues.apache.org/jira/browse/HDFS-13215) | RBF: Move Router to its own module | Major | . | Íñigo Goiri | Wei Yan | +| [HDFS-13307](https://issues.apache.org/jira/browse/HDFS-13307) | RBF: Improve the use of setQuota command | Major | . | liuhongtong | liuhongtong | +| [HDFS-13250](https://issues.apache.org/jira/browse/HDFS-13250) | RBF: Router to manage requests across multiple subclusters | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13318](https://issues.apache.org/jira/browse/HDFS-13318) | RBF: Fix FindBugs in hadoop-hdfs-rbf | Minor | . | Íñigo Goiri | Ekanth Sethuramalingam | +| [HDFS-12792](https://issues.apache.org/jira/browse/HDFS-12792) | RBF: Test Router-based federation using HDFSContract | Major | . | Íñigo Goiri | Íñigo Goiri | +| [YARN-7581](https://issues.apache.org/jira/browse/YARN-7581) | HBase filters are not constructed correctly in ATSv2 | Major | ATSv2 | Haibo Chen | Haibo Chen | +| [YARN-7986](https://issues.apache.org/jira/browse/YARN-7986) | ATSv2 REST API queries do not return results for uppercase application tags | Critical | . | Charan Hebri | Charan Hebri | +| [HDFS-12512](https://issues.apache.org/jira/browse/HDFS-12512) | RBF: Add WebHDFS | Major | fs | Íñigo Goiri | Wei Yan | +| [HDFS-13291](https://issues.apache.org/jira/browse/HDFS-13291) | RBF: Implement available space based OrderResolver | Major | . | Yiqun Lin | Yiqun Lin | +| [HDFS-13204](https://issues.apache.org/jira/browse/HDFS-13204) | RBF: Optimize name service safe mode icon | Minor | . | liuhongtong | liuhongtong | +| [HDFS-13352](https://issues.apache.org/jira/browse/HDFS-13352) | RBF: Add xsl stylesheet for hdfs-rbf-default.xml | Major | documentation | Takanobu Asanuma | Takanobu Asanuma | +| [YARN-8010](https://issues.apache.org/jira/browse/YARN-8010) | Add config in FederationRMFailoverProxy to not bypass facade cache when failing over | Minor | . | Botong Huang | Botong Huang | +| [HDFS-13347](https://issues.apache.org/jira/browse/HDFS-13347) | RBF: Cache datanode reports | Minor | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13289](https://issues.apache.org/jira/browse/HDFS-13289) | RBF: TestConnectionManager#testCleanup() test case need correction | Minor | . | Dibyendu Karmakar | Dibyendu Karmakar | +| [HDFS-13364](https://issues.apache.org/jira/browse/HDFS-13364) | RBF: Support NamenodeProtocol in the Router | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HADOOP-14651](https://issues.apache.org/jira/browse/HADOOP-14651) | Update okhttp version to 2.7.5 | Major | fs/adl | Ray Chiang | Ray Chiang | +| [YARN-6936](https://issues.apache.org/jira/browse/YARN-6936) | [Atsv2] Retrospect storing entities into sub application table from client perspective | Major | . | Rohith Sharma K S | Rohith Sharma K S | +| [HDFS-13353](https://issues.apache.org/jira/browse/HDFS-13353) | RBF: TestRouterWebHDFSContractCreate failed | Major | test | Takanobu Asanuma | Takanobu Asanuma | +| [YARN-8107](https://issues.apache.org/jira/browse/YARN-8107) | Give an informative message when incorrect format is used in ATSv2 filter attributes | Major | ATSv2 | Charan Hebri | Rohith Sharma K S | +| [YARN-8110](https://issues.apache.org/jira/browse/YARN-8110) | AMRMProxy recover should catch for all throwable to avoid premature exit | Major | . | Botong Huang | Botong Huang | +| [HDFS-13402](https://issues.apache.org/jira/browse/HDFS-13402) | RBF: Fix java doc for StateStoreFileSystemImpl | Minor | hdfs | Yiran Wu | Yiran Wu | +| [HDFS-13380](https://issues.apache.org/jira/browse/HDFS-13380) | RBF: mv/rm fail after the directory exceeded the quota limit | Major | . | Wu Weiwei | Yiqun Lin | +| [HDFS-13410](https://issues.apache.org/jira/browse/HDFS-13410) | RBF: Support federation with no subclusters | Minor | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13384](https://issues.apache.org/jira/browse/HDFS-13384) | RBF: Improve timeout RPC call mechanism | Minor | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13045](https://issues.apache.org/jira/browse/HDFS-13045) | RBF: Improve error message returned from subcluster | Minor | . | Wei Yan | Íñigo Goiri | +| [HDFS-13428](https://issues.apache.org/jira/browse/HDFS-13428) | RBF: Remove LinkedList From StateStoreFileImpl.java | Trivial | federation | David Mollitor | David Mollitor | +| [HADOOP-14999](https://issues.apache.org/jira/browse/HADOOP-14999) | AliyunOSS: provide one asynchronous multi-part based uploading mechanism | Major | fs/oss | Genmao Yu | Genmao Yu | +| [YARN-7810](https://issues.apache.org/jira/browse/YARN-7810) | TestDockerContainerRuntime test failures due to UID lookup of a non-existent user | Major | . | Shane Kumpf | Shane Kumpf | +| [HDFS-13435](https://issues.apache.org/jira/browse/HDFS-13435) | RBF: Improve the error loggings for printing the stack trace | Major | . | Yiqun Lin | Yiqun Lin | +| [YARN-7189](https://issues.apache.org/jira/browse/YARN-7189) | Container-executor doesn't remove Docker containers that error out early | Major | yarn | Eric Badger | Eric Badger | +| [HDFS-13466](https://issues.apache.org/jira/browse/HDFS-13466) | RBF: Add more router-related information to the UI | Minor | . | Wei Yan | Wei Yan | +| [HDFS-13453](https://issues.apache.org/jira/browse/HDFS-13453) | RBF: getMountPointDates should fetch latest subdir time/date when parent dir is not present but /parent/child dirs are present in mount table | Major | . | Dibyendu Karmakar | Dibyendu Karmakar | +| [HDFS-13478](https://issues.apache.org/jira/browse/HDFS-13478) | RBF: Disabled Nameservice store API | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13490](https://issues.apache.org/jira/browse/HDFS-13490) | RBF: Fix setSafeMode in the Router | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13484](https://issues.apache.org/jira/browse/HDFS-13484) | RBF: Disable Nameservices from the federation | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13326](https://issues.apache.org/jira/browse/HDFS-13326) | RBF: Improve the interfaces to modify and view mount tables | Minor | . | Wei Yan | Gang Li | +| [HDFS-13499](https://issues.apache.org/jira/browse/HDFS-13499) | RBF: Show disabled name services in the UI | Minor | . | Íñigo Goiri | Íñigo Goiri | +| [YARN-8215](https://issues.apache.org/jira/browse/YARN-8215) | ATS v2 returns invalid YARN\_CONTAINER\_ALLOCATED\_HOST\_HTTP\_ADDRESS from NM | Critical | ATSv2 | Yesha Vora | Rohith Sharma K S | +| [HDFS-13508](https://issues.apache.org/jira/browse/HDFS-13508) | RBF: Normalize paths (automatically) when adding, updating, removing or listing mount table entries | Minor | . | Ekanth Sethuramalingam | Ekanth Sethuramalingam | +| [HDFS-13434](https://issues.apache.org/jira/browse/HDFS-13434) | RBF: Fix dead links in RBF document | Major | documentation | Akira Ajisaka | Chetna Chaudhari | +| [HDFS-13488](https://issues.apache.org/jira/browse/HDFS-13488) | RBF: Reject requests when a Router is overloaded | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HDFS-13525](https://issues.apache.org/jira/browse/HDFS-13525) | RBF: Add unit test TestStateStoreDisabledNameservice | Major | . | Yiqun Lin | Yiqun Lin | +| [YARN-8253](https://issues.apache.org/jira/browse/YARN-8253) | HTTPS Ats v2 api call fails with "bad HTTP parsed" | Critical | ATSv2 | Yesha Vora | Charan Hebri | +| [HADOOP-15454](https://issues.apache.org/jira/browse/HADOOP-15454) | TestRollingFileSystemSinkWithLocal fails on Windows | Major | test | Xiao Liang | Xiao Liang | +| [HDFS-13346](https://issues.apache.org/jira/browse/HDFS-13346) | RBF: Fix synchronization of router quota and nameservice quota | Major | . | liuhongtong | Yiqun Lin | +| [YARN-8247](https://issues.apache.org/jira/browse/YARN-8247) | Incorrect HTTP status code returned by ATSv2 for non-whitelisted users | Critical | ATSv2 | Charan Hebri | Rohith Sharma K S | +| [YARN-8130](https://issues.apache.org/jira/browse/YARN-8130) | Race condition when container events are published for KILLED applications | Major | ATSv2 | Charan Hebri | Rohith Sharma K S | +| [YARN-7900](https://issues.apache.org/jira/browse/YARN-7900) | [AMRMProxy] AMRMClientRelayer for stateful FederationInterceptor | Major | . | Botong Huang | Botong Huang | +| [HADOOP-15498](https://issues.apache.org/jira/browse/HADOOP-15498) | TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows | Minor | . | Anbang Hu | Anbang Hu | +| [HADOOP-15497](https://issues.apache.org/jira/browse/HADOOP-15497) | TestTrash should use proper test path to avoid failing on Windows | Minor | . | Anbang Hu | Anbang Hu | +| [HDFS-13637](https://issues.apache.org/jira/browse/HDFS-13637) | RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN\_VALUE | Critical | federation | CR Hota | CR Hota | +| [YARN-4781](https://issues.apache.org/jira/browse/YARN-4781) | Support intra-queue preemption for fairness ordering policy. | Major | scheduler | Wangda Tan | Eric Payne | +| [HADOOP-15506](https://issues.apache.org/jira/browse/HADOOP-15506) | Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks | Minor | fs/azure | Esfandiar Manii | Esfandiar Manii | +| [HADOOP-15529](https://issues.apache.org/jira/browse/HADOOP-15529) | ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows | Minor | . | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola | +| [HADOOP-15533](https://issues.apache.org/jira/browse/HADOOP-15533) | Make WASB listStatus messages consistent | Trivial | fs/azure | Esfandiar Manii | Esfandiar Manii | +| [HADOOP-15458](https://issues.apache.org/jira/browse/HADOOP-15458) | TestLocalFileSystem#testFSOutputStreamBuilder fails on Windows | Minor | test | Xiao Liang | Xiao Liang | +| [YARN-8481](https://issues.apache.org/jira/browse/YARN-8481) | AMRMProxyPolicies should accept heartbeat response from new/unknown subclusters | Minor | amrmproxy, federation | Botong Huang | Botong Huang | +| [HDFS-13528](https://issues.apache.org/jira/browse/HDFS-13528) | RBF: If a directory exceeds quota limit then quota usage is not refreshed for other mount entries | Major | . | Dibyendu Karmakar | Dibyendu Karmakar | +| [HDFS-13710](https://issues.apache.org/jira/browse/HDFS-13710) | RBF: setQuota and getQuotaUsage should check the dfs.federation.router.quota.enable | Major | federation, hdfs | yanghuafeng | yanghuafeng | +| [HDFS-13726](https://issues.apache.org/jira/browse/HDFS-13726) | RBF: Fix RBF configuration links | Minor | documentation | Takanobu Asanuma | Takanobu Asanuma | +| [HDFS-13475](https://issues.apache.org/jira/browse/HDFS-13475) | RBF: Admin cannot enforce Router enter SafeMode | Major | . | Wei Yan | Chao Sun | +| [HDFS-13733](https://issues.apache.org/jira/browse/HDFS-13733) | RBF: Add Web UI configurations and descriptions to RBF document | Minor | documentation | Takanobu Asanuma | Takanobu Asanuma | +| [HDFS-13743](https://issues.apache.org/jira/browse/HDFS-13743) | RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolver | Major | . | Takanobu Asanuma | Takanobu Asanuma | +| [HDFS-13583](https://issues.apache.org/jira/browse/HDFS-13583) | RBF: Router admin clrQuota is not synchronized with nameservice | Major | . | Dibyendu Karmakar | Dibyendu Karmakar | +| [HDFS-13750](https://issues.apache.org/jira/browse/HDFS-13750) | RBF: Router ID in RouterRpcClient is always null | Major | . | Takanobu Asanuma | Takanobu Asanuma | +| [YARN-8129](https://issues.apache.org/jira/browse/YARN-8129) | Improve error message for invalid value in fields attribute | Minor | ATSv2 | Charan Hebri | Abhishek Modi | +| [YARN-8581](https://issues.apache.org/jira/browse/YARN-8581) | [AMRMProxy] Add sub-cluster timeout in LocalityMulticastAMRMProxyPolicy | Major | amrmproxy, federation | Botong Huang | Botong Huang | +| [YARN-8673](https://issues.apache.org/jira/browse/YARN-8673) | [AMRMProxy] More robust responseId resync after an YarnRM master slave switch | Major | amrmproxy | Botong Huang | Botong Huang | +| [HDFS-13848](https://issues.apache.org/jira/browse/HDFS-13848) | Refactor NameNode failover proxy providers | Major | ha, hdfs-client | Konstantin Shvachko | Konstantin Shvachko | +| [HDFS-13634](https://issues.apache.org/jira/browse/HDFS-13634) | RBF: Configurable value in xml for async connection request queue size. | Major | federation | CR Hota | CR Hota | +| [HADOOP-15731](https://issues.apache.org/jira/browse/HADOOP-15731) | TestDistributedShell fails on Windows | Major | . | Botong Huang | Botong Huang | +| [HADOOP-15759](https://issues.apache.org/jira/browse/HADOOP-15759) | AliyunOSS: update oss-sdk version to 3.0.0 | Major | fs/oss | wujinhu | wujinhu | +| [HADOOP-15748](https://issues.apache.org/jira/browse/HADOOP-15748) | S3 listing inconsistency can raise NPE in globber | Major | fs | Steve Loughran | Steve Loughran | +| [YARN-8696](https://issues.apache.org/jira/browse/YARN-8696) | [AMRMProxy] FederationInterceptor upgrade: home sub-cluster heartbeat async | Major | nodemanager | Botong Huang | Botong Huang | +| [HADOOP-15671](https://issues.apache.org/jira/browse/HADOOP-15671) | AliyunOSS: Support Assume Roles in AliyunOSS | Major | fs/oss | wujinhu | wujinhu | +| [HDFS-13790](https://issues.apache.org/jira/browse/HDFS-13790) | RBF: Move ClientProtocol APIs to its own module | Major | . | Íñigo Goiri | Chao Sun | +| [YARN-7652](https://issues.apache.org/jira/browse/YARN-7652) | Handle AM register requests asynchronously in FederationInterceptor | Major | amrmproxy, federation | Subramaniam Krishnan | Botong Huang | +| [YARN-6989](https://issues.apache.org/jira/browse/YARN-6989) | Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way | Major | timelineserver | Vrushali C | Abhishek Modi | +| [YARN-3879](https://issues.apache.org/jira/browse/YARN-3879) | [Storage implementation] Create HDFS backing storage implementation for ATS reads | Major | timelineserver | Tsuyoshi Ozawa | Abhishek Modi | +| [HADOOP-15837](https://issues.apache.org/jira/browse/HADOOP-15837) | DynamoDB table Update can fail S3A FS init | Major | fs/s3 | Steve Loughran | Steve Loughran | +| [HADOOP-15607](https://issues.apache.org/jira/browse/HADOOP-15607) | AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream | Critical | . | wujinhu | wujinhu | +| [HADOOP-15868](https://issues.apache.org/jira/browse/HADOOP-15868) | AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy | Major | fs/oss | wujinhu | wujinhu | +| [YARN-8893](https://issues.apache.org/jira/browse/YARN-8893) | [AMRMProxy] Fix thread leak in AMRMClientRelayer and UAM client | Major | amrmproxy, federation | Botong Huang | Botong Huang | +| [YARN-8905](https://issues.apache.org/jira/browse/YARN-8905) | [Router] Add JvmMetricsInfo and pause monitor | Minor | . | Bibin Chundatt | Bilwa S T | +| [HADOOP-15917](https://issues.apache.org/jira/browse/HADOOP-15917) | AliyunOSS: fix incorrect ReadOps and WriteOps in statistics | Major | fs/oss | wujinhu | wujinhu | +| [HADOOP-16009](https://issues.apache.org/jira/browse/HADOOP-16009) | Replace the url of the repository in Apache Hadoop source code | Major | documentation | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-15323](https://issues.apache.org/jira/browse/HADOOP-15323) | AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore | Major | fs/oss | wujinhu | wujinhu | +| [YARN-9182](https://issues.apache.org/jira/browse/YARN-9182) | Backport YARN-6445 resource profile performance improvements to branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9181](https://issues.apache.org/jira/browse/YARN-9181) | Backport YARN-6232 for generic resource type usage to branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9177](https://issues.apache.org/jira/browse/YARN-9177) | Use resource map for app metrics in TestCombinedSystemMetricsPublisher for branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9188](https://issues.apache.org/jira/browse/YARN-9188) | Port YARN-7136 to branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9187](https://issues.apache.org/jira/browse/YARN-9187) | Backport YARN-6852 for GPU-specific native changes to branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9180](https://issues.apache.org/jira/browse/YARN-9180) | Port YARN-7033 NM recovery of assigned resources to branch-3.0/branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9280](https://issues.apache.org/jira/browse/YARN-9280) | Backport YARN-6620 to YARN-8200/branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9174](https://issues.apache.org/jira/browse/YARN-9174) | Backport YARN-7224 for refactoring of GpuDevice class | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9289](https://issues.apache.org/jira/browse/YARN-9289) | Backport YARN-7330 for GPU in UI to branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [HDFS-14262](https://issues.apache.org/jira/browse/HDFS-14262) | [SBN read] Unclear Log.WARN message in GlobalStateIdContext | Major | hdfs | Shweta | Shweta | +| [YARN-8549](https://issues.apache.org/jira/browse/YARN-8549) | Adding a NoOp timeline writer and reader plugin classes for ATSv2 | Minor | ATSv2, timelineclient, timelineserver | Prabha Manepalli | Prabha Manepalli | +| [HADOOP-16109](https://issues.apache.org/jira/browse/HADOOP-16109) | Parquet reading S3AFileSystem causes EOF | Blocker | fs/s3 | Dave Christianson | Steve Loughran | +| [YARN-9397](https://issues.apache.org/jira/browse/YARN-9397) | Fix empty NMResourceInfo object test failures in branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [HADOOP-16191](https://issues.apache.org/jira/browse/HADOOP-16191) | AliyunOSS: improvements for copyFile/copyDirectory and logging | Major | fs/oss | wujinhu | wujinhu | +| [YARN-9271](https://issues.apache.org/jira/browse/YARN-9271) | Backport YARN-6927 for resource type support in MapReduce | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9291](https://issues.apache.org/jira/browse/YARN-9291) | Backport YARN-7637 to branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9409](https://issues.apache.org/jira/browse/YARN-9409) | Port resource type changes from YARN-7237 to branch-3.0/branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9272](https://issues.apache.org/jira/browse/YARN-9272) | Backport YARN-7738 for refreshing max allocation for multiple resource types | Major | . | Jonathan Hung | Jonathan Hung | +| [HADOOP-16205](https://issues.apache.org/jira/browse/HADOOP-16205) | Backporting ABFS driver from trunk to branch 2.0 | Major | fs/azure | Esfandiar Manii | Yuan Gao | +| [HADOOP-16269](https://issues.apache.org/jira/browse/HADOOP-16269) | ABFS: add listFileStatus with StartFrom | Major | fs/azure | Da Zhou | Da Zhou | +| [HADOOP-16306](https://issues.apache.org/jira/browse/HADOOP-16306) | AliyunOSS: Remove temporary files when upload small files to OSS | Major | fs/oss | wujinhu | wujinhu | +| [HDFS-14034](https://issues.apache.org/jira/browse/HDFS-14034) | Support getQuotaUsage API in WebHDFS | Major | fs, webhdfs | Erik Krogen | Chao Sun | +| [YARN-9775](https://issues.apache.org/jira/browse/YARN-9775) | RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore | Major | restapi | Prabhu Joseph | Prabhu Joseph | +| [HDFS-14771](https://issues.apache.org/jira/browse/HDFS-14771) | Backport HDFS-14617 to branch-2 (Improve fsimage load time by writing sub-sections to the fsimage index) | Major | namenode | Xiaoqiao He | Xiaoqiao He | +| [HDFS-14822](https://issues.apache.org/jira/browse/HDFS-14822) | [SBN read] Revisit GlobalStateIdContext locking when getting server state id | Major | hdfs | Chen Liang | Chen Liang | +| [HDFS-14785](https://issues.apache.org/jira/browse/HDFS-14785) | [SBN read] Change client logging to be less aggressive | Major | hdfs | Chen Liang | Chen Liang | +| [HDFS-14858](https://issues.apache.org/jira/browse/HDFS-14858) | [SBN read] Allow configurably enable/disable AlignmentContext on NameNode | Major | hdfs | Chen Liang | Chen Liang | +| [HDFS-12979](https://issues.apache.org/jira/browse/HDFS-12979) | StandbyNode should upload FsImage to ObserverNode after checkpointing. | Major | hdfs | Konstantin Shvachko | Chen Liang | +| [HADOOP-16630](https://issues.apache.org/jira/browse/HADOOP-16630) | Backport HADOOP-16548 - "ABFS: Config to enable/disable flush operation" to branch-2 | Minor | fs/azure | Sneha Vijayarajan | Sneha Vijayarajan | +| [HADOOP-16631](https://issues.apache.org/jira/browse/HADOOP-16631) | Backport HADOOP-16578 - "ABFS: fileSystemExists() should not call container level apis" to Branch-2 | Major | fs/azure | Sneha Vijayarajan | Sneha Vijayarajan | +| [HDFS-14162](https://issues.apache.org/jira/browse/HDFS-14162) | Balancer should work with ObserverNode | Major | . | Konstantin Shvachko | Erik Krogen | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-15149](https://issues.apache.org/jira/browse/HADOOP-15149) | CryptoOutputStream should implement StreamCapabilities | Major | fs | Mike Drob | Xiao Chen | +| [HADOOP-15177](https://issues.apache.org/jira/browse/HADOOP-15177) | Update the release year to 2018 | Blocker | build | Akira Ajisaka | Bharat Viswanadham | +| [YARN-8412](https://issues.apache.org/jira/browse/YARN-8412) | Move ResourceRequest.clone logic everywhere into a proper API | Minor | . | Botong Huang | Botong Huang | +| [HDFS-13870](https://issues.apache.org/jira/browse/HDFS-13870) | WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API doc | Minor | documentation, webhdfs | Siyao Meng | Siyao Meng | +| [HDFS-12729](https://issues.apache.org/jira/browse/HDFS-12729) | Document special paths in HDFS | Major | documentation | Christopher Douglas | Masatake Iwasaki | +| [HADOOP-15711](https://issues.apache.org/jira/browse/HADOOP-15711) | Move branch-2 precommit/nightly test builds to java 8 | Critical | . | Jonathan Hung | Jonathan Hung | +| [HDFS-14510](https://issues.apache.org/jira/browse/HDFS-14510) | Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information should be immutable) | Major | encryption, snapshots | Wei-Chiu Chuang | Siyao Meng | +| [HDFS-14585](https://issues.apache.org/jira/browse/HDFS-14585) | Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch2.9 | Major | . | Lisheng Sun | Lisheng Sun | +| [HDFS-14483](https://issues.apache.org/jira/browse/HDFS-14483) | Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9 | Major | . | Zheng Hu | Lisheng Sun | +| [YARN-9559](https://issues.apache.org/jira/browse/YARN-9559) | Create AbstractContainersLauncher for pluggable ContainersLauncher logic | Major | . | Jonathan Hung | Jonathan Hung | +| [HDFS-14725](https://issues.apache.org/jira/browse/HDFS-14725) | Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report) | Major | namenode | Wei-Chiu Chuang | Xiaoqiao He | +| [YARN-8200](https://issues.apache.org/jira/browse/YARN-8200) | Backport resource types/GPU features to branch-3.0/branch-2 | Major | . | Jonathan Hung | Jonathan Hung | +| [HADOOP-16555](https://issues.apache.org/jira/browse/HADOOP-16555) | Update commons-compress to 1.19 | Major | . | Wei-Chiu Chuang | YiSheng Lien | +| [YARN-9730](https://issues.apache.org/jira/browse/YARN-9730) | Support forcing configured partitions to be exclusive based on app node label | Major | . | Jonathan Hung | Jonathan Hung | +| [HADOOP-16544](https://issues.apache.org/jira/browse/HADOOP-16544) | update io.netty in branch-2 | Major | . | Wei-Chiu Chuang | Masatake Iwasaki | +| [HADOOP-16588](https://issues.apache.org/jira/browse/HADOOP-16588) | Update commons-beanutils version to 1.9.4 in branch-2 | Critical | . | Wei-Chiu Chuang | Wei-Chiu Chuang | + + diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/2.10.0/RELEASENOTES.2.10.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/2.10.0/RELEASENOTES.2.10.0.md index ca8949646a1d8..ebea56ba6da02 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/2.10.0/RELEASENOTES.2.10.0.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/2.10.0/RELEASENOTES.2.10.0.md @@ -16,11 +16,51 @@ # See the License for the specific language governing permissions and # limitations under the License. --> -# Apache Hadoop 2.10.0 Release Notes +# Apache Hadoop 2.10.0 Release Notes These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. +--- + +* [YARN-8200](https://issues.apache.org/jira/browse/YARN-8200) | *Major* | **Backport resource types/GPU features to branch-3.0/branch-2** + +The generic resource types feature allows admins to configure custom resource types outside of memory and CPU. Users can request these resource types which YARN will take into account for resource scheduling. + +This also adds GPU as a native resource type, built on top of the generic resource types feature. It adds support for GPU resource discovery, GPU scheduling and GPU isolation. + + +--- + +* [HDFS-12943](https://issues.apache.org/jira/browse/HDFS-12943) | *Major* | **Consistent Reads from Standby Node** + +Observer is a new type of a NameNode in addition to Active and Standby Nodes in HA settings. An Observer Node maintains a replica of the namespace same as a Standby Node. It additionally allows execution of clients read requests. + +To ensure read-after-write consistency within a single client, a state ID is introduced in RPC headers. The Observer responds to the client request only after its own state has caught up with the client’s state ID, which it previously received from the Active NameNode. + +Clients can explicitly invoke a new client protocol call msync(), which ensures that subsequent reads by this client from an Observer are consistent. + +A new client-side ObserverReadProxyProvider is introduced to provide automatic switching between Active and Observer NameNodes for submitting respectively write and read requests. + + +--- + +* [HDFS-13541](https://issues.apache.org/jira/browse/HDFS-13541) | *Major* | **NameNode Port based selective encryption** + +This feature allows HDFS to selectively enforce encryption for both RPC (NameNode) and data transfer (DataNode). With this feature enabled, NameNode can listen on multiple ports, and different ports can have different security configurations. Depending on which NameNode port clients connect to, the RPC calls and the following data transfer will enforce security configuration corresponding to this NameNode port. This can help when there is requirement to enforce different security policies depending on the location where the clients are connecting from. + +This can be enabled by setting `hadoop.security.saslproperties.resolver.class` configuration to `org.apache.hadoop.security.IngressPortBasedResolver`, and add the additional NameNode auxiliary ports by setting `dfs.namenode.rpc-address.auxiliary-ports`, and set the security individual ports by configuring `ingress.port.sasl.configured.ports`. + + +--- + +* [HDFS-14403](https://issues.apache.org/jira/browse/HDFS-14403) | *Major* | **Cost-Based RPC FairCallQueue** + +This adds an extension to the IPC FairCallQueue which allows for the consideration of the *cost* of a user's operations when deciding how they should be prioritized, as opposed to the number of operations. This can be helpful for protecting the NameNode from clients which submit very expensive operations (e.g. large listStatus operations or recursive getContentSummary operations). + +This can be enabled by setting the `ipc..costprovder.impl` configuration to `org.apache.hadoop.ipc.WeightedTimeCostProvider`. + + --- * [HDFS-12883](https://issues.apache.org/jira/browse/HDFS-12883) | *Major* | **RBF: Document Router and State Store metrics** @@ -113,3 +153,77 @@ WASB: Fix Spark process hang at shutdown due to use of non-daemon threads by upd Federation supports and controls global quota at mount table level. In a federated environment, a folder can be spread across multiple subclusters. Router aggregates quota that queried from these subclusters and uses that for the quota-verification. + + +--- + +* [HADOOP-15547](https://issues.apache.org/jira/browse/HADOOP-15547) | *Major* | **WASB: improve listStatus performance** + +WASB: listStatus 10x performance improvement for listing 700,000 files + + +--- + +* [HADOOP-16055](https://issues.apache.org/jira/browse/HADOOP-16055) | *Blocker* | **Upgrade AWS SDK to 1.11.271 in branch-2** + +This change was required to address license compatibility issues with the JSON parser in the older AWS SDKs. + +A consequence of this, where needed, the applied patch contains HADOOP-12705 Upgrade Jackson 2.2.3 to 2.7.8. + + +--- + +* [HADOOP-16053](https://issues.apache.org/jira/browse/HADOOP-16053) | *Major* | **Backport HADOOP-14816 to branch-2** + +This patch changed the default build and test environment from Ubuntu "Trusty" 14.04 to Ubuntu "Xenial" 16.04. + + +--- + +* [HDFS-14617](https://issues.apache.org/jira/browse/HDFS-14617) | *Major* | **Improve fsimage load time by writing sub-sections to the fsimage index** + +This change allows the inode and inode directory sections of the fsimage to be loaded in parallel. Tests on large images have shown this change to reduce the image load time to about 50% of the pre-change run time. + +It works by writing sub-section entries to the image index, effectively splitting each image section into many sub-sections which can be processed in parallel. By default 12 sub-sections per image section are created when the image is saved, and 4 threads are used to load the image at startup. + +This is disabled by default for any image with more than 1M inodes (dfs.image.parallel.inode.threshold) and can be enabled by setting dfs.image.parallel.load to true. When the feature is enabled, the next HDFS checkpoint will write the image sub-sections and subsequent namenode restarts can load the image in parallel. + +A image with the parallel sections can be read even if the feature is disabled, but HDFS versions without this Jira cannot load an image with parallel sections. OIV can process a parallel enabled image without issues. + +Key configuration parameters are: + +dfs.image.parallel.load=false - enable or disable the feature + +dfs.image.parallel.target.sections = 12 - The target number of subsections. Aim for 2 to 3 times the number of dfs.image.parallel.threads. + +dfs.image.parallel.inode.threshold = 1000000 - Only save and load in parallel if the image has more than this number of inodes. + +dfs.image.parallel.threads = 4 - The number of threads used to load the image. Testing has shown 4 to be optimal, but this may depends on the environment + + +--- + +* [HDFS-14771](https://issues.apache.org/jira/browse/HDFS-14771) | *Major* | **Backport HDFS-14617 to branch-2 (Improve fsimage load time by writing sub-sections to the fsimage index)** + +This change allows the inode and inode directory sections of the fsimage to be loaded in parallel. Tests on large images have shown this change to reduce the image load time to about 50% of the pre-change run time. + +It works by writing sub-section entries to the image index, effectively splitting each image section into many sub-sections which can be processed in parallel. By default 12 sub-sections per image section are created when the image is saved, and 4 threads are used to load the image at startup. + +This is disabled by default for any image with more than 1M inodes (dfs.image.parallel.inode.threshold) and can be enabled by setting dfs.image.parallel.load to true. When the feature is enabled, the next HDFS checkpoint will write the image sub-sections and subsequent namenode restarts can load the image in parallel. + +A image with the parallel sections can be read even if the feature is disabled, but HDFS versions without this Jira cannot load an image with parallel sections. OIV can process a parallel enabled image without issues. + +Key configuration parameters are: + +dfs.image.parallel.load=false - enable or disable the feature + +dfs.image.parallel.target.sections = 12 - The target number of subsections. Aim for 2 to 3 times the number of dfs.image.parallel.threads. + +dfs.image.parallel.inode.threshold = 1000000 - Only save and load in parallel if the image has more than this number of inodes. + +dfs.image.parallel.threads = 4 - The number of threads used to load the image. Testing has shown 4 to be optimal, but this may depends on the environment. + +UPGRADE WARN: +1. It can upgrade smoothly from 2.10 to 3.\* if not enable this feature ever. +2. Only path to do upgrade from 2.10 to 3.3 currently when enable fsimage parallel loading feature. +3. If someone want to upgrade 2.10 to 3.\*(3.1.\*/3.2.\*) prior release, please make sure that save at least one fsimage file after disable this feature. It relies on change configuration parameter(dfs.image.parallel.load=false) first and restart namenode before upgrade operation. diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.3/CHANGES.3.1.3.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.3/CHANGES.3.1.3.md new file mode 100644 index 0000000000000..70187e9739b9f --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.3/CHANGES.3.1.3.md @@ -0,0 +1,336 @@ + + +# Apache Hadoop Changelog + +## Release 3.1.3 - 2019-09-12 + +### INCOMPATIBLE CHANGES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-15922](https://issues.apache.org/jira/browse/HADOOP-15922) | DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL | Major | common, kms | He Xiaoqiao | He Xiaoqiao | + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-15950](https://issues.apache.org/jira/browse/HADOOP-15950) | Failover for LdapGroupsMapping | Major | common, security | Lukas Majercak | Lukas Majercak | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-15481](https://issues.apache.org/jira/browse/HADOOP-15481) | Emit FairCallQueue stats as metrics | Major | metrics, rpc-server | Erik Krogen | Christopher Gregorian | +| [HDFS-14213](https://issues.apache.org/jira/browse/HDFS-14213) | Remove Jansson from BUILDING.txt | Minor | documentation | Akira Ajisaka | Dinesh Chitlangia | +| [HDFS-14221](https://issues.apache.org/jira/browse/HDFS-14221) | Replace Guava Optional with Java Optional | Major | . | Arpit Agarwal | Arpit Agarwal | +| [HDFS-14222](https://issues.apache.org/jira/browse/HDFS-14222) | Make ThrottledAsyncChecker constructor public | Major | . | Arpit Agarwal | Arpit Agarwal | +| [HADOOP-16089](https://issues.apache.org/jira/browse/HADOOP-16089) | AliyunOSS: update oss-sdk version to 3.4.1 | Major | fs/oss | wujinhu | wujinhu | +| [HDFS-14231](https://issues.apache.org/jira/browse/HDFS-14231) | DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error | Major | hdfs | Kitti Nanasi | Kitti Nanasi | +| [YARN-7171](https://issues.apache.org/jira/browse/YARN-7171) | RM UI should sort memory / cores numerically | Major | . | Eric Maynard | Ahmed Hussein | +| [YARN-9282](https://issues.apache.org/jira/browse/YARN-9282) | Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be 'authentication' | Trivial | . | Szilard Nemeth | Charan Hebri | +| [HADOOP-16108](https://issues.apache.org/jira/browse/HADOOP-16108) | Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's | Major | . | Harshakiran Reddy | Ayush Saxena | +| [YARN-8295](https://issues.apache.org/jira/browse/YARN-8295) | [UI2] Improve "Resource Usage" tab error message when there are no data available. | Minor | yarn-ui-v2 | Gergely Novák | Charan Hebri | +| [YARN-7824](https://issues.apache.org/jira/browse/YARN-7824) | [UI2] Yarn Component Instance page should include link to container logs | Major | yarn-ui-v2 | Yesha Vora | Akhil PB | +| [HADOOP-15281](https://issues.apache.org/jira/browse/HADOOP-15281) | Distcp to add no-rename copy option | Major | tools/distcp | Steve Loughran | Andrew Olson | +| [YARN-9309](https://issues.apache.org/jira/browse/YARN-9309) | Improve graph text in SLS to avoid overlapping | Minor | . | Bilwa S T | Bilwa S T | +| [YARN-9168](https://issues.apache.org/jira/browse/YARN-9168) | DistributedShell client timeout should be -1 by default | Minor | . | Zhankun Tang | Zhankun Tang | +| [YARN-9087](https://issues.apache.org/jira/browse/YARN-9087) | Improve logging for initialization of Resource plugins | Major | yarn | Szilard Nemeth | Szilard Nemeth | +| [YARN-9121](https://issues.apache.org/jira/browse/YARN-9121) | Replace GpuDiscoverer.getInstance() to a readable object for easy access control | Major | . | Szilard Nemeth | Szilard Nemeth | +| [YARN-9139](https://issues.apache.org/jira/browse/YARN-9139) | Simplify initializer code of GpuDiscoverer | Major | . | Szilard Nemeth | Szilard Nemeth | +| [HDFS-14247](https://issues.apache.org/jira/browse/HDFS-14247) | Repeat adding node description into network topology | Minor | datanode | HuangTao | HuangTao | +| [YARN-9138](https://issues.apache.org/jira/browse/YARN-9138) | Improve test coverage for nvidia-smi binary execution of GpuDiscoverer | Major | . | Szilard Nemeth | Szilard Nemeth | +| [MAPREDUCE-7191](https://issues.apache.org/jira/browse/MAPREDUCE-7191) | JobHistoryServer should log exception when loading/parsing history file failed | Minor | mrv2 | Jiandan Yang | Jiandan Yang | +| [HDFS-14346](https://issues.apache.org/jira/browse/HDFS-14346) | Better time precision in getTimeDuration | Minor | namenode | Chao Sun | Chao Sun | +| [HDFS-14366](https://issues.apache.org/jira/browse/HDFS-14366) | Improve HDFS append performance | Major | hdfs | Chao Sun | Chao Sun | +| [MAPREDUCE-7190](https://issues.apache.org/jira/browse/MAPREDUCE-7190) | Add SleepJob additional parameter to make parallel runs distinguishable | Major | . | Adam Antal | Adam Antal | +| [HADOOP-16208](https://issues.apache.org/jira/browse/HADOOP-16208) | Do Not Log InterruptedException in Client | Minor | common | David Mollitor | David Mollitor | +| [YARN-9463](https://issues.apache.org/jira/browse/YARN-9463) | Add queueName info when failing with queue capacity sanity check | Trivial | capacity scheduler | Aihua Xu | Aihua Xu | +| [HADOOP-16227](https://issues.apache.org/jira/browse/HADOOP-16227) | Upgrade checkstyle to 8.19 | Major | build | Akira Ajisaka | Akira Ajisaka | +| [HDFS-14432](https://issues.apache.org/jira/browse/HDFS-14432) | dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml | Minor | hdfs | puleya7 | puleya7 | +| [HDFS-14463](https://issues.apache.org/jira/browse/HDFS-14463) | Add Log Level link under NameNode and DataNode Web UI Utilities dropdown | Trivial | webhdfs | Siyao Meng | Siyao Meng | +| [YARN-9529](https://issues.apache.org/jira/browse/YARN-9529) | Log correct cpu controller path on error while initializing CGroups. | Major | nodemanager | Jonathan Hung | Jonathan Hung | +| [HADOOP-16289](https://issues.apache.org/jira/browse/HADOOP-16289) | Allow extra jsvc startup option in hadoop\_start\_secure\_daemon in hadoop-functions.sh | Major | scripts | Siyao Meng | Siyao Meng | +| [HADOOP-16307](https://issues.apache.org/jira/browse/HADOOP-16307) | Intern User Name and Group Name in FileStatus | Major | fs | David Mollitor | David Mollitor | +| [HDFS-14507](https://issues.apache.org/jira/browse/HDFS-14507) | Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles | Minor | documentation | Siyao Meng | Siyao Meng | +| [HDFS-14451](https://issues.apache.org/jira/browse/HDFS-14451) | Incorrect header or version mismatch log message | Minor | ipc | David Mollitor | Shweta | +| [HDFS-14502](https://issues.apache.org/jira/browse/HDFS-14502) | keepResults option in NNThroughputBenchmark should call saveNamespace() | Major | benchmarks, hdfs | Konstantin Shvachko | Konstantin Shvachko | +| [HADOOP-16323](https://issues.apache.org/jira/browse/HADOOP-16323) | https everywhere in Maven settings | Minor | build | Akira Ajisaka | Akira Ajisaka | +| [YARN-9563](https://issues.apache.org/jira/browse/YARN-9563) | Resource report REST API could return NaN or Inf | Minor | . | Ahmed Hussein | Ahmed Hussein | +| [YARN-9545](https://issues.apache.org/jira/browse/YARN-9545) | Create healthcheck REST endpoint for ATSv2 | Major | ATSv2 | Zoltan Siegl | Zoltan Siegl | +| [HDFS-10659](https://issues.apache.org/jira/browse/HDFS-10659) | Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory | Major | ha, journal-node | Amit Anand | star | +| [HDFS-14513](https://issues.apache.org/jira/browse/HDFS-14513) | FSImage which is saving should be clean while NameNode shutdown | Major | namenode | He Xiaoqiao | He Xiaoqiao | +| [YARN-9543](https://issues.apache.org/jira/browse/YARN-9543) | [UI2] Handle ATSv2 server down or failures cases gracefully in YARN UI v2 | Major | ATSv2, yarn-ui-v2 | Zoltan Siegl | Zoltan Siegl | +| [HADOOP-16369](https://issues.apache.org/jira/browse/HADOOP-16369) | Fix zstandard shortname misspelled as zts | Major | . | Jonathan Eagles | Jonathan Eagles | +| [HDFS-14560](https://issues.apache.org/jira/browse/HDFS-14560) | Allow block replication parameters to be refreshable | Major | namenode | Stephen O'Donnell | Stephen O'Donnell | +| [HDFS-12770](https://issues.apache.org/jira/browse/HDFS-12770) | Add doc about how to disable client socket cache | Trivial | hdfs-client | Weiwei Yang | Weiwei Yang | +| [HADOOP-9157](https://issues.apache.org/jira/browse/HADOOP-9157) | Better option for curl in hadoop-auth-examples | Minor | documentation | Jingguo Yao | Andras Bokor | +| [HDFS-14340](https://issues.apache.org/jira/browse/HDFS-14340) | Lower the log level when can't get postOpAttr | Minor | nfs | Anuhan Torgonshar | Anuhan Torgonshar | +| [HADOOP-15914](https://issues.apache.org/jira/browse/HADOOP-15914) | hadoop jar command has no help argument | Major | common | Adam Antal | Adam Antal | +| [HADOOP-16156](https://issues.apache.org/jira/browse/HADOOP-16156) | [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl | Minor | . | Shweta | Shweta | +| [HADOOP-14385](https://issues.apache.org/jira/browse/HADOOP-14385) | HttpExceptionUtils#validateResponse swallows exceptions | Trivial | . | Wei-Chiu Chuang | Wei-Chiu Chuang | +| [HDFS-12564](https://issues.apache.org/jira/browse/HDFS-12564) | Add the documents of swebhdfs configurations on the client side | Major | documentation, webhdfs | Takanobu Asanuma | Takanobu Asanuma | +| [HDFS-14403](https://issues.apache.org/jira/browse/HDFS-14403) | Cost-Based RPC FairCallQueue | Major | ipc, namenode | Erik Krogen | Christopher Gregorian | +| [HADOOP-16266](https://issues.apache.org/jira/browse/HADOOP-16266) | Add more fine-grained processing time metrics to the RPC layer | Minor | ipc | Christopher Gregorian | Erik Krogen | +| [YARN-9629](https://issues.apache.org/jira/browse/YARN-9629) | Support configurable MIN\_LOG\_ROLLING\_INTERVAL | Minor | log-aggregation, nodemanager, yarn | Adam Antal | Adam Antal | +| [HDFS-13694](https://issues.apache.org/jira/browse/HDFS-13694) | Making md5 computing being in parallel with image loading | Major | . | zhouyingchao | Lisheng Sun | +| [HDFS-14632](https://issues.apache.org/jira/browse/HDFS-14632) | Reduce useless #getNumLiveDataNodes call in SafeModeMonitor | Major | namenode | He Xiaoqiao | He Xiaoqiao | +| [YARN-9573](https://issues.apache.org/jira/browse/YARN-9573) | DistributedShell cannot specify LogAggregationContext | Major | distributed-shell, log-aggregation, yarn | Adam Antal | Adam Antal | +| [YARN-9337](https://issues.apache.org/jira/browse/YARN-9337) | GPU auto-discovery script runs even when the resource is given by hand | Major | yarn | Adam Antal | Adam Antal | +| [YARN-9127](https://issues.apache.org/jira/browse/YARN-9127) | Create more tests to verify GpuDeviceInformationParser | Major | . | Szilard Nemeth | Peter Bacsko | +| [HDFS-14547](https://issues.apache.org/jira/browse/HDFS-14547) | DirectoryWithQuotaFeature.quota costs additional memory even the storage type quota is not set. | Major | . | Jinglun | Jinglun | +| [HDFS-14697](https://issues.apache.org/jira/browse/HDFS-14697) | Backport HDFS-14513 to branch-2 | Minor | namenode | He Xiaoqiao | He Xiaoqiao | +| [YARN-8045](https://issues.apache.org/jira/browse/YARN-8045) | Reduce log output from container status calls | Major | . | Shane Kumpf | Craig Condit | +| [HDFS-14693](https://issues.apache.org/jira/browse/HDFS-14693) | NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. | Minor | namenode | Xudong Cao | Xudong Cao | +| [YARN-9094](https://issues.apache.org/jira/browse/YARN-9094) | Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM | Trivial | . | Szilard Nemeth | Gergely Pollak | +| [YARN-9096](https://issues.apache.org/jira/browse/YARN-9096) | Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily | Major | . | Szilard Nemeth | Gergely Pollak | +| [YARN-9092](https://issues.apache.org/jira/browse/YARN-9092) | Create an object for cgroups mount enable and cgroups mount path as they belong together | Minor | . | Szilard Nemeth | Gergely Pollak | +| [YARN-9124](https://issues.apache.org/jira/browse/YARN-9124) | Resolve contradiction in ResourceUtils: addMandatoryResources / checkMandatoryResources work differently | Minor | . | Szilard Nemeth | Adam Antal | +| [YARN-8199](https://issues.apache.org/jira/browse/YARN-8199) | Logging fileSize of log files under NM Local Dir | Major | log-aggregation | Prabhu Joseph | Prabhu Joseph | +| [YARN-9729](https://issues.apache.org/jira/browse/YARN-9729) | [UI2] Fix error message for logs when ATSv2 is offline | Major | yarn-ui-v2 | Zoltan Siegl | Zoltan Siegl | +| [YARN-9135](https://issues.apache.org/jira/browse/YARN-9135) | NM State store ResourceMappings serialization are tested with Strings instead of real Device objects | Major | . | Szilard Nemeth | Peter Bacsko | +| [HDFS-14370](https://issues.apache.org/jira/browse/HDFS-14370) | Edit log tailing fast-path should allow for backoff | Major | namenode, qjm | Erik Krogen | Erik Krogen | +| [YARN-9442](https://issues.apache.org/jira/browse/YARN-9442) | container working directory has group read permissions | Minor | yarn | Jim Brennan | Jim Brennan | +| [HADOOP-16459](https://issues.apache.org/jira/browse/HADOOP-16459) | Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2 | Major | . | Erik Krogen | Erik Krogen | +| [HDFS-14491](https://issues.apache.org/jira/browse/HDFS-14491) | More Clarity on Namenode UI Around Blocks and Replicas | Minor | . | Alan Jackoway | Siyao Meng | +| [YARN-9140](https://issues.apache.org/jira/browse/YARN-9140) | Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager | Trivial | . | Szilard Nemeth | Peter Bacsko | +| [YARN-9488](https://issues.apache.org/jira/browse/YARN-9488) | Skip YARNFeatureNotEnabledException from ClientRMService | Minor | resourcemanager | Prabhu Joseph | Prabhu Joseph | +| [YARN-8586](https://issues.apache.org/jira/browse/YARN-8586) | Extract log aggregation related fields and methods from RMAppImpl | Major | . | Szilard Nemeth | Peter Bacsko | +| [YARN-9100](https://issues.apache.org/jira/browse/YARN-9100) | Add tests for GpuResourceAllocator and do minor code cleanup | Major | . | Szilard Nemeth | Peter Bacsko | +| [HADOOP-15246](https://issues.apache.org/jira/browse/HADOOP-15246) | SpanReceiverInfo - Prefer ArrayList over LinkedList | Trivial | common | David Mollitor | David Mollitor | +| [HADOOP-16158](https://issues.apache.org/jira/browse/HADOOP-16158) | DistCp to support checksum validation when copy blocks in parallel | Major | tools/distcp | Kai Xie | Kai Xie | +| [HDFS-14746](https://issues.apache.org/jira/browse/HDFS-14746) | Trivial test code update after HDFS-14687 | Trivial | ec | Wei-Chiu Chuang | kevin su | +| [HDFS-13709](https://issues.apache.org/jira/browse/HDFS-13709) | Report bad block to NN when transfer block encounter EIO exception | Major | datanode | Chen Zhang | Chen Zhang | +| [HDFS-14665](https://issues.apache.org/jira/browse/HDFS-14665) | HttpFS: LISTSTATUS response is missing HDFS-specific fields | Major | httpfs | Siyao Meng | Siyao Meng | +| [HDFS-14276](https://issues.apache.org/jira/browse/HDFS-14276) | [SBN read] Reduce tailing overhead | Major | ha, namenode | Wei-Chiu Chuang | Ayush Saxena | +| [HDFS-14748](https://issues.apache.org/jira/browse/HDFS-14748) | Make DataNodePeerMetrics#minOutlierDetectionSamples configurable | Major | . | Lisheng Sun | Lisheng Sun | +| [HADOOP-15998](https://issues.apache.org/jira/browse/HADOOP-15998) | Ensure jar validation works on Windows. | Blocker | build | Brian Grunkemeyer | Brian Grunkemeyer | +| [HDFS-14633](https://issues.apache.org/jira/browse/HDFS-14633) | The StorageType quota and consume in QuotaFeature is not handled for rename | Major | . | Jinglun | Jinglun | +| [YARN-9795](https://issues.apache.org/jira/browse/YARN-9795) | ClusterMetrics to include AM allocation delay | Minor | . | Fengnan Li | Fengnan Li | +| [YARN-8995](https://issues.apache.org/jira/browse/YARN-8995) | Log events info in AsyncDispatcher when event queue size cumulatively reaches a certain number every time. | Major | metrics, nodemanager, resourcemanager | zhuqi | zhuqi | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HDFS-13642](https://issues.apache.org/jira/browse/HDFS-13642) | Creating a file with block size smaller than EC policy's cell size should fail | Major | erasure-coding | Xiao Chen | Xiao Chen | +| [HADOOP-15948](https://issues.apache.org/jira/browse/HADOOP-15948) | Inconsistency in get and put syntax if filename/dirname contains space | Minor | fs | vivek kumar | Ayush Saxena | +| [HDFS-13816](https://issues.apache.org/jira/browse/HDFS-13816) | dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException | Major | namenode | Vinayakumar B | Vinayakumar B | +| [HADOOP-15966](https://issues.apache.org/jira/browse/HADOOP-15966) | Hadoop Kerberos broken on macos as java.security.krb5.realm is reset: Null realm name (601) | Major | scripts | Steve Loughran | Steve Loughran | +| [HADOOP-16028](https://issues.apache.org/jira/browse/HADOOP-16028) | Fix NetworkTopology chooseRandom function to support excluded nodes | Major | . | Sihai Ke | Sihai Ke | +| [YARN-9162](https://issues.apache.org/jira/browse/YARN-9162) | Fix TestRMAdminCLI#testHelp | Major | resourcemanager, test | Ayush Saxena | Ayush Saxena | +| [HADOOP-16031](https://issues.apache.org/jira/browse/HADOOP-16031) | TestSecureLogins#testValidKerberosName fails | Major | security | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-16016](https://issues.apache.org/jira/browse/HADOOP-16016) | TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds | Major | security, test | Jason Lowe | Akira Ajisaka | +| [HDFS-14198](https://issues.apache.org/jira/browse/HDFS-14198) | Upload and Create button doesn't get enabled after getting reset. | Major | . | Ayush Saxena | Ayush Saxena | +| [YARN-9203](https://issues.apache.org/jira/browse/YARN-9203) | Fix typos in yarn-default.xml | Trivial | documentation | Rahul Padmanabhan | Rahul Padmanabhan | +| [HDFS-14207](https://issues.apache.org/jira/browse/HDFS-14207) | ZKFC should catch exception when ha configuration missing | Major | hdfs | Fei Hui | Fei Hui | +| [HDFS-14218](https://issues.apache.org/jira/browse/HDFS-14218) | EC: Ls -e throw NPE when directory ec policy is disabled | Major | . | Surendra Singh Lilhore | Ayush Saxena | +| [YARN-9210](https://issues.apache.org/jira/browse/YARN-9210) | RM nodes web page can not display node info | Blocker | yarn | Jiandan Yang | Jiandan Yang | +| [YARN-8961](https://issues.apache.org/jira/browse/YARN-8961) | [UI2] Flow Run End Time shows 'Invalid date' | Major | . | Charan Hebri | Akhil PB | +| [YARN-7088](https://issues.apache.org/jira/browse/YARN-7088) | Add application launch time to Resource Manager REST API | Major | . | Abdullah Yousufi | Kanwaljeet Sachdev | +| [YARN-9222](https://issues.apache.org/jira/browse/YARN-9222) | Print launchTime in ApplicationSummary | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-8901](https://issues.apache.org/jira/browse/YARN-8901) | Restart "NEVER" policy does not work with component dependency | Critical | . | Yesha Vora | Suma Shivaprasad | +| [YARN-9237](https://issues.apache.org/jira/browse/YARN-9237) | NM should ignore sending finished apps to RM during RM fail-over | Major | yarn | Jiandan Yang | Jiandan Yang | +| [YARN-6616](https://issues.apache.org/jira/browse/YARN-6616) | YARN AHS shows submitTime for jobs same as startTime | Minor | . | Prabhu Joseph | Prabhu Joseph | +| [YARN-9099](https://issues.apache.org/jira/browse/YARN-9099) | GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way | Major | . | Szilard Nemeth | Szilard Nemeth | +| [HADOOP-16086](https://issues.apache.org/jira/browse/HADOOP-16086) | Backport HADOOP-15549 to branch-3.1 | Major | metrics | Yuming Wang | Todd Lipcon | +| [YARN-9206](https://issues.apache.org/jira/browse/YARN-9206) | RMServerUtils does not count SHUTDOWN as an accepted state | Major | . | Kuhu Shukla | Kuhu Shukla | +| [HADOOP-16096](https://issues.apache.org/jira/browse/HADOOP-16096) | HADOOP-15281/distcp -Xdirect needs to use commons-logging on 3.1 | Critical | . | Eric Payne | Steve Loughran | +| [HDFS-14140](https://issues.apache.org/jira/browse/HDFS-14140) | JournalNodeSyncer authentication is failing in secure cluster | Major | journal-node, security | Surendra Singh Lilhore | Surendra Singh Lilhore | +| [YARN-9257](https://issues.apache.org/jira/browse/YARN-9257) | Distributed Shell client throws a NPE for a non-existent queue | Major | distributed-shell | Charan Hebri | Charan Hebri | +| [YARN-8761](https://issues.apache.org/jira/browse/YARN-8761) | Service AM support for decommissioning component instances | Major | . | Billie Rinaldi | Billie Rinaldi | +| [HDFS-14266](https://issues.apache.org/jira/browse/HDFS-14266) | EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. | Major | . | Harshakiran Reddy | Ayush Saxena | +| [HDFS-14274](https://issues.apache.org/jira/browse/HDFS-14274) | EC: NPE While Listing EC Policy For A Directory Following Replication Policy. | Major | erasure-coding | Souryakanta Dwivedy | Ayush Saxena | +| [HDFS-14263](https://issues.apache.org/jira/browse/HDFS-14263) | Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream() | Major | datanode | Surendra Singh Lilhore | Surendra Singh Lilhore | +| [YARN-7761](https://issues.apache.org/jira/browse/YARN-7761) | [UI2] Clicking 'master container log' or 'Link' next to 'log' under application's appAttempt goes to Old UI's Log link | Major | yarn-ui-v2 | Sumana Sathish | Akhil PB | +| [YARN-9295](https://issues.apache.org/jira/browse/YARN-9295) | [UI2] Fix label typo in Cluster Overview page | Trivial | yarn-ui-v2 | Charan Hebri | Charan Hebri | +| [YARN-9284](https://issues.apache.org/jira/browse/YARN-9284) | Fix the unit of yarn.service.am-resource.memory in the document | Minor | documentation, yarn-native-services | Masahiro Tanaka | Masahiro Tanaka | +| [YARN-9283](https://issues.apache.org/jira/browse/YARN-9283) | Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference | Minor | documentation | Szilard Nemeth | Adam Antal | +| [YARN-9286](https://issues.apache.org/jira/browse/YARN-9286) | [Timeline Server] Sorting based on FinalStatus shows pop-up message | Minor | timelineserver | Nallasivan | Bilwa S T | +| [HDFS-14081](https://issues.apache.org/jira/browse/HDFS-14081) | hdfs dfsadmin -metasave metasave\_test results NPE | Major | hdfs | Shweta | Shweta | +| [HADOOP-15813](https://issues.apache.org/jira/browse/HADOOP-15813) | Enable more reliable SSL connection reuse | Major | common | Daryn Sharp | Daryn Sharp | +| [HADOOP-16105](https://issues.apache.org/jira/browse/HADOOP-16105) | WASB in secure mode does not set connectingUsingSAS | Major | fs/azure | Steve Loughran | Steve Loughran | +| [YARN-9238](https://issues.apache.org/jira/browse/YARN-9238) | Avoid allocating opportunistic containers to previous/removed/non-exist application attempt | Critical | . | lujie | lujie | +| [YARN-9118](https://issues.apache.org/jira/browse/YARN-9118) | Handle exceptions with parsing user defined GPU devices in GpuDiscoverer | Major | . | Szilard Nemeth | Szilard Nemeth | +| [YARN-9317](https://issues.apache.org/jira/browse/YARN-9317) | Avoid repeated YarnConfiguration#timelineServiceV2Enabled check | Major | . | Bibin A Chundatt | Prabhu Joseph | +| [YARN-9213](https://issues.apache.org/jira/browse/YARN-9213) | RM Web UI v1 does not show custom resource allocations for containers page | Major | . | Szilard Nemeth | Szilard Nemeth | +| [YARN-9248](https://issues.apache.org/jira/browse/YARN-9248) | RMContainerImpl:Invalid event: ACQUIRED at KILLED | Major | . | lujie | lujie | +| [HADOOP-16018](https://issues.apache.org/jira/browse/HADOOP-16018) | DistCp won't reassemble chunks when blocks per chunk \> 0 | Major | tools/distcp | Kai Xie | Kai Xie | +| [YARN-9334](https://issues.apache.org/jira/browse/YARN-9334) | YARN Service Client does not work with SPNEGO when knox is configured | Major | yarn-native-services | Tarun Parimi | Billie Rinaldi | +| [HDFS-14305](https://issues.apache.org/jira/browse/HDFS-14305) | Serial number in BlockTokenSecretManager could overlap between different namenodes | Major | namenode, security | Chao Sun | He Xiaoqiao | +| [HDFS-14314](https://issues.apache.org/jira/browse/HDFS-14314) | fullBlockReportLeaseId should be reset after registering to NN | Critical | datanode | star | star | +| [YARN-8803](https://issues.apache.org/jira/browse/YARN-8803) | [UI2] Show flow runs in the order of recently created time in graph widgets | Major | yarn-ui-v2 | Akhil PB | Akhil PB | +| [HADOOP-16114](https://issues.apache.org/jira/browse/HADOOP-16114) | NetUtils#canonicalizeHost gives different value for same host | Minor | net | Praveen Krishna | Praveen Krishna | +| [HDFS-14317](https://issues.apache.org/jira/browse/HDFS-14317) | Standby does not trigger edit log rolling when in-progress edit log tailing is enabled | Critical | . | Ekanth Sethuramalingam | Ekanth Sethuramalingam | +| [HDFS-14333](https://issues.apache.org/jira/browse/HDFS-14333) | Datanode fails to start if any disk has errors during Namenode registration | Major | datanode | Stephen O'Donnell | Stephen O'Donnell | +| [HADOOP-16192](https://issues.apache.org/jira/browse/HADOOP-16192) | CallQueue backoff bug fixes: doesn't perform backoff when add() is used, and doesn't update backoff when refreshed | Major | ipc | Erik Krogen | Erik Krogen | +| [HDFS-14037](https://issues.apache.org/jira/browse/HDFS-14037) | Fix SSLFactory truststore reloader thread leak in URLConnectionFactory | Major | hdfs-client, webhdfs | Takanobu Asanuma | Takanobu Asanuma | +| [HADOOP-16225](https://issues.apache.org/jira/browse/HADOOP-16225) | Fix links to the developer mailing lists in DownstreamDev.md | Minor | documentation | Akira Ajisaka | Wanqiang Ji | +| [HADOOP-16232](https://issues.apache.org/jira/browse/HADOOP-16232) | Fix errors in the checkstyle configration xmls | Major | build | Akira Ajisaka | Wanqiang Ji | +| [HDFS-14389](https://issues.apache.org/jira/browse/HDFS-14389) | getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured | Major | namenode | Stephen O'Donnell | Stephen O'Donnell | +| [HDFS-14407](https://issues.apache.org/jira/browse/HDFS-14407) | Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes | Minor | . | Wanqiang Ji | Wanqiang Ji | +| [YARN-9413](https://issues.apache.org/jira/browse/YARN-9413) | Queue resource leak after app fail for CapacityScheduler | Major | capacityscheduler | Tao Yang | Tao Yang | +| [HADOOP-14544](https://issues.apache.org/jira/browse/HADOOP-14544) | DistCp documentation for command line options is misaligned. | Minor | documentation | Chris Nauroth | Masatake Iwasaki | +| [HDFS-10477](https://issues.apache.org/jira/browse/HDFS-10477) | Stop decommission a rack of DataNodes caused NameNode fail over to standby | Major | namenode | yunjiong zhao | yunjiong zhao | +| [YARN-6695](https://issues.apache.org/jira/browse/YARN-6695) | Race condition in RM for publishing container events vs appFinished events causes NPE | Critical | . | Rohith Sharma K S | Prabhu Joseph | +| [YARN-8622](https://issues.apache.org/jira/browse/YARN-8622) | NodeManager native build fails due to getgrouplist not found on macOS | Major | nodemanager | Ewan Higgs | Siyao Meng | +| [HADOOP-16265](https://issues.apache.org/jira/browse/HADOOP-16265) | Configuration#getTimeDuration is not consistent between default value and manual settings. | Major | . | star | star | +| [YARN-9307](https://issues.apache.org/jira/browse/YARN-9307) | node\_partitions constraint does not work | Major | . | kyungwan nam | kyungwan nam | +| [HDFS-13677](https://issues.apache.org/jira/browse/HDFS-13677) | Dynamic refresh Disk configuration results in overwriting VolumeMap | Blocker | . | xuzq | xuzq | +| [YARN-9285](https://issues.apache.org/jira/browse/YARN-9285) | RM UI progress column is of wrong type | Minor | yarn | Ahmed Hussein | Ahmed Hussein | +| [HADOOP-16278](https://issues.apache.org/jira/browse/HADOOP-16278) | With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die | Major | common, hadoop-aws, metrics | Rajat Khandelwal | Rajat Khandelwal | +| [YARN-9504](https://issues.apache.org/jira/browse/YARN-9504) | [UI2] Fair scheduler queue view page does not show actual capacity | Major | fairscheduler, yarn-ui-v2 | Zoltan Siegl | Zoltan Siegl | +| [YARN-9519](https://issues.apache.org/jira/browse/YARN-9519) | TFile log aggregation file format is not working for yarn.log-aggregation.TFile.remote-app-log-dir config | Major | log-aggregation | Adam Antal | Adam Antal | +| [HADOOP-16247](https://issues.apache.org/jira/browse/HADOOP-16247) | NPE in FsUrlConnection | Major | hdfs-client | Karthik Palanisamy | Karthik Palanisamy | +| [HADOOP-16248](https://issues.apache.org/jira/browse/HADOOP-16248) | MutableQuantiles leak memory under heavy load | Major | metrics | Alexis Daboville | Alexis Daboville | +| [HDFS-14323](https://issues.apache.org/jira/browse/HDFS-14323) | Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path | Major | webhdfs | Srinivasu Majeti | Srinivasu Majeti | +| [MAPREDUCE-7205](https://issues.apache.org/jira/browse/MAPREDUCE-7205) | Treat container scheduler kill exit code as a task attempt killing event | Major | applicationmaster, mr-am, mrv2 | Wanqiang Ji | Wanqiang Ji | +| [HDFS-14500](https://issues.apache.org/jira/browse/HDFS-14500) | NameNode StartupProgress continues to report edit log segments after the LOADING\_EDITS phase is finished | Major | namenode | Erik Krogen | Erik Krogen | +| [HADOOP-16331](https://issues.apache.org/jira/browse/HADOOP-16331) | Fix ASF License check in pom.xml | Major | . | Wanqiang Ji | Akira Ajisaka | +| [YARN-9542](https://issues.apache.org/jira/browse/YARN-9542) | Fix LogsCLI guessAppOwner ignores custom file format suffix | Minor | log-aggregation | Prabhu Joseph | Prabhu Joseph | +| [HDFS-14512](https://issues.apache.org/jira/browse/HDFS-14512) | ONE\_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes) | Major | . | Shen Yinjie | Ayush Saxena | +| [HADOOP-16334](https://issues.apache.org/jira/browse/HADOOP-16334) | Fix yetus-wrapper not working when HADOOP\_YETUS\_VERSION \>= 0.9.0 | Major | yetus | Wanqiang Ji | Wanqiang Ji | +| [HDFS-14521](https://issues.apache.org/jira/browse/HDFS-14521) | Suppress setReplication logging. | Major | . | Kihwal Lee | Kihwal Lee | +| [YARN-9507](https://issues.apache.org/jira/browse/YARN-9507) | Fix NPE in NodeManager#serviceStop on startup failure | Minor | . | Bilwa S T | Bilwa S T | +| [YARN-8947](https://issues.apache.org/jira/browse/YARN-8947) | [UI2] Active User info missing from UI2 | Major | yarn-ui-v2 | Akhil PB | Akhil PB | +| [YARN-8906](https://issues.apache.org/jira/browse/YARN-8906) | [UI2] NM hostnames not displayed correctly in Node Heatmap Chart | Major | . | Charan Hebri | Akhil PB | +| [YARN-8625](https://issues.apache.org/jira/browse/YARN-8625) | Aggregate Resource Allocation for each job is not present in ATS | Major | ATSv2 | Prabhu Joseph | Prabhu Joseph | +| [HADOOP-16345](https://issues.apache.org/jira/browse/HADOOP-16345) | Potential NPE when instantiating FairCallQueue metrics | Major | ipc | Erik Krogen | Erik Krogen | +| [YARN-9594](https://issues.apache.org/jira/browse/YARN-9594) | Fix missing break statement in ContainerScheduler#handle | Major | . | lujie | lujie | +| [YARN-9565](https://issues.apache.org/jira/browse/YARN-9565) | RMAppImpl#ranNodes not cleared on FinalTransition | Major | . | Bibin A Chundatt | Bilwa S T | +| [YARN-9547](https://issues.apache.org/jira/browse/YARN-9547) | ContainerStatusPBImpl default execution type is not returned | Major | . | Bibin A Chundatt | Bilwa S T | +| [HDFS-13231](https://issues.apache.org/jira/browse/HDFS-13231) | Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI | Major | datanode, namenode | Haibo Yan | Stephen O'Donnell | +| [YARN-9621](https://issues.apache.org/jira/browse/YARN-9621) | FIX TestDSWithMultipleNodeManager.testDistributedShellWithPlacementConstraint on branch-3.1 | Major | distributed-shell, test | Peter Bacsko | Prabhu Joseph | +| [HDFS-14535](https://issues.apache.org/jira/browse/HDFS-14535) | The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read | Major | hdfs-client | Zheng Hu | Zheng Hu | +| [HDFS-13730](https://issues.apache.org/jira/browse/HDFS-13730) | BlockReaderRemote.sendReadResult throws NPE | Major | hdfs-client | Wei-Chiu Chuang | Yuanbo Liu | +| [YARN-9584](https://issues.apache.org/jira/browse/YARN-9584) | Should put initializeProcessTrees method call before get pid | Critical | nodemanager | Wanqiang Ji | Wanqiang Ji | +| [HDFS-14010](https://issues.apache.org/jira/browse/HDFS-14010) | Pass correct DF usage to ReservedSpaceCalculator builder | Minor | . | Lukas Majercak | Lukas Majercak | +| [HDFS-14078](https://issues.apache.org/jira/browse/HDFS-14078) | Admin helper fails to prettify NullPointerExceptions | Major | . | Elek, Marton | Elek, Marton | +| [HDFS-14101](https://issues.apache.org/jira/browse/HDFS-14101) | Random failure of testListCorruptFilesCorruptedBlock | Major | test | Kihwal Lee | Zsolt Venczel | +| [HDFS-14465](https://issues.apache.org/jira/browse/HDFS-14465) | When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. | Major | . | Yicong Cai | Yicong Cai | +| [HDFS-12487](https://issues.apache.org/jira/browse/HDFS-12487) | FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers | Major | balancer & mover, diskbalancer | liumi | liumi | +| [HDFS-14074](https://issues.apache.org/jira/browse/HDFS-14074) | DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. | Major | hdfs | guangyi lu | guangyi lu | +| [HDFS-14541](https://issues.apache.org/jira/browse/HDFS-14541) | When evictableMmapped or evictable size is zero, do not throw NoSuchElementException | Major | hdfs-client, performance | Zheng Hu | Lisheng Sun | +| [HDFS-14598](https://issues.apache.org/jira/browse/HDFS-14598) | Findbugs warning caused by HDFS-12487 | Minor | diskbalancer | Wei-Chiu Chuang | He Xiaoqiao | +| [YARN-9639](https://issues.apache.org/jira/browse/YARN-9639) | DecommissioningNodesWatcher cause memory leak | Blocker | . | Bibin A Chundatt | Bilwa S T | +| [YARN-9327](https://issues.apache.org/jira/browse/YARN-9327) | Improve synchronisation in ProtoUtils#convertToProtoFormat block | Critical | . | Bibin A Chundatt | Bibin A Chundatt | +| [YARN-9655](https://issues.apache.org/jira/browse/YARN-9655) | AllocateResponse in FederationInterceptor lost applicationPriority | Major | federation | hunshenshi | hunshenshi | +| [HADOOP-16385](https://issues.apache.org/jira/browse/HADOOP-16385) | Namenode crashes with "RedundancyMonitor thread received Runtime exception" | Major | . | krishna reddy | Ayush Saxena | +| [YARN-9644](https://issues.apache.org/jira/browse/YARN-9644) | First RMContext object is always leaked during switch over | Blocker | . | Bibin A Chundatt | Bibin A Chundatt | +| [HDFS-14629](https://issues.apache.org/jira/browse/HDFS-14629) | Property value Hard Coded in DNConf.java | Trivial | . | hemanthboyina | hemanthboyina | +| [YARN-9557](https://issues.apache.org/jira/browse/YARN-9557) | Application fails in diskchecker when ReadWriteDiskValidator is configured. | Critical | nodemanager | Anuruddh Nayak | Bilwa S T | +| [HDFS-12703](https://issues.apache.org/jira/browse/HDFS-12703) | Exceptions are fatal to decommissioning monitor | Critical | namenode | Daryn Sharp | He Xiaoqiao | +| [HDFS-12748](https://issues.apache.org/jira/browse/HDFS-12748) | NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY | Major | hdfs | Jiandan Yang | Weiwei Yang | +| [YARN-9625](https://issues.apache.org/jira/browse/YARN-9625) | UI2 - No link to a queue on the Queues page for Fair Scheduler | Major | . | Charan Hebri | Zoltan Siegl | +| [HDFS-14466](https://issues.apache.org/jira/browse/HDFS-14466) | Add a regression test for HDFS-14323 | Minor | fs, test, webhdfs | Yuya Ebihara | Masatake Iwasaki | +| [YARN-9235](https://issues.apache.org/jira/browse/YARN-9235) | If linux container executor is not set for a GPU cluster GpuResourceHandlerImpl is not initialized and NPE is thrown | Major | yarn | Antal Bálint Steinbach | Adam Antal | +| [YARN-9626](https://issues.apache.org/jira/browse/YARN-9626) | UI2 - Fair scheduler queue apps page issues | Major | . | Charan Hebri | Zoltan Siegl | +| [YARN-9682](https://issues.apache.org/jira/browse/YARN-9682) | Wrong log message when finalizing the upgrade | Trivial | . | kyungwan nam | kyungwan nam | +| [HADOOP-16440](https://issues.apache.org/jira/browse/HADOOP-16440) | Distcp can not preserve timestamp with -delete option | Major | . | ludun | ludun | +| [MAPREDUCE-7076](https://issues.apache.org/jira/browse/MAPREDUCE-7076) | TestNNBench#testNNBenchCreateReadAndDelete failing in our internal build | Minor | test | Rushabh S Shah | kevin su | +| [YARN-9668](https://issues.apache.org/jira/browse/YARN-9668) | UGI conf doesn't read user overridden configurations on RM and NM startup | Major | . | Jonathan Hung | Jonathan Hung | +| [HADOOP-9844](https://issues.apache.org/jira/browse/HADOOP-9844) | NPE when trying to create an error message response of SASL RPC | Major | ipc | Steve Loughran | Steve Loughran | +| [HADOOP-16245](https://issues.apache.org/jira/browse/HADOOP-16245) | Enabling SSL within LdapGroupsMapping can break system SSL configs | Major | common, security | Erik Krogen | Erik Krogen | +| [HDFS-14429](https://issues.apache.org/jira/browse/HDFS-14429) | Block remain in COMMITTED but not COMPLETE caused by Decommission | Major | . | Yicong Cai | Yicong Cai | +| [HADOOP-16435](https://issues.apache.org/jira/browse/HADOOP-16435) | RpcMetrics should not be retained forever | Critical | rpc-server | Zoltan Haindrich | Zoltan Haindrich | +| [YARN-9596](https://issues.apache.org/jira/browse/YARN-9596) | QueueMetrics has incorrect metrics when labelled partitions are involved | Major | capacity scheduler | Muhammad Samir Khan | Muhammad Samir Khan | +| [MAPREDUCE-7225](https://issues.apache.org/jira/browse/MAPREDUCE-7225) | Fix broken current folder expansion during MR job start | Major | mrv2 | Adam Antal | Peter Bacsko | +| [HDFS-13529](https://issues.apache.org/jira/browse/HDFS-13529) | Fix default trash policy emptier trigger time correctly | Major | namenode | He Xiaoqiao | He Xiaoqiao | +| [HADOOP-15681](https://issues.apache.org/jira/browse/HADOOP-15681) | AuthenticationFilter should generate valid date format for Set-Cookie header regardless of default Locale | Minor | security | Cao Manh Dat | Cao Manh Dat | +| [HDFS-14685](https://issues.apache.org/jira/browse/HDFS-14685) | DefaultAuditLogger doesn't print CallerContext | Major | hdfs | xuzq | xuzq | +| [HDFS-14462](https://issues.apache.org/jira/browse/HDFS-14462) | WebHDFS throws "Error writing request body to server" instead of DSQuotaExceededException | Major | webhdfs | Erik Krogen | Simbarashe Dzinamarira | +| [HDFS-14557](https://issues.apache.org/jira/browse/HDFS-14557) | JournalNode error: Can't scan a pre-transactional edit log | Major | ha | Wei-Chiu Chuang | Stephen O'Donnell | +| [HDFS-14692](https://issues.apache.org/jira/browse/HDFS-14692) | Upload button should not encode complete url | Major | . | Lokesh Jain | Lokesh Jain | +| [HDFS-14631](https://issues.apache.org/jira/browse/HDFS-14631) | The DirectoryScanner doesn't fix the wrongly placed replica. | Major | . | Jinglun | Jinglun | +| [YARN-9685](https://issues.apache.org/jira/browse/YARN-9685) | NPE when rendering the info table of leaf queue in non-accessible partitions | Major | capacityscheduler | Tao Yang | Tao Yang | +| [HDFS-14459](https://issues.apache.org/jira/browse/HDFS-14459) | ClosedChannelException silently ignored in FsVolumeList.addBlockPool() | Major | datanode | Stephen O'Donnell | Stephen O'Donnell | +| [HDFS-13359](https://issues.apache.org/jira/browse/HDFS-13359) | DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream | Major | datanode | Yiqun Lin | Yiqun Lin | +| [YARN-9451](https://issues.apache.org/jira/browse/YARN-9451) | AggregatedLogsBlock shows wrong NM http port | Minor | nodemanager | Prabhu Joseph | Prabhu Joseph | +| [YARN-9723](https://issues.apache.org/jira/browse/YARN-9723) | ApplicationPlacementContext is not required for terminated jobs during recovery | Major | resourcemanager | Prabhu Joseph | Prabhu Joseph | +| [HDFS-12914](https://issues.apache.org/jira/browse/HDFS-12914) | Block report leases cause missing blocks until next report | Critical | namenode | Daryn Sharp | Santosh Marella | +| [HDFS-14148](https://issues.apache.org/jira/browse/HDFS-14148) | HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory | Major | hdfs | Siyao Meng | Siyao Meng | +| [HDFS-14595](https://issues.apache.org/jira/browse/HDFS-14595) | HDFS-11848 breaks API compatibility | Blocker | . | Wei-Chiu Chuang | Siyao Meng | +| [HDFS-14423](https://issues.apache.org/jira/browse/HDFS-14423) | Percent (%) and plus (+) characters no longer work in WebHDFS | Major | webhdfs | Jing Wang | Masatake Iwasaki | +| [MAPREDUCE-7230](https://issues.apache.org/jira/browse/MAPREDUCE-7230) | TestHSWebApp.testLogsViewSingle fails | Major | jobhistoryserver, test | Prabhu Joseph | Prabhu Joseph | +| [HDFS-14687](https://issues.apache.org/jira/browse/HDFS-14687) | Standby Namenode never come out of safemode when EC files are being written. | Critical | ec, namenode | Surendra Singh Lilhore | Surendra Singh Lilhore | +| [HDFS-13101](https://issues.apache.org/jira/browse/HDFS-13101) | Yet another fsimage corruption related to snapshot | Major | snapshots | Yongjun Zhang | Shashikant Banerjee | +| [HDFS-13201](https://issues.apache.org/jira/browse/HDFS-13201) | Fix prompt message in testPolicyAndStateCantBeNull | Minor | . | chencan | chencan | +| [HDFS-14311](https://issues.apache.org/jira/browse/HDFS-14311) | Multi-threading conflict at layoutVersion when loading block pool storage | Major | rolling upgrades | Yicong Cai | Yicong Cai | +| [HDFS-14582](https://issues.apache.org/jira/browse/HDFS-14582) | Failed to start DN with ArithmeticException when NULL checksum used | Major | datanode | Surendra Singh Lilhore | Surendra Singh Lilhore | +| [HADOOP-16494](https://issues.apache.org/jira/browse/HADOOP-16494) | Add SHA-256 or SHA-512 checksum to release artifacts to comply with the release distribution policy | Blocker | build | Akira Ajisaka | Akira Ajisaka | +| [YARN-9774](https://issues.apache.org/jira/browse/YARN-9774) | Fix order of arguments for assertEquals in TestSLSUtils | Minor | test | Nikhil Navadiya | Nikhil Navadiya | +| [HDFS-13596](https://issues.apache.org/jira/browse/HDFS-13596) | NN restart fails after RollingUpgrade from 2.x to 3.x | Blocker | hdfs | Hanisha Koneru | Fei Hui | +| [HDFS-14396](https://issues.apache.org/jira/browse/HDFS-14396) | Failed to load image from FSImageFile when downgrade from 3.x to 2.x | Blocker | rolling upgrades | Fei Hui | Fei Hui | +| [YARN-9642](https://issues.apache.org/jira/browse/YARN-9642) | Fix Memory Leak in AbstractYarnScheduler caused by timer | Blocker | resourcemanager | Bibin A Chundatt | Bibin A Chundatt | +| [HDFS-13977](https://issues.apache.org/jira/browse/HDFS-13977) | NameNode can kill itself if it tries to send too many txns to a QJM simultaneously | Major | namenode, qjm | Erik Krogen | Erik Krogen | +| [YARN-9438](https://issues.apache.org/jira/browse/YARN-9438) | launchTime not written to state store for running applications | Major | . | Jonathan Hung | Jonathan Hung | +| [HDFS-12212](https://issues.apache.org/jira/browse/HDFS-12212) | Options.Rename.To\_TRASH is considered even when Options.Rename.NONE is specified | Major | namenode | Vinayakumar B | Vinayakumar B | +| [HDFS-8178](https://issues.apache.org/jira/browse/HDFS-8178) | QJM doesn't move aside stale inprogress edits files | Major | qjm | Zhe Zhang | Istvan Fajth | +| [HDFS-14706](https://issues.apache.org/jira/browse/HDFS-14706) | Checksums are not checked if block meta file is less than 7 bytes | Major | . | Stephen O'Donnell | Stephen O'Donnell | +| [YARN-9797](https://issues.apache.org/jira/browse/YARN-9797) | LeafQueue#activateApplications should use resourceCalculator#fitsIn | Blocker | . | Bibin A Chundatt | Bilwa S T | +| [YARN-9785](https://issues.apache.org/jira/browse/YARN-9785) | Fix DominantResourceCalculator when one resource is zero | Blocker | . | Bilwa S T | Bilwa S T | +| [YARN-9817](https://issues.apache.org/jira/browse/YARN-9817) | Fix failing testcases due to not initialized AsyncDispatcher - ArithmeticException: / by zero | Major | test | Prabhu Joseph | Prabhu Joseph | + + +### TESTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [YARN-9315](https://issues.apache.org/jira/browse/YARN-9315) | TestCapacitySchedulerMetrics fails intermittently | Minor | capacity scheduler | Prabhu Joseph | Prabhu Joseph | +| [YARN-9316](https://issues.apache.org/jira/browse/YARN-9316) | TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently | Minor | capacity scheduler | Prabhu Joseph | Prabhu Joseph | +| [YARN-9325](https://issues.apache.org/jira/browse/YARN-9325) | TestQueueManagementDynamicEditPolicy fails intermittent | Minor | capacity scheduler | Prabhu Joseph | Prabhu Joseph | +| [HDFS-11950](https://issues.apache.org/jira/browse/HDFS-11950) | Disable libhdfs zerocopy test on Mac | Minor | libhdfs | John Zhuge | Akira Ajisaka | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-16045](https://issues.apache.org/jira/browse/HADOOP-16045) | Don't run TestDU on Windows | Trivial | common, test | Lukas Majercak | Lukas Majercak | +| [HADOOP-16079](https://issues.apache.org/jira/browse/HADOOP-16079) | Token.toString faulting if any token listed can't load. | Blocker | security | Steve Loughran | Steve Loughran | +| [YARN-9253](https://issues.apache.org/jira/browse/YARN-9253) | Add UT to verify Placement Constraint in Distributed Shell | Major | . | Prabhu Joseph | Prabhu Joseph | +| [YARN-9293](https://issues.apache.org/jira/browse/YARN-9293) | Optimize MockAMLauncher event handling | Major | . | Bibin A Chundatt | Bibin A Chundatt | +| [HADOOP-16109](https://issues.apache.org/jira/browse/HADOOP-16109) | Parquet reading S3AFileSystem causes EOF | Blocker | fs/s3 | Dave Christianson | Steve Loughran | +| [HADOOP-16191](https://issues.apache.org/jira/browse/HADOOP-16191) | AliyunOSS: improvements for copyFile/copyDirectory and logging | Major | fs/oss | wujinhu | wujinhu | +| [YARN-9391](https://issues.apache.org/jira/browse/YARN-9391) | Disable PATH variable to be passed to Docker container | Major | . | Eric Yang | Jim Brennan | +| [HADOOP-16220](https://issues.apache.org/jira/browse/HADOOP-16220) | Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project | Major | . | Gabor Bota | Gabor Bota | +| [HADOOP-16233](https://issues.apache.org/jira/browse/HADOOP-16233) | S3AFileStatus to declare that isEncrypted() is always true | Minor | fs/s3 | Steve Loughran | Steve Loughran | +| [HADOOP-16306](https://issues.apache.org/jira/browse/HADOOP-16306) | AliyunOSS: Remove temporary files when upload small files to OSS | Major | fs/oss | wujinhu | wujinhu | +| [HDFS-14553](https://issues.apache.org/jira/browse/HDFS-14553) | Make queue size of BlockReportProcessingThread configurable | Major | namenode | He Xiaoqiao | He Xiaoqiao | +| [HDFS-14034](https://issues.apache.org/jira/browse/HDFS-14034) | Support getQuotaUsage API in WebHDFS | Major | fs, webhdfs | Erik Krogen | Chao Sun | +| [YARN-9765](https://issues.apache.org/jira/browse/YARN-9765) | SLS runner crashes when run with metrics turned off. | Major | . | Abhishek Modi | Abhishek Modi | +| [HDFS-14674](https://issues.apache.org/jira/browse/HDFS-14674) | [SBN read] Got an unexpected txid when tail editlog | Blocker | . | wangzhaohui | wangzhaohui | +| [YARN-9775](https://issues.apache.org/jira/browse/YARN-9775) | RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore | Major | restapi | Prabhu Joseph | Prabhu Joseph | +| [HDFS-14779](https://issues.apache.org/jira/browse/HDFS-14779) | Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns | Major | . | Jonathan Hung | Jonathan Hung | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-16025](https://issues.apache.org/jira/browse/HADOOP-16025) | Update the year to 2019 | Major | build | Ayush Saxena | Ayush Saxena | +| [HDFS-12729](https://issues.apache.org/jira/browse/HDFS-12729) | Document special paths in HDFS | Major | documentation | Chris Douglas | Masatake Iwasaki | +| [YARN-9191](https://issues.apache.org/jira/browse/YARN-9191) | Add cli option in DS to support enforceExecutionType in resource requests. | Major | . | Abhishek Modi | Abhishek Modi | +| [HADOOP-16263](https://issues.apache.org/jira/browse/HADOOP-16263) | Update BUILDING.txt with macOS native build instructions | Minor | . | Siyao Meng | Siyao Meng | +| [YARN-9559](https://issues.apache.org/jira/browse/YARN-9559) | Create AbstractContainersLauncher for pluggable ContainersLauncher logic | Major | . | Jonathan Hung | Jonathan Hung | +| [HADOOP-16551](https://issues.apache.org/jira/browse/HADOOP-16551) | The changelog\*.md seems not generated when create-release | Blocker | . | Zhankun Tang | | + + diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.3/RELEASENOTES.3.1.3.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.3/RELEASENOTES.3.1.3.md new file mode 100644 index 0000000000000..c806810ae98f8 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.3/RELEASENOTES.3.1.3.md @@ -0,0 +1,59 @@ + + +# Apache Hadoop 3.1.3 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HADOOP-15922](https://issues.apache.org/jira/browse/HADOOP-15922) | *Major* | **DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL** + +- Fix DelegationTokenAuthentication filter for incorrectly double encode doAs user parameter. + + +--- + +* [YARN-8761](https://issues.apache.org/jira/browse/YARN-8761) | *Major* | **Service AM support for decommissioning component instances** + +- Component instance number is not linear increment when decommission feature is used. Application with assumption of linear increment component instance number maybe impacted by introduction of this feature. + + +--- + +* [HDFS-14305](https://issues.apache.org/jira/browse/HDFS-14305) | *Major* | **Serial number in BlockTokenSecretManager could overlap between different namenodes** + +NameNodes rely on independent block token key ranges to communicate block token identities to DataNodes and clients in a way that does not create conflicts between the tokens issued by multiple NameNodes. HDFS-6440 introduced the potential for overlaps in key ranges; this fixes the issue by creating 64 possible key ranges that NameNodes assign themselves to, allowing for up to 64 NameNodes to run safely. This limitation only applies within a single Namespace; there may be more than 64 NameNodes total spread among multiple federated Namespaces. + + +--- + +* [HADOOP-16114](https://issues.apache.org/jira/browse/HADOOP-16114) | *Minor* | **NetUtils#canonicalizeHost gives different value for same host** + +The above patch will resolve the race condition + + +--- + +* [HDFS-14396](https://issues.apache.org/jira/browse/HDFS-14396) | *Blocker* | **Failed to load image from FSImageFile when downgrade from 3.x to 2.x** + +During a rolling upgrade from Hadoop 2.x to 3.x, NameNode cannot persist erasure coding information, and therefore a user cannot start using erasure coding feature until finalize is done. + + + diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/CHANGELOG.3.2.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/CHANGELOG.3.2.1.md new file mode 100644 index 0000000000000..64e249ee36431 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/CHANGELOG.3.2.1.md @@ -0,0 +1,553 @@ + + +# Apache Hadoop Changelog + +## Release 3.2.1 - 2019-09-10 + +### INCOMPATIBLE CHANGES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-15922](https://issues.apache.org/jira/browse/HADOOP-15922) | DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL | Major | common, kms | He Xiaoqiao | He Xiaoqiao | + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-15950](https://issues.apache.org/jira/browse/HADOOP-15950) | Failover for LdapGroupsMapping | Major | common, security | Lukas Majercak | Lukas Majercak | +| [YARN-7055](https://issues.apache.org/jira/browse/YARN-7055) | YARN Timeline Service v.2: beta 1 / GA | Major | timelineclient, timelinereader, timelineserver | Vrushali C | | +| [YARN-9761](https://issues.apache.org/jira/browse/YARN-9761) | Allow overriding application submissions based on server side configs | Major | . | Jonathan Hung | pralabhkumar | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-15676](https://issues.apache.org/jira/browse/HADOOP-15676) | Cleanup TestSSLHttpServer | Minor | common | Szilard Nemeth | Szilard Nemeth | +| [YARN-8896](https://issues.apache.org/jira/browse/YARN-8896) | Limit the maximum number of container assignments per heartbeat | Major | . | Weiwei Yang | Zhankun Tang | +| [YARN-8618](https://issues.apache.org/jira/browse/YARN-8618) | Yarn Service: When all the components of a service have restart policy NEVER then initiation of service upgrade should fail | Major | . | Chandni Singh | Chandni Singh | +| [HADOOP-15804](https://issues.apache.org/jira/browse/HADOOP-15804) | upgrade to commons-compress 1.18 | Major | . | PJ Fanning | Akira Ajisaka | +| [YARN-8916](https://issues.apache.org/jira/browse/YARN-8916) | Define a constant "docker" string in "ContainerRuntimeConstants.java" for better maintainability | Minor | . | Zhankun Tang | Zhankun Tang | +| [YARN-8908](https://issues.apache.org/jira/browse/YARN-8908) | Fix errors in yarn-default.xml related to GPU/FPGA | Major | . | Zhankun Tang | Zhankun Tang | +| [HDFS-13941](https://issues.apache.org/jira/browse/HDFS-13941) | make storageId in BlockPoolTokenSecretManager.checkAccess optional | Major | . | Ajay Kumar | Ajay Kumar | +| [HDFS-14029](https://issues.apache.org/jira/browse/HDFS-14029) | Sleep in TestLazyPersistFiles should be put into a loop | Trivial | hdfs | Adam Antal | Adam Antal | +| [YARN-8915](https://issues.apache.org/jira/browse/YARN-8915) | Update the doc about the default value of "maximum-container-assignments" for capacity scheduler | Minor | . | Zhankun Tang | Zhankun Tang | +| [HADOOP-15855](https://issues.apache.org/jira/browse/HADOOP-15855) | Review hadoop credential doc, including object store details | Minor | documentation, security | Steve Loughran | Steve Loughran | +| [YARN-7225](https://issues.apache.org/jira/browse/YARN-7225) | Add queue and partition info to RM audit log | Major | resourcemanager | Jonathan Hung | Eric Payne | +| [HADOOP-15687](https://issues.apache.org/jira/browse/HADOOP-15687) | Credentials class should allow access to aliases | Trivial | . | Lars Francke | Lars Francke | +| [YARN-8969](https://issues.apache.org/jira/browse/YARN-8969) | AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting | Major | . | Wanqiang Ji | Wanqiang Ji | +| [YARN-8977](https://issues.apache.org/jira/browse/YARN-8977) | Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode | Trivial | . | Wanqiang Ji | Wanqiang Ji | +| [HDFS-14070](https://issues.apache.org/jira/browse/HDFS-14070) | Refactor NameNodeWebHdfsMethods to allow better extensibility | Major | . | CR Hota | CR Hota | +| [HADOOP-15926](https://issues.apache.org/jira/browse/HADOOP-15926) | Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK | Minor | documentation | Akira Ajisaka | Dinesh Chitlangia | +| [HADOOP-12558](https://issues.apache.org/jira/browse/HADOOP-12558) | distcp documentation is woefully out of date | Critical | documentation, tools/distcp | Allen Wittenauer | Dinesh Chitlangia | +| [HDFS-14063](https://issues.apache.org/jira/browse/HDFS-14063) | Support noredirect param for CREATE/APPEND/OPEN/GETFILECHECKSUM in HttpFS | Major | . | Íñigo Goiri | Íñigo Goiri | +| [HADOOP-15919](https://issues.apache.org/jira/browse/HADOOP-15919) | AliyunOSS: Enable Yarn to use OSS | Major | fs/oss | wujinhu | wujinhu | +| [HDFS-14064](https://issues.apache.org/jira/browse/HDFS-14064) | WEBHDFS: Support Enable/Disable EC Policy | Major | . | Ayush Saxena | Ayush Saxena | +| [HADOOP-15943](https://issues.apache.org/jira/browse/HADOOP-15943) | AliyunOSS: add missing owner & group attributes for oss FileStatus | Major | fs/oss | wujinhu | wujinhu | +| [MAPREDUCE-7164](https://issues.apache.org/jira/browse/MAPREDUCE-7164) | FileOutputCommitter does not report progress while merging paths. | Major | . | Kuhu Shukla | Kuhu Shukla | +| [YARN-9069](https://issues.apache.org/jira/browse/YARN-9069) | Fix SchedulerInfo#getSchedulerType for custom schedulers | Minor | . | Bilwa S T | Bilwa S T | +| [HDFS-14095](https://issues.apache.org/jira/browse/HDFS-14095) | EC: Track Erasure Coding commands in DFS statistics | Major | erasure-coding | Ayush Saxena | Ayush Saxena | +| [HDFS-14112](https://issues.apache.org/jira/browse/HDFS-14112) | Avoid recursive call to external authorizer for getContentSummary. | Critical | namenode | Jitendra Nath Pandey | Tsz Wo Nicholas Sze | +| [YARN-9036](https://issues.apache.org/jira/browse/YARN-9036) | Escape newlines in health report in YARN UI | Major | . | Jonathan Hung | Keqiu Hu | +| [YARN-9041](https://issues.apache.org/jira/browse/YARN-9041) | Performance Optimization of method FSPreemptionThread#identifyContainersToPreempt | Major | fairscheduler, scheduler preemption | Wanqiang Ji | Wanqiang Ji | +| [YARN-9085](https://issues.apache.org/jira/browse/YARN-9085) | Add Guaranteed and MaxCapacity to CSQueueMetrics | Major | . | Jonathan Hung | Jonathan Hung | +| [HDFS-14124](https://issues.apache.org/jira/browse/HDFS-14124) | EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs | Major | erasure-coding, httpfs, webhdfs | Souryakanta Dwivedy | Ayush Saxena | +| [HADOOP-15808](https://issues.apache.org/jira/browse/HADOOP-15808) | Harden Token service loader use | Major | security | Steve Loughran | Steve Loughran | +| [YARN-9122](https://issues.apache.org/jira/browse/YARN-9122) | Add table of contents to YARN Service API document | Minor | documentation | Akira Ajisaka | Zhankun Tang | +| [HDFS-14171](https://issues.apache.org/jira/browse/HDFS-14171) | Performance improvement in Tailing EditLog | Major | namenode | Kenneth Yang | Kenneth Yang | +| [HADOOP-15481](https://issues.apache.org/jira/browse/HADOOP-15481) | Emit FairCallQueue stats as metrics | Major | metrics, rpc-server | Erik Krogen | Christopher Gregorian | +| [HADOOP-15994](https://issues.apache.org/jira/browse/HADOOP-15994) | Upgrade Jackson2 to 2.9.8 | Major | security | Akira Ajisaka | lqjacklee | +| [HADOOP-16019](https://issues.apache.org/jira/browse/HADOOP-16019) | ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration | Minor | common | luhuachao | luhuachao | +| [HDFS-14213](https://issues.apache.org/jira/browse/HDFS-14213) | Remove Jansson from BUILDING.txt | Minor | documentation | Akira Ajisaka | Dinesh Chitlangia | +| [HDFS-14221](https://issues.apache.org/jira/browse/HDFS-14221) | Replace Guava Optional with Java Optional | Major | . | Arpit Agarwal | Arpit Agarwal | +| [HDFS-14222](https://issues.apache.org/jira/browse/HDFS-14222) | Make ThrottledAsyncChecker constructor public | Major | . | Arpit Agarwal | Arpit Agarwal | +| [HADOOP-16075](https://issues.apache.org/jira/browse/HADOOP-16075) | Upgrade checkstyle version to 8.16 | Minor | build | Dinesh Chitlangia | Dinesh Chitlangia | +| [HADOOP-16089](https://issues.apache.org/jira/browse/HADOOP-16089) | AliyunOSS: update oss-sdk version to 3.4.1 | Major | fs/oss | wujinhu | wujinhu | +| [HDFS-14231](https://issues.apache.org/jira/browse/HDFS-14231) | DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error | Major | hdfs | Kitti Nanasi | Kitti Nanasi | +| [YARN-7171](https://issues.apache.org/jira/browse/YARN-7171) | RM UI should sort memory / cores numerically | Major | . | Eric Maynard | Ahmed Hussein | +| [YARN-9282](https://issues.apache.org/jira/browse/YARN-9282) | Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be 'authentication' | Trivial | . | Szilard Nemeth | Charan Hebri | +| [HADOOP-16108](https://issues.apache.org/jira/browse/HADOOP-16108) | Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's | Major | . | Harshakiran Reddy | Ayush Saxena | +| [YARN-8295](https://issues.apache.org/jira/browse/YARN-8295) | [UI2] Improve "Resource Usage" tab error message when there are no data available. | Minor | yarn-ui-v2 | Gergely Novák | Charan Hebri | +| [YARN-7824](https://issues.apache.org/jira/browse/YARN-7824) | [UI2] Yarn Component Instance page should include link to container logs | Major | yarn-ui-v2 | Yesha Vora | Akhil PB | +| [HADOOP-15281](https://issues.apache.org/jira/browse/HADOOP-15281) | Distcp to add no-rename copy option | Major | tools/distcp | Steve Loughran | Andrew Olson | +| [YARN-9309](https://issues.apache.org/jira/browse/YARN-9309) | Improve graph text in SLS to avoid overlapping | Minor | . | Bilwa S T | Bilwa S T | +| [HDFS-14235](https://issues.apache.org/jira/browse/HDFS-14235) | Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon | Major | . | Surendra Singh Lilhore | Ranith Sardar | +| [YARN-9168](https://issues.apache.org/jira/browse/YARN-9168) | DistributedShell client timeout should be -1 by default | Minor | . | Zhankun Tang | Zhankun Tang | +| [YARN-9087](https://issues.apache.org/jira/browse/YARN-9087) | Improve logging for initialization of Resource plugins | Major | yarn | Szilard Nemeth | Szilard Nemeth | +| [YARN-9121](https://issues.apache.org/jira/browse/YARN-9121) | Replace GpuDiscoverer.getInstance() to a readable object for easy access control | Major | . | Szilard Nemeth | Szilard Nemeth | +| [YARN-9139](https://issues.apache.org/jira/browse/YARN-9139) | Simplify initializer code of GpuDiscoverer | Major | . | Szilard Nemeth | Szilard Nemeth | +| [HDFS-14247](https://issues.apache.org/jira/browse/HDFS-14247) | Repeat adding node description into network topology | Minor | datanode | HuangTao | HuangTao | +| [YARN-9332](https://issues.apache.org/jira/browse/YARN-9332) | RackResolver tool should accept multiple hosts | Minor | yarn | Lantao Jin | Lantao Jin | +| [HADOOP-16140](https://issues.apache.org/jira/browse/HADOOP-16140) | hadoop fs expunge to add -immediate option to purge trash immediately | Major | fs | Stephen O'Donnell | Stephen O'Donnell | +| [YARN-9138](https://issues.apache.org/jira/browse/YARN-9138) | Improve test coverage for nvidia-smi binary execution of GpuDiscoverer | Major | . | Szilard Nemeth | Szilard Nemeth | +| [MAPREDUCE-7191](https://issues.apache.org/jira/browse/MAPREDUCE-7191) | JobHistoryServer should log exception when loading/parsing history file failed | Minor | mrv2 | Jiandan Yang | Jiandan Yang | +| [MAPREDUCE-7192](https://issues.apache.org/jira/browse/MAPREDUCE-7192) | JobHistoryServer attempts page support jump to containers log page in NM when logAggregation is disable | Major | mrv2 | Jiandan Yang | Jiandan Yang | +| [HDFS-14346](https://issues.apache.org/jira/browse/HDFS-14346) | Better time precision in getTimeDuration | Minor | namenode | Chao Sun | Chao Sun | +| [HDFS-14366](https://issues.apache.org/jira/browse/HDFS-14366) | Improve HDFS append performance | Major | hdfs | Chao Sun | Chao Sun | +| [HADOOP-16196](https://issues.apache.org/jira/browse/HADOOP-16196) | Path Parameterize Comparable | Minor | common | David Mollitor | David Mollitor | +| [HADOOP-16181](https://issues.apache.org/jira/browse/HADOOP-16181) | HadoopExecutors shutdown Cleanup | Minor | util | David Mollitor | David Mollitor | +| [HADOOP-16147](https://issues.apache.org/jira/browse/HADOOP-16147) | Allow CopyListing sequence file keys and values to be more easily customized | Major | tools/distcp | Andrew Olson | Andrew Olson | +| [MAPREDUCE-7190](https://issues.apache.org/jira/browse/MAPREDUCE-7190) | Add SleepJob additional parameter to make parallel runs distinguishable | Major | . | Adam Antal | Adam Antal | +| [YARN-9394](https://issues.apache.org/jira/browse/YARN-9394) | Use new API of RackResolver to get better performance | Major | yarn | Lantao Jin | Lantao Jin | +| [HADOOP-16208](https://issues.apache.org/jira/browse/HADOOP-16208) | Do Not Log InterruptedException in Client | Minor | common | David Mollitor | David Mollitor | +| [YARN-9463](https://issues.apache.org/jira/browse/YARN-9463) | Add queueName info when failing with queue capacity sanity check | Trivial | capacity scheduler | Aihua Xu | Aihua Xu | +| [HADOOP-16227](https://issues.apache.org/jira/browse/HADOOP-16227) | Upgrade checkstyle to 8.19 | Major | build | Akira Ajisaka | Akira Ajisaka | +| [HDFS-14432](https://issues.apache.org/jira/browse/HDFS-14432) | dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml | Minor | hdfs | puleya7 | puleya7 | +| [HDFS-14463](https://issues.apache.org/jira/browse/HDFS-14463) | Add Log Level link under NameNode and DataNode Web UI Utilities dropdown | Trivial | webhdfs | Siyao Meng | Siyao Meng | +| [YARN-9529](https://issues.apache.org/jira/browse/YARN-9529) | Log correct cpu controller path on error while initializing CGroups. | Major | nodemanager | Jonathan Hung | Jonathan Hung | +| [HADOOP-16289](https://issues.apache.org/jira/browse/HADOOP-16289) | Allow extra jsvc startup option in hadoop\_start\_secure\_daemon in hadoop-functions.sh | Major | scripts | Siyao Meng | Siyao Meng | +| [HADOOP-16307](https://issues.apache.org/jira/browse/HADOOP-16307) | Intern User Name and Group Name in FileStatus | Major | fs | David Mollitor | David Mollitor | +| [HADOOP-16294](https://issues.apache.org/jira/browse/HADOOP-16294) | Enable access to context by DistCp subclasses | Trivial | tools/distcp | Andrew Olson | Andrew Olson | +| [HDFS-14507](https://issues.apache.org/jira/browse/HDFS-14507) | Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles | Minor | documentation | Siyao Meng | Siyao Meng | +| [HDFS-14451](https://issues.apache.org/jira/browse/HDFS-14451) | Incorrect header or version mismatch log message | Minor | ipc | David Mollitor | Shweta | +| [HDFS-14502](https://issues.apache.org/jira/browse/HDFS-14502) | keepResults option in NNThroughputBenchmark should call saveNamespace() | Major | benchmarks, hdfs | Konstantin Shvachko | Konstantin Shvachko | +| [HADOOP-16323](https://issues.apache.org/jira/browse/HADOOP-16323) | https everywhere in Maven settings | Minor | build | Akira Ajisaka | Akira Ajisaka | +| [YARN-9563](https://issues.apache.org/jira/browse/YARN-9563) | Resource report REST API could return NaN or Inf | Minor | . | Ahmed Hussein | Ahmed Hussein | +| [YARN-9545](https://issues.apache.org/jira/browse/YARN-9545) | Create healthcheck REST endpoint for ATSv2 | Major | ATSv2 | Zoltan Siegl | Zoltan Siegl | +| [HDFS-10659](https://issues.apache.org/jira/browse/HDFS-10659) | Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory | Major | ha, journal-node | Amit Anand | star | +| [HDFS-14513](https://issues.apache.org/jira/browse/HDFS-14513) | FSImage which is saving should be clean while NameNode shutdown | Major | namenode | He Xiaoqiao | He Xiaoqiao | +| [YARN-9543](https://issues.apache.org/jira/browse/YARN-9543) | [UI2] Handle ATSv2 server down or failures cases gracefully in YARN UI v2 | Major | ATSv2, yarn-ui-v2 | Zoltan Siegl | Zoltan Siegl | +| [HADOOP-16369](https://issues.apache.org/jira/browse/HADOOP-16369) | Fix zstandard shortname misspelled as zts | Major | . | Jonathan Eagles | Jonathan Eagles | +| [HDFS-14560](https://issues.apache.org/jira/browse/HDFS-14560) | Allow block replication parameters to be refreshable | Major | namenode | Stephen O'Donnell | Stephen O'Donnell | +| [HDFS-12770](https://issues.apache.org/jira/browse/HDFS-12770) | Add doc about how to disable client socket cache | Trivial | hdfs-client | Weiwei Yang | Weiwei Yang | +| [HADOOP-9157](https://issues.apache.org/jira/browse/HADOOP-9157) | Better option for curl in hadoop-auth-examples | Minor | documentation | Jingguo Yao | Andras Bokor | +| [HDFS-14340](https://issues.apache.org/jira/browse/HDFS-14340) | Lower the log level when can't get postOpAttr | Minor | nfs | Anuhan Torgonshar | Anuhan Torgonshar | +| [HADOOP-15914](https://issues.apache.org/jira/browse/HADOOP-15914) | hadoop jar command has no help argument | Major | common | Adam Antal | Adam Antal | +| [YARN-9630](https://issues.apache.org/jira/browse/YARN-9630) | [UI2] Add a link in docs's top page | Major | documentation, yarn-ui-v2 | Wanqiang Ji | Wanqiang Ji | +| [HADOOP-16156](https://issues.apache.org/jira/browse/HADOOP-16156) | [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl | Minor | . | Shweta | Shweta | +| [HADOOP-14385](https://issues.apache.org/jira/browse/HADOOP-14385) | HttpExceptionUtils#validateResponse swallows exceptions | Trivial | . | Wei-Chiu Chuang | Wei-Chiu Chuang | +| [HDFS-12564](https://issues.apache.org/jira/browse/HDFS-12564) | Add the documents of swebhdfs configurations on the client side | Major | documentation, webhdfs | Takanobu Asanuma | Takanobu Asanuma | +| [HDFS-14403](https://issues.apache.org/jira/browse/HDFS-14403) | Cost-Based RPC FairCallQueue | Major | ipc, namenode | Erik Krogen | Christopher Gregorian | +| [HADOOP-16266](https://issues.apache.org/jira/browse/HADOOP-16266) | Add more fine-grained processing time metrics to the RPC layer | Minor | ipc | Christopher Gregorian | Erik Krogen | +| [YARN-9629](https://issues.apache.org/jira/browse/YARN-9629) | Support configurable MIN\_LOG\_ROLLING\_INTERVAL | Minor | log-aggregation, nodemanager, yarn | Adam Antal | Adam Antal | +| [HDFS-13694](https://issues.apache.org/jira/browse/HDFS-13694) | Making md5 computing being in parallel with image loading | Major | . | zhouyingchao | Lisheng Sun | +| [HDFS-14632](https://issues.apache.org/jira/browse/HDFS-14632) | Reduce useless #getNumLiveDataNodes call in SafeModeMonitor | Major | namenode | He Xiaoqiao | He Xiaoqiao | +| [YARN-9573](https://issues.apache.org/jira/browse/YARN-9573) | DistributedShell cannot specify LogAggregationContext | Major | distributed-shell, log-aggregation, yarn | Adam Antal | Adam Antal | +| [YARN-9337](https://issues.apache.org/jira/browse/YARN-9337) | GPU auto-discovery script runs even when the resource is given by hand | Major | yarn | Adam Antal | Adam Antal | +| [YARN-9127](https://issues.apache.org/jira/browse/YARN-9127) | Create more tests to verify GpuDeviceInformationParser | Major | . | Szilard Nemeth | Peter Bacsko | +| [YARN-9326](https://issues.apache.org/jira/browse/YARN-9326) | Fair Scheduler configuration defaults are not documented in case of min and maxResources | Major | docs, documentation, fairscheduler, yarn | Adam Antal | Adam Antal | +| [HDFS-14547](https://issues.apache.org/jira/browse/HDFS-14547) | DirectoryWithQuotaFeature.quota costs additional memory even the storage type quota is not set. | Major | . | Jinglun | Jinglun | +| [HDFS-14693](https://issues.apache.org/jira/browse/HDFS-14693) | NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. | Minor | namenode | Xudong Cao | Xudong Cao | +| [YARN-9094](https://issues.apache.org/jira/browse/YARN-9094) | Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM | Trivial | . | Szilard Nemeth | Gergely Pollak | +| [YARN-9096](https://issues.apache.org/jira/browse/YARN-9096) | Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily | Major | . | Szilard Nemeth | Gergely Pollak | +| [YARN-9092](https://issues.apache.org/jira/browse/YARN-9092) | Create an object for cgroups mount enable and cgroups mount path as they belong together | Minor | . | Szilard Nemeth | Gergely Pollak | +| [YARN-9124](https://issues.apache.org/jira/browse/YARN-9124) | Resolve contradiction in ResourceUtils: addMandatoryResources / checkMandatoryResources work differently | Minor | . | Szilard Nemeth | Adam Antal | +| [YARN-8199](https://issues.apache.org/jira/browse/YARN-8199) | Logging fileSize of log files under NM Local Dir | Major | log-aggregation | Prabhu Joseph | Prabhu Joseph | +| [YARN-9729](https://issues.apache.org/jira/browse/YARN-9729) | [UI2] Fix error message for logs when ATSv2 is offline | Major | yarn-ui-v2 | Zoltan Siegl | Zoltan Siegl | +| [YARN-9135](https://issues.apache.org/jira/browse/YARN-9135) | NM State store ResourceMappings serialization are tested with Strings instead of real Device objects | Major | . | Szilard Nemeth | Peter Bacsko | +| [HDFS-14370](https://issues.apache.org/jira/browse/HDFS-14370) | Edit log tailing fast-path should allow for backoff | Major | namenode, qjm | Erik Krogen | Erik Krogen | +| [YARN-9442](https://issues.apache.org/jira/browse/YARN-9442) | container working directory has group read permissions | Minor | yarn | Jim Brennan | Jim Brennan | +| [HADOOP-16459](https://issues.apache.org/jira/browse/HADOOP-16459) | Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2 | Major | . | Erik Krogen | Erik Krogen | +| [HDFS-14491](https://issues.apache.org/jira/browse/HDFS-14491) | More Clarity on Namenode UI Around Blocks and Replicas | Minor | . | Alan Jackoway | Siyao Meng | +| [YARN-9134](https://issues.apache.org/jira/browse/YARN-9134) | No test coverage for redefining FPGA / GPU resource types in TestResourceUtils | Major | . | Szilard Nemeth | Peter Bacsko | +| [YARN-9133](https://issues.apache.org/jira/browse/YARN-9133) | Make tests more easy to comprehend in TestGpuResourceHandler | Major | . | Szilard Nemeth | Peter Bacsko | +| [YARN-9140](https://issues.apache.org/jira/browse/YARN-9140) | Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager | Trivial | . | Szilard Nemeth | Peter Bacsko | +| [YARN-9676](https://issues.apache.org/jira/browse/YARN-9676) | Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected classes | Major | . | Adam Antal | Adam Antal | +| [YARN-9488](https://issues.apache.org/jira/browse/YARN-9488) | Skip YARNFeatureNotEnabledException from ClientRMService | Minor | resourcemanager | Prabhu Joseph | Prabhu Joseph | +| [YARN-8586](https://issues.apache.org/jira/browse/YARN-8586) | Extract log aggregation related fields and methods from RMAppImpl | Major | . | Szilard Nemeth | Peter Bacsko | +| [YARN-9100](https://issues.apache.org/jira/browse/YARN-9100) | Add tests for GpuResourceAllocator and do minor code cleanup | Major | . | Szilard Nemeth | Peter Bacsko | +| [HADOOP-15246](https://issues.apache.org/jira/browse/HADOOP-15246) | SpanReceiverInfo - Prefer ArrayList over LinkedList | Trivial | common | David Mollitor | David Mollitor | +| [HADOOP-16158](https://issues.apache.org/jira/browse/HADOOP-16158) | DistCp to support checksum validation when copy blocks in parallel | Major | tools/distcp | Kai Xie | Kai Xie | +| [HDFS-14746](https://issues.apache.org/jira/browse/HDFS-14746) | Trivial test code update after HDFS-14687 | Trivial | ec | Wei-Chiu Chuang | kevin su | +| [HDFS-13709](https://issues.apache.org/jira/browse/HDFS-13709) | Report bad block to NN when transfer block encounter EIO exception | Major | datanode | Chen Zhang | Chen Zhang | +| [HDFS-14665](https://issues.apache.org/jira/browse/HDFS-14665) | HttpFS: LISTSTATUS response is missing HDFS-specific fields | Major | httpfs | Siyao Meng | Siyao Meng | +| [HDFS-14276](https://issues.apache.org/jira/browse/HDFS-14276) | [SBN read] Reduce tailing overhead | Major | ha, namenode | Wei-Chiu Chuang | Ayush Saxena | +| [HADOOP-16061](https://issues.apache.org/jira/browse/HADOOP-16061) | Update Apache Yetus to 0.10.0 | Major | build | Akira Ajisaka | Akira Ajisaka | +| [YARN-9756](https://issues.apache.org/jira/browse/YARN-9756) | Create metric that sums total memory/vcores preempted per round | Major | capacity scheduler | Eric Payne | Manikandan R | +| [HDFS-14748](https://issues.apache.org/jira/browse/HDFS-14748) | Make DataNodePeerMetrics#minOutlierDetectionSamples configurable | Major | . | Lisheng Sun | Lisheng Sun | +| [HADOOP-15998](https://issues.apache.org/jira/browse/HADOOP-15998) | Ensure jar validation works on Windows. | Blocker | build | Brian Grunkemeyer | Brian Grunkemeyer | +| [HDFS-14633](https://issues.apache.org/jira/browse/HDFS-14633) | The StorageType quota and consume in QuotaFeature is not handled for rename | Major | . | Jinglun | Jinglun | +| [YARN-9810](https://issues.apache.org/jira/browse/YARN-9810) | Add queue capacity/maxcapacity percentage metrics | Major | . | Jonathan Hung | Shubham Gupta | +| [YARN-9763](https://issues.apache.org/jira/browse/YARN-9763) | Print application tags in application summary | Major | . | Jonathan Hung | Manoj Kumar | +| [YARN-9795](https://issues.apache.org/jira/browse/YARN-9795) | ClusterMetrics to include AM allocation delay | Minor | . | Fengnan Li | Fengnan Li | +| [YARN-8995](https://issues.apache.org/jira/browse/YARN-8995) | Log events info in AsyncDispatcher when event queue size cumulatively reaches a certain number every time. | Major | metrics, nodemanager, resourcemanager | zhuqi | zhuqi | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HDFS-13973](https://issues.apache.org/jira/browse/HDFS-13973) | getErasureCodingPolicy should log path in audit event | Major | hdfs | Shweta | Shweta | +| [YARN-8868](https://issues.apache.org/jira/browse/YARN-8868) | Set HTTPOnly attribute to Cookie | Major | . | Chandni Singh | Chandni Singh | +| [HDFS-14003](https://issues.apache.org/jira/browse/HDFS-14003) | Fix findbugs warning in trunk for FSImageFormatPBINode | Major | . | Yiqun Lin | Yiqun Lin | +| [YARN-8910](https://issues.apache.org/jira/browse/YARN-8910) | Misleading log statement in NM when max retries is -1 | Minor | . | Chandni Singh | Chandni Singh | +| [YARN-7502](https://issues.apache.org/jira/browse/YARN-7502) | Nodemanager restart docs should describe nodemanager supervised property | Major | documentation | Jason Lowe | Suma Shivaprasad | +| [YARN-8826](https://issues.apache.org/jira/browse/YARN-8826) | Fix lingering timeline collector after serviceStop in TimelineCollectorManager | Trivial | ATSv2 | Prabha Manepalli | Prabha Manepalli | +| [HDFS-14021](https://issues.apache.org/jira/browse/HDFS-14021) | TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently | Major | erasure-coding, test | Xiao Chen | Xiao Chen | +| [MAPREDUCE-7151](https://issues.apache.org/jira/browse/MAPREDUCE-7151) | RMContainerAllocator#handleJobPriorityChange expects application\_priority always | Major | . | Bibin A Chundatt | Bilwa S T | +| [HDFS-14028](https://issues.apache.org/jira/browse/HDFS-14028) | HDFS OIV temporary dir deletes folder | Major | hdfs | Adam Antal | Adam Antal | +| [HDFS-14027](https://issues.apache.org/jira/browse/HDFS-14027) | DFSStripedOutputStream should implement both hsync methods | Critical | erasure-coding | Xiao Chen | Xiao Chen | +| [HADOOP-15899](https://issues.apache.org/jira/browse/HADOOP-15899) | Update AWS Java SDK versions in NOTICE.txt | Major | . | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-15900](https://issues.apache.org/jira/browse/HADOOP-15900) | Update JSch versions in LICENSE.txt | Major | . | Akira Ajisaka | Akira Ajisaka | +| [HDFS-14042](https://issues.apache.org/jira/browse/HDFS-14042) | Fix NPE when PROVIDED storage is missing | Major | . | Íñigo Goiri | Virajith Jalaparti | +| [HDFS-14043](https://issues.apache.org/jira/browse/HDFS-14043) | Tolerate corrupted seen\_txid file | Major | hdfs, namenode | Lukas Majercak | Lukas Majercak | +| [YARN-8970](https://issues.apache.org/jira/browse/YARN-8970) | Improve the debug message in CS#allocateContainerOnSingleNode | Trivial | . | Weiwei Yang | Zhankun Tang | +| [YARN-8865](https://issues.apache.org/jira/browse/YARN-8865) | RMStateStore contains large number of expired RMDelegationToken | Major | resourcemanager | Wilfred Spiegelenburg | Wilfred Spiegelenburg | +| [HDFS-14048](https://issues.apache.org/jira/browse/HDFS-14048) | DFSOutputStream close() throws exception on subsequent call after DataNode restart | Major | hdfs-client | Erik Krogen | Erik Krogen | +| [MAPREDUCE-7156](https://issues.apache.org/jira/browse/MAPREDUCE-7156) | NullPointerException when reaching max shuffle connections | Major | mrv2 | Peter Bacsko | Peter Bacsko | +| [YARN-8866](https://issues.apache.org/jira/browse/YARN-8866) | Fix a parsing error for crossdomain.xml | Major | build, yarn-ui-v2 | Takanobu Asanuma | Takanobu Asanuma | +| [HDFS-14039](https://issues.apache.org/jira/browse/HDFS-14039) | ec -listPolicies doesn't show correct state for the default policy when the default is not RS(6,3) | Major | erasure-coding | Xiao Chen | Kitti Nanasi | +| [HADOOP-15916](https://issues.apache.org/jira/browse/HADOOP-15916) | Upgrade Maven Surefire plugin to 3.0.0-M1 | Blocker | build | Akira Ajisaka | Akira Ajisaka | +| [YARN-9002](https://issues.apache.org/jira/browse/YARN-9002) | YARN Service keytab does not support s3, wasb, gs and is restricted to HDFS and local filesystem only | Major | yarn-native-services | Gour Saha | Gour Saha | +| [YARN-8233](https://issues.apache.org/jira/browse/YARN-8233) | NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null | Critical | capacityscheduler | Tao Yang | Tao Yang | +| [HDFS-14065](https://issues.apache.org/jira/browse/HDFS-14065) | Failed Storage Locations shows nothing in the Datanode Volume Failures | Major | . | Ayush Saxena | Ayush Saxena | +| [HADOOP-15923](https://issues.apache.org/jira/browse/HADOOP-15923) | create-release script should set max-cache-ttl as well as default-cache-ttl for gpg-agent | Blocker | build | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-15912](https://issues.apache.org/jira/browse/HADOOP-15912) | start-build-env.sh still creates an invalid /etc/sudoers.d/hadoop-build-${USER\_ID} file entry after HADOOP-15802 | Major | build | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-15869](https://issues.apache.org/jira/browse/HADOOP-15869) | BlockDecompressorStream#decompress should not return -1 in case of IOException. | Major | . | Surendra Singh Lilhore | Surendra Singh Lilhore | +| [MAPREDUCE-7158](https://issues.apache.org/jira/browse/MAPREDUCE-7158) | Inefficient Flush Logic in JobHistory EventWriter | Major | . | Zichen Sun | Zichen Sun | +| [HADOOP-15930](https://issues.apache.org/jira/browse/HADOOP-15930) | Exclude MD5 checksum files from release artifact | Critical | build | Akira Ajisaka | Akira Ajisaka | +| [YARN-8856](https://issues.apache.org/jira/browse/YARN-8856) | TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError | Major | . | Jason Lowe | Sushil Ks | +| [HADOOP-15925](https://issues.apache.org/jira/browse/HADOOP-15925) | The config and log of gpg-agent are removed in create-release script | Major | build | Akira Ajisaka | Dinesh Chitlangia | +| [HDFS-13963](https://issues.apache.org/jira/browse/HDFS-13963) | NN UI is broken with IE11 | Minor | namenode, ui | Daisuke Kobayashi | Ayush Saxena | +| [HDFS-14056](https://issues.apache.org/jira/browse/HDFS-14056) | Fix error messages in HDFS-12716 | Minor | hdfs | Adam Antal | Ayush Saxena | +| [YARN-8992](https://issues.apache.org/jira/browse/YARN-8992) | Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue | Major | fairscheduler | Haibo Chen | Wilfred Spiegelenburg | +| [YARN-8984](https://issues.apache.org/jira/browse/YARN-8984) | AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty | Critical | . | Yang Wang | Yang Wang | +| [HADOOP-15948](https://issues.apache.org/jira/browse/HADOOP-15948) | Inconsistency in get and put syntax if filename/dirname contains space | Minor | fs | vivek kumar | Ayush Saxena | +| [HDFS-13816](https://issues.apache.org/jira/browse/HDFS-13816) | dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException | Major | namenode | Vinayakumar B | Vinayakumar B | +| [MAPREDUCE-7162](https://issues.apache.org/jira/browse/MAPREDUCE-7162) | TestEvents#testEvents fails | Critical | jobhistoryserver, test | Zhaohui Xin | Zhaohui Xin | +| [YARN-9056](https://issues.apache.org/jira/browse/YARN-9056) | Yarn Service Upgrade: Instance state changes from UPGRADING to READY without performing a readiness check | Critical | . | Chandni Singh | Chandni Singh | +| [YARN-8812](https://issues.apache.org/jira/browse/YARN-8812) | Containers fail during creating a symlink which started with hyphen for a resource file | Minor | . | Oleksandr Shevchenko | Oleksandr Shevchenko | +| [YARN-9030](https://issues.apache.org/jira/browse/YARN-9030) | Log aggregation changes to handle filesystems which do not support setting permissions | Major | log-aggregation | Suma Shivaprasad | Suma Shivaprasad | +| [YARN-9067](https://issues.apache.org/jira/browse/YARN-9067) | YARN Resource Manager is running OOM because of leak of Configuration Object | Major | yarn-native-services | Eric Yang | Eric Yang | +| [MAPREDUCE-7165](https://issues.apache.org/jira/browse/MAPREDUCE-7165) | mapred-site.xml is misformatted in single node setup document | Major | documentation | Akira Ajisaka | Zhaohui Xin | +| [HADOOP-15970](https://issues.apache.org/jira/browse/HADOOP-15970) | Upgrade plexus-utils from 2.0.5 to 3.1.0 | Major | security | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-15966](https://issues.apache.org/jira/browse/HADOOP-15966) | Hadoop Kerberos broken on macos as java.security.krb5.realm is reset: Null realm name (601) | Major | scripts | Steve Loughran | Steve Loughran | +| [HADOOP-15974](https://issues.apache.org/jira/browse/HADOOP-15974) | Upgrade Curator version to 2.13.0 to fix ZK tests | Major | . | Jason Lowe | Akira Ajisaka | +| [YARN-9071](https://issues.apache.org/jira/browse/YARN-9071) | NM and service AM don't have updated status for reinitialized containers | Critical | . | Billie Rinaldi | Chandni Singh | +| [MAPREDUCE-7159](https://issues.apache.org/jira/browse/MAPREDUCE-7159) | FrameworkUploader: ensure proper permissions of generated framework tar.gz if restrictive umask is used | Major | mrv2 | Peter Bacsko | Peter Bacsko | +| [YARN-9009](https://issues.apache.org/jira/browse/YARN-9009) | Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs | Minor | . | OrDTesters | OrDTesters | +| [MAPREDUCE-7170](https://issues.apache.org/jira/browse/MAPREDUCE-7170) | Doc typo in PluggableShuffleAndPluggableSort.md | Minor | documentation | Zhaohui Xin | Zhaohui Xin | +| [YARN-9040](https://issues.apache.org/jira/browse/YARN-9040) | LevelDBCacheTimelineStore in ATS 1.5 leaks native memory | Major | timelineserver | Tarun Parimi | Tarun Parimi | +| [YARN-9084](https://issues.apache.org/jira/browse/YARN-9084) | Service Upgrade: With default readiness check, the status of upgrade is reported to be successful prematurely | Major | . | Chandni Singh | Chandni Singh | +| [HDFS-13661](https://issues.apache.org/jira/browse/HDFS-13661) | Ls command with e option fails when the filesystem is not HDFS | Major | erasure-coding, tools | Takanobu Asanuma | Takanobu Asanuma | +| [YARN-9154](https://issues.apache.org/jira/browse/YARN-9154) | Fix itemization in YARN service quickstart document | Minor | documentation | Akira Ajisaka | Ayush Saxena | +| [HDFS-14166](https://issues.apache.org/jira/browse/HDFS-14166) | Ls with -e option not giving the result in proper format | Major | . | Soumyapn | Shubham Dewan | +| [HDFS-14046](https://issues.apache.org/jira/browse/HDFS-14046) | In-Maintenance ICON is missing in datanode info page | Major | datanode | Harshakiran Reddy | Ranith Sardar | +| [HDFS-14183](https://issues.apache.org/jira/browse/HDFS-14183) | [SPS] Remove the -w parameter from the -satisfystoragepolicy usage | Major | . | Ayush Saxena | Ayush Saxena | +| [YARN-9164](https://issues.apache.org/jira/browse/YARN-9164) | Shutdown NM may cause NPE when opportunistic container scheduling is enabled | Critical | . | lujie | lujie | +| [YARN-8567](https://issues.apache.org/jira/browse/YARN-8567) | Fetching yarn logs fails for long running application if it is not present in timeline store | Major | log-aggregation | Tarun Parimi | Tarun Parimi | +| [HADOOP-16028](https://issues.apache.org/jira/browse/HADOOP-16028) | Fix NetworkTopology chooseRandom function to support excluded nodes | Major | . | Sihai Ke | Sihai Ke | +| [HADOOP-16030](https://issues.apache.org/jira/browse/HADOOP-16030) | AliyunOSS: bring fixes back from HADOOP-15671 | Blocker | fs/oss | wujinhu | wujinhu | +| [YARN-9162](https://issues.apache.org/jira/browse/YARN-9162) | Fix TestRMAdminCLI#testHelp | Major | resourcemanager, test | Ayush Saxena | Ayush Saxena | +| [YARN-9173](https://issues.apache.org/jira/browse/YARN-9173) | FairShare calculation broken for large values after YARN-8833 | Major | fairscheduler | Wilfred Spiegelenburg | Wilfred Spiegelenburg | +| [YARN-8833](https://issues.apache.org/jira/browse/YARN-8833) | Avoid potential integer overflow when computing fair shares | Major | fairscheduler | liyakun | liyakun | +| [HADOOP-16036](https://issues.apache.org/jira/browse/HADOOP-16036) | WASB: Disable jetty logging configuration announcement | Major | fs/azure | Da Zhou | Da Zhou | +| [HADOOP-16016](https://issues.apache.org/jira/browse/HADOOP-16016) | TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds | Major | security, test | Jason Lowe | Akira Ajisaka | +| [HDFS-14198](https://issues.apache.org/jira/browse/HDFS-14198) | Upload and Create button doesn't get enabled after getting reset. | Major | . | Ayush Saxena | Ayush Saxena | +| [YARN-8747](https://issues.apache.org/jira/browse/YARN-8747) | [UI2] YARN UI2 page loading failed due to js error under some time zone configuration | Critical | webapp | collinma | collinma | +| [YARN-9203](https://issues.apache.org/jira/browse/YARN-9203) | Fix typos in yarn-default.xml | Trivial | documentation | Rahul Padmanabhan | Rahul Padmanabhan | +| [YARN-9194](https://issues.apache.org/jira/browse/YARN-9194) | Invalid event: REGISTERED and LAUNCH\_FAILED at FAILED, and NullPointerException happens in RM while shutdown a NM | Critical | . | lujie | lujie | +| [YARN-9204](https://issues.apache.org/jira/browse/YARN-9204) | RM fails to start if absolute resource is specified for partition capacity in CS queues | Blocker | yarn | Jiandan Yang | Jiandan Yang | +| [HDFS-14207](https://issues.apache.org/jira/browse/HDFS-14207) | ZKFC should catch exception when ha configuration missing | Major | hdfs | Fei Hui | Fei Hui | +| [HDFS-14218](https://issues.apache.org/jira/browse/HDFS-14218) | EC: Ls -e throw NPE when directory ec policy is disabled | Major | . | Surendra Singh Lilhore | Ayush Saxena | +| [YARN-9210](https://issues.apache.org/jira/browse/YARN-9210) | RM nodes web page can not display node info | Blocker | yarn | Jiandan Yang | Jiandan Yang | +| [YARN-9205](https://issues.apache.org/jira/browse/YARN-9205) | When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER\_THEN\_MAX\_ALLOCATION) | Critical | . | Zhankun Tang | Zhankun Tang | +| [YARN-8961](https://issues.apache.org/jira/browse/YARN-8961) | [UI2] Flow Run End Time shows 'Invalid date' | Major | . | Charan Hebri | Akhil PB | +| [HADOOP-16065](https://issues.apache.org/jira/browse/HADOOP-16065) | -Ddynamodb should be -Ddynamo in AWS SDK testing document | Minor | documentation | Akira Ajisaka | Akira Ajisaka | +| [HDFS-14228](https://issues.apache.org/jira/browse/HDFS-14228) | Incorrect getSnapshottableDirListing() javadoc | Major | snapshots | Wei-Chiu Chuang | Dinesh Chitlangia | +| [YARN-9222](https://issues.apache.org/jira/browse/YARN-9222) | Print launchTime in ApplicationSummary | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-8901](https://issues.apache.org/jira/browse/YARN-8901) | Restart "NEVER" policy does not work with component dependency | Critical | . | Yesha Vora | Suma Shivaprasad | +| [YARN-9237](https://issues.apache.org/jira/browse/YARN-9237) | NM should ignore sending finished apps to RM during RM fail-over | Major | yarn | Jiandan Yang | Jiandan Yang | +| [YARN-6616](https://issues.apache.org/jira/browse/YARN-6616) | YARN AHS shows submitTime for jobs same as startTime | Minor | . | Prabhu Joseph | Prabhu Joseph | +| [YARN-9099](https://issues.apache.org/jira/browse/YARN-9099) | GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way | Major | . | Szilard Nemeth | Szilard Nemeth | +| [YARN-9262](https://issues.apache.org/jira/browse/YARN-9262) | TestRMAppAttemptTransitions is failing with an NPE | Critical | resourcemanager | Sunil Govindan | lujie | +| [HDFS-14232](https://issues.apache.org/jira/browse/HDFS-14232) | libhdfs is not included in binary tarball | Critical | build, libhdfs | Akira Ajisaka | Akira Ajisaka | +| [MAPREDUCE-7177](https://issues.apache.org/jira/browse/MAPREDUCE-7177) | Disable speculative execution in TestDFSIO | Major | . | Kihwal Lee | Zhaohui Xin | +| [HADOOP-16076](https://issues.apache.org/jira/browse/HADOOP-16076) | SPNEGO+SSL Client Connections with HttpClient Broken | Major | build, security | Larry McCay | Larry McCay | +| [HADOOP-16074](https://issues.apache.org/jira/browse/HADOOP-16074) | WASB: Update container not found error code | Major | fs/azure | Da Zhou | Da Zhou | +| [YARN-8498](https://issues.apache.org/jira/browse/YARN-8498) | Yarn NodeManager OOM Listener Fails Compilation on Ubuntu 18.04 | Blocker | . | Jack Bearden | Ayush Saxena | +| [YARN-9206](https://issues.apache.org/jira/browse/YARN-9206) | RMServerUtils does not count SHUTDOWN as an accepted state | Major | . | Kuhu Shukla | Kuhu Shukla | +| [HADOOP-16032](https://issues.apache.org/jira/browse/HADOOP-16032) | Distcp It should clear sub directory ACL before applying new ACL on it. | Major | tools/distcp | Ranith Sardar | Ranith Sardar | +| [HDFS-14140](https://issues.apache.org/jira/browse/HDFS-14140) | JournalNodeSyncer authentication is failing in secure cluster | Major | journal-node, security | Surendra Singh Lilhore | Surendra Singh Lilhore | +| [YARN-9257](https://issues.apache.org/jira/browse/YARN-9257) | Distributed Shell client throws a NPE for a non-existent queue | Major | distributed-shell | Charan Hebri | Charan Hebri | +| [YARN-8761](https://issues.apache.org/jira/browse/YARN-8761) | Service AM support for decommissioning component instances | Major | . | Billie Rinaldi | Billie Rinaldi | +| [HDFS-14266](https://issues.apache.org/jira/browse/HDFS-14266) | EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. | Major | . | Harshakiran Reddy | Ayush Saxena | +| [HDFS-14274](https://issues.apache.org/jira/browse/HDFS-14274) | EC: NPE While Listing EC Policy For A Directory Following Replication Policy. | Major | erasure-coding | Souryakanta Dwivedy | Ayush Saxena | +| [HDFS-14263](https://issues.apache.org/jira/browse/HDFS-14263) | Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream() | Major | datanode | Surendra Singh Lilhore | Surendra Singh Lilhore | +| [YARN-7761](https://issues.apache.org/jira/browse/YARN-7761) | [UI2] Clicking 'master container log' or 'Link' next to 'log' under application's appAttempt goes to Old UI's Log link | Major | yarn-ui-v2 | Sumana Sathish | Akhil PB | +| [YARN-9295](https://issues.apache.org/jira/browse/YARN-9295) | [UI2] Fix label typo in Cluster Overview page | Trivial | yarn-ui-v2 | Charan Hebri | Charan Hebri | +| [YARN-9308](https://issues.apache.org/jira/browse/YARN-9308) | fairscheduler-statedump.log gets generated regardless of service again after the merge of HDFS-7240 | Blocker | fairscheduler, scheduler | Akira Ajisaka | Wilfred Spiegelenburg | +| [YARN-9284](https://issues.apache.org/jira/browse/YARN-9284) | Fix the unit of yarn.service.am-resource.memory in the document | Minor | documentation, yarn-native-services | Masahiro Tanaka | Masahiro Tanaka | +| [YARN-9283](https://issues.apache.org/jira/browse/YARN-9283) | Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference | Minor | documentation | Szilard Nemeth | Adam Antal | +| [YARN-9286](https://issues.apache.org/jira/browse/YARN-9286) | [Timeline Server] Sorting based on FinalStatus shows pop-up message | Minor | timelineserver | Nallasivan | Bilwa S T | +| [HDFS-14081](https://issues.apache.org/jira/browse/HDFS-14081) | hdfs dfsadmin -metasave metasave\_test results NPE | Major | hdfs | Shweta | Shweta | +| [HADOOP-15813](https://issues.apache.org/jira/browse/HADOOP-15813) | Enable more reliable SSL connection reuse | Major | common | Daryn Sharp | Daryn Sharp | +| [HDFS-14216](https://issues.apache.org/jira/browse/HDFS-14216) | NullPointerException happens in NamenodeWebHdfs | Critical | . | lujie | lujie | +| [HADOOP-16105](https://issues.apache.org/jira/browse/HADOOP-16105) | WASB in secure mode does not set connectingUsingSAS | Major | fs/azure | Steve Loughran | Steve Loughran | +| [YARN-9238](https://issues.apache.org/jira/browse/YARN-9238) | Avoid allocating opportunistic containers to previous/removed/non-exist application attempt | Critical | . | lujie | lujie | +| [YARN-9118](https://issues.apache.org/jira/browse/YARN-9118) | Handle exceptions with parsing user defined GPU devices in GpuDiscoverer | Major | . | Szilard Nemeth | Szilard Nemeth | +| [YARN-9317](https://issues.apache.org/jira/browse/YARN-9317) | Avoid repeated YarnConfiguration#timelineServiceV2Enabled check | Major | . | Bibin A Chundatt | Prabhu Joseph | +| [YARN-9213](https://issues.apache.org/jira/browse/YARN-9213) | RM Web UI v1 does not show custom resource allocations for containers page | Major | . | Szilard Nemeth | Szilard Nemeth | +| [YARN-9329](https://issues.apache.org/jira/browse/YARN-9329) | updatePriority is blocked when using FairScheduler | Major | . | Jiandan Yang | Jiandan Yang | +| [YARN-9248](https://issues.apache.org/jira/browse/YARN-9248) | RMContainerImpl:Invalid event: ACQUIRED at KILLED | Major | . | lujie | lujie | +| [HADOOP-16018](https://issues.apache.org/jira/browse/HADOOP-16018) | DistCp won't reassemble chunks when blocks per chunk \> 0 | Major | tools/distcp | Kai Xie | Kai Xie | +| [YARN-9334](https://issues.apache.org/jira/browse/YARN-9334) | YARN Service Client does not work with SPNEGO when knox is configured | Major | yarn-native-services | Tarun Parimi | Billie Rinaldi | +| [HDFS-14305](https://issues.apache.org/jira/browse/HDFS-14305) | Serial number in BlockTokenSecretManager could overlap between different namenodes | Major | namenode, security | Chao Sun | He Xiaoqiao | +| [HDFS-14314](https://issues.apache.org/jira/browse/HDFS-14314) | fullBlockReportLeaseId should be reset after registering to NN | Critical | datanode | star | star | +| [YARN-8803](https://issues.apache.org/jira/browse/YARN-8803) | [UI2] Show flow runs in the order of recently created time in graph widgets | Major | yarn-ui-v2 | Akhil PB | Akhil PB | +| [HDFS-14317](https://issues.apache.org/jira/browse/HDFS-14317) | Standby does not trigger edit log rolling when in-progress edit log tailing is enabled | Critical | . | Ekanth Sethuramalingam | Ekanth Sethuramalingam | +| [HDFS-14333](https://issues.apache.org/jira/browse/HDFS-14333) | Datanode fails to start if any disk has errors during Namenode registration | Major | datanode | Stephen O'Donnell | Stephen O'Donnell | +| [HADOOP-16192](https://issues.apache.org/jira/browse/HADOOP-16192) | CallQueue backoff bug fixes: doesn't perform backoff when add() is used, and doesn't update backoff when refreshed | Major | ipc | Erik Krogen | Erik Krogen | +| [HDFS-14037](https://issues.apache.org/jira/browse/HDFS-14037) | Fix SSLFactory truststore reloader thread leak in URLConnectionFactory | Major | hdfs-client, webhdfs | Takanobu Asanuma | Takanobu Asanuma | +| [HADOOP-16225](https://issues.apache.org/jira/browse/HADOOP-16225) | Fix links to the developer mailing lists in DownstreamDev.md | Minor | documentation | Akira Ajisaka | Wanqiang Ji | +| [HADOOP-16226](https://issues.apache.org/jira/browse/HADOOP-16226) | new Path(String str) does not remove all the trailing slashes of str | Minor | fs | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-16232](https://issues.apache.org/jira/browse/HADOOP-16232) | Fix errors in the checkstyle configration xmls | Major | build | Akira Ajisaka | Wanqiang Ji | +| [HDFS-14389](https://issues.apache.org/jira/browse/HDFS-14389) | getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured | Major | namenode | Stephen O'Donnell | Stephen O'Donnell | +| [HDFS-14407](https://issues.apache.org/jira/browse/HDFS-14407) | Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes | Minor | . | Wanqiang Ji | Wanqiang Ji | +| [YARN-9413](https://issues.apache.org/jira/browse/YARN-9413) | Queue resource leak after app fail for CapacityScheduler | Major | capacityscheduler | Tao Yang | Tao Yang | +| [HADOOP-14635](https://issues.apache.org/jira/browse/HADOOP-14635) | Javadoc correction for AccessControlList#buildACL | Minor | documentation | Bibin A Chundatt | Yeliang Cang | +| [HADOOP-14544](https://issues.apache.org/jira/browse/HADOOP-14544) | DistCp documentation for command line options is misaligned. | Minor | documentation | Chris Nauroth | Masatake Iwasaki | +| [HDFS-10477](https://issues.apache.org/jira/browse/HDFS-10477) | Stop decommission a rack of DataNodes caused NameNode fail over to standby | Major | namenode | yunjiong zhao | yunjiong zhao | +| [YARN-9487](https://issues.apache.org/jira/browse/YARN-9487) | NodeManager native build shouldn't link against librt on macOS | Major | nodemanager | Siyao Meng | Siyao Meng | +| [YARN-6695](https://issues.apache.org/jira/browse/YARN-6695) | Race condition in RM for publishing container events vs appFinished events causes NPE | Critical | . | Rohith Sharma K S | Prabhu Joseph | +| [YARN-8622](https://issues.apache.org/jira/browse/YARN-8622) | NodeManager native build fails due to getgrouplist not found on macOS | Major | nodemanager | Ewan Higgs | Siyao Meng | +| [HADOOP-16265](https://issues.apache.org/jira/browse/HADOOP-16265) | Configuration#getTimeDuration is not consistent between default value and manual settings. | Major | . | star | star | +| [HDFS-13677](https://issues.apache.org/jira/browse/HDFS-13677) | Dynamic refresh Disk configuration results in overwriting VolumeMap | Blocker | . | xuzq | xuzq | +| [YARN-9285](https://issues.apache.org/jira/browse/YARN-9285) | RM UI progress column is of wrong type | Minor | yarn | Ahmed Hussein | Ahmed Hussein | +| [HADOOP-16278](https://issues.apache.org/jira/browse/HADOOP-16278) | With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die | Major | common, hadoop-aws, metrics | Rajat Khandelwal | Rajat Khandelwal | +| [YARN-9504](https://issues.apache.org/jira/browse/YARN-9504) | [UI2] Fair scheduler queue view page does not show actual capacity | Major | fairscheduler, yarn-ui-v2 | Zoltan Siegl | Zoltan Siegl | +| [YARN-9519](https://issues.apache.org/jira/browse/YARN-9519) | TFile log aggregation file format is not working for yarn.log-aggregation.TFile.remote-app-log-dir config | Major | log-aggregation | Adam Antal | Adam Antal | +| [YARN-9508](https://issues.apache.org/jira/browse/YARN-9508) | YarnConfiguration areNodeLabel enabled is costly in allocation flow | Critical | . | Bibin A Chundatt | Bilwa S T | +| [HADOOP-16247](https://issues.apache.org/jira/browse/HADOOP-16247) | NPE in FsUrlConnection | Major | hdfs-client | Karthik Palanisamy | Karthik Palanisamy | +| [HADOOP-16248](https://issues.apache.org/jira/browse/HADOOP-16248) | MutableQuantiles leak memory under heavy load | Major | metrics | Alexis Daboville | Alexis Daboville | +| [HDFS-14323](https://issues.apache.org/jira/browse/HDFS-14323) | Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path | Major | webhdfs | Srinivasu Majeti | Srinivasu Majeti | +| [MAPREDUCE-7205](https://issues.apache.org/jira/browse/MAPREDUCE-7205) | Treat container scheduler kill exit code as a task attempt killing event | Major | applicationmaster, mr-am, mrv2 | Wanqiang Ji | Wanqiang Ji | +| [HDFS-14500](https://issues.apache.org/jira/browse/HDFS-14500) | NameNode StartupProgress continues to report edit log segments after the LOADING\_EDITS phase is finished | Major | namenode | Erik Krogen | Erik Krogen | +| [YARN-9500](https://issues.apache.org/jira/browse/YARN-9500) | Fix typos in ResourceModel.md | Trivial | documentation | leiqiang | leiqiang | +| [HADOOP-16331](https://issues.apache.org/jira/browse/HADOOP-16331) | Fix ASF License check in pom.xml | Major | . | Wanqiang Ji | Akira Ajisaka | +| [YARN-9542](https://issues.apache.org/jira/browse/YARN-9542) | Fix LogsCLI guessAppOwner ignores custom file format suffix | Minor | log-aggregation | Prabhu Joseph | Prabhu Joseph | +| [HDFS-14512](https://issues.apache.org/jira/browse/HDFS-14512) | ONE\_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes) | Major | . | Shen Yinjie | Ayush Saxena | +| [HADOOP-16334](https://issues.apache.org/jira/browse/HADOOP-16334) | Fix yetus-wrapper not working when HADOOP\_YETUS\_VERSION \>= 0.9.0 | Major | yetus | Wanqiang Ji | Wanqiang Ji | +| [HDFS-14521](https://issues.apache.org/jira/browse/HDFS-14521) | Suppress setReplication logging. | Major | . | Kihwal Lee | Kihwal Lee | +| [YARN-9507](https://issues.apache.org/jira/browse/YARN-9507) | Fix NPE in NodeManager#serviceStop on startup failure | Minor | . | Bilwa S T | Bilwa S T | +| [YARN-8947](https://issues.apache.org/jira/browse/YARN-8947) | [UI2] Active User info missing from UI2 | Major | yarn-ui-v2 | Akhil PB | Akhil PB | +| [YARN-8906](https://issues.apache.org/jira/browse/YARN-8906) | [UI2] NM hostnames not displayed correctly in Node Heatmap Chart | Major | . | Charan Hebri | Akhil PB | +| [YARN-9580](https://issues.apache.org/jira/browse/YARN-9580) | Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers | Major | capacityscheduler | Tao Yang | Tao Yang | +| [YARN-8625](https://issues.apache.org/jira/browse/YARN-8625) | Aggregate Resource Allocation for each job is not present in ATS | Major | ATSv2 | Prabhu Joseph | Prabhu Joseph | +| [HADOOP-16345](https://issues.apache.org/jira/browse/HADOOP-16345) | Potential NPE when instantiating FairCallQueue metrics | Major | ipc | Erik Krogen | Erik Krogen | +| [YARN-9594](https://issues.apache.org/jira/browse/YARN-9594) | Fix missing break statement in ContainerScheduler#handle | Major | . | lujie | lujie | +| [YARN-9565](https://issues.apache.org/jira/browse/YARN-9565) | RMAppImpl#ranNodes not cleared on FinalTransition | Major | . | Bibin A Chundatt | Bilwa S T | +| [YARN-9547](https://issues.apache.org/jira/browse/YARN-9547) | ContainerStatusPBImpl default execution type is not returned | Major | . | Bibin A Chundatt | Bilwa S T | +| [HDFS-13231](https://issues.apache.org/jira/browse/HDFS-13231) | Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI | Major | datanode, namenode | Haibo Yan | Stephen O'Donnell | +| [HDFS-14535](https://issues.apache.org/jira/browse/HDFS-14535) | The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read | Major | hdfs-client | Zheng Hu | Zheng Hu | +| [HDFS-13730](https://issues.apache.org/jira/browse/HDFS-13730) | BlockReaderRemote.sendReadResult throws NPE | Major | hdfs-client | Wei-Chiu Chuang | Yuanbo Liu | +| [YARN-9584](https://issues.apache.org/jira/browse/YARN-9584) | Should put initializeProcessTrees method call before get pid | Critical | nodemanager | Wanqiang Ji | Wanqiang Ji | +| [HDFS-14010](https://issues.apache.org/jira/browse/HDFS-14010) | Pass correct DF usage to ReservedSpaceCalculator builder | Minor | . | Lukas Majercak | Lukas Majercak | +| [HDFS-14078](https://issues.apache.org/jira/browse/HDFS-14078) | Admin helper fails to prettify NullPointerExceptions | Major | . | Elek, Marton | Elek, Marton | +| [HDFS-14101](https://issues.apache.org/jira/browse/HDFS-14101) | Random failure of testListCorruptFilesCorruptedBlock | Major | test | Kihwal Lee | Zsolt Venczel | +| [HDFS-14465](https://issues.apache.org/jira/browse/HDFS-14465) | When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. | Major | . | Yicong Cai | Yicong Cai | +| [HDFS-13893](https://issues.apache.org/jira/browse/HDFS-13893) | DiskBalancer: no validations for Disk balancer commands | Major | diskbalancer | Harshakiran Reddy | Lokesh Jain | +| [YARN-9209](https://issues.apache.org/jira/browse/YARN-9209) | When nodePartition is not set in Placement Constraints, containers are allocated only in default partition | Major | capacity scheduler, scheduler | Tarun Parimi | Tarun Parimi | +| [HDFS-12487](https://issues.apache.org/jira/browse/HDFS-12487) | FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers | Major | balancer & mover, diskbalancer | liumi | liumi | +| [HDFS-14074](https://issues.apache.org/jira/browse/HDFS-14074) | DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. | Major | hdfs | guangyi lu | guangyi lu | +| [HDFS-14541](https://issues.apache.org/jira/browse/HDFS-14541) | When evictableMmapped or evictable size is zero, do not throw NoSuchElementException | Major | hdfs-client, performance | Zheng Hu | Lisheng Sun | +| [HDFS-14598](https://issues.apache.org/jira/browse/HDFS-14598) | Findbugs warning caused by HDFS-12487 | Minor | diskbalancer | Wei-Chiu Chuang | He Xiaoqiao | +| [YARN-9639](https://issues.apache.org/jira/browse/YARN-9639) | DecommissioningNodesWatcher cause memory leak | Blocker | . | Bibin A Chundatt | Bilwa S T | +| [YARN-9581](https://issues.apache.org/jira/browse/YARN-9581) | Fix WebAppUtils#getRMWebAppURLWithScheme ignores rm2 | Major | client | Prabhu Joseph | Prabhu Joseph | +| [YARN-9327](https://issues.apache.org/jira/browse/YARN-9327) | Improve synchronisation in ProtoUtils#convertToProtoFormat block | Critical | . | Bibin A Chundatt | Bibin A Chundatt | +| [YARN-9655](https://issues.apache.org/jira/browse/YARN-9655) | AllocateResponse in FederationInterceptor lost applicationPriority | Major | federation | hunshenshi | hunshenshi | +| [HADOOP-16385](https://issues.apache.org/jira/browse/HADOOP-16385) | Namenode crashes with "RedundancyMonitor thread received Runtime exception" | Major | . | krishna reddy | Ayush Saxena | +| [YARN-9644](https://issues.apache.org/jira/browse/YARN-9644) | First RMContext object is always leaked during switch over | Blocker | . | Bibin A Chundatt | Bibin A Chundatt | +| [HDFS-14629](https://issues.apache.org/jira/browse/HDFS-14629) | Property value Hard Coded in DNConf.java | Trivial | . | hemanthboyina | hemanthboyina | +| [YARN-9557](https://issues.apache.org/jira/browse/YARN-9557) | Application fails in diskchecker when ReadWriteDiskValidator is configured. | Critical | nodemanager | Anuruddh Nayak | Bilwa S T | +| [HDFS-12703](https://issues.apache.org/jira/browse/HDFS-12703) | Exceptions are fatal to decommissioning monitor | Critical | namenode | Daryn Sharp | He Xiaoqiao | +| [HDFS-12748](https://issues.apache.org/jira/browse/HDFS-12748) | NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY | Major | hdfs | Jiandan Yang | Weiwei Yang | +| [YARN-9625](https://issues.apache.org/jira/browse/YARN-9625) | UI2 - No link to a queue on the Queues page for Fair Scheduler | Major | . | Charan Hebri | Zoltan Siegl | +| [HDFS-14466](https://issues.apache.org/jira/browse/HDFS-14466) | Add a regression test for HDFS-14323 | Minor | fs, test, webhdfs | Yuya Ebihara | Masatake Iwasaki | +| [YARN-9235](https://issues.apache.org/jira/browse/YARN-9235) | If linux container executor is not set for a GPU cluster GpuResourceHandlerImpl is not initialized and NPE is thrown | Major | yarn | Antal Bálint Steinbach | Adam Antal | +| [YARN-9626](https://issues.apache.org/jira/browse/YARN-9626) | UI2 - Fair scheduler queue apps page issues | Major | . | Charan Hebri | Zoltan Siegl | +| [YARN-9645](https://issues.apache.org/jira/browse/YARN-9645) | Fix Invalid event FINISHED\_CONTAINERS\_PULLED\_BY\_AM at NEW on NM restart | Major | . | krishna reddy | Bilwa S T | +| [YARN-9682](https://issues.apache.org/jira/browse/YARN-9682) | Wrong log message when finalizing the upgrade | Trivial | . | kyungwan nam | kyungwan nam | +| [HADOOP-16440](https://issues.apache.org/jira/browse/HADOOP-16440) | Distcp can not preserve timestamp with -delete option | Major | . | ludun | ludun | +| [MAPREDUCE-7076](https://issues.apache.org/jira/browse/MAPREDUCE-7076) | TestNNBench#testNNBenchCreateReadAndDelete failing in our internal build | Minor | test | Rushabh S Shah | kevin su | +| [HADOOP-16443](https://issues.apache.org/jira/browse/HADOOP-16443) | Improve help text for setfacl --set option | Minor | fs | Stephen O'Donnell | Stephen O'Donnell | +| [YARN-9668](https://issues.apache.org/jira/browse/YARN-9668) | UGI conf doesn't read user overridden configurations on RM and NM startup | Major | . | Jonathan Hung | Jonathan Hung | +| [HADOOP-9844](https://issues.apache.org/jira/browse/HADOOP-9844) | NPE when trying to create an error message response of SASL RPC | Major | ipc | Steve Loughran | Steve Loughran | +| [HADOOP-16245](https://issues.apache.org/jira/browse/HADOOP-16245) | Enabling SSL within LdapGroupsMapping can break system SSL configs | Major | common, security | Erik Krogen | Erik Krogen | +| [HDFS-14429](https://issues.apache.org/jira/browse/HDFS-14429) | Block remain in COMMITTED but not COMPLETE caused by Decommission | Major | . | Yicong Cai | Yicong Cai | +| [HADOOP-16435](https://issues.apache.org/jira/browse/HADOOP-16435) | RpcMetrics should not be retained forever | Critical | rpc-server | Zoltan Haindrich | Zoltan Haindrich | +| [YARN-9596](https://issues.apache.org/jira/browse/YARN-9596) | QueueMetrics has incorrect metrics when labelled partitions are involved | Major | capacity scheduler | Muhammad Samir Khan | Muhammad Samir Khan | +| [MAPREDUCE-7225](https://issues.apache.org/jira/browse/MAPREDUCE-7225) | Fix broken current folder expansion during MR job start | Major | mrv2 | Adam Antal | Peter Bacsko | +| [HDFS-13529](https://issues.apache.org/jira/browse/HDFS-13529) | Fix default trash policy emptier trigger time correctly | Major | namenode | He Xiaoqiao | He Xiaoqiao | +| [HADOOP-15681](https://issues.apache.org/jira/browse/HADOOP-15681) | AuthenticationFilter should generate valid date format for Set-Cookie header regardless of default Locale | Minor | security | Cao Manh Dat | Cao Manh Dat | +| [HDFS-14685](https://issues.apache.org/jira/browse/HDFS-14685) | DefaultAuditLogger doesn't print CallerContext | Major | hdfs | xuzq | xuzq | +| [HDFS-14462](https://issues.apache.org/jira/browse/HDFS-14462) | WebHDFS throws "Error writing request body to server" instead of DSQuotaExceededException | Major | webhdfs | Erik Krogen | Simbarashe Dzinamarira | +| [HDFS-14691](https://issues.apache.org/jira/browse/HDFS-14691) | Wrong usage hint for hadoop fs command test | Minor | hdfs | Jianfei Jiang | Jianfei Jiang | +| [HDFS-14557](https://issues.apache.org/jira/browse/HDFS-14557) | JournalNode error: Can't scan a pre-transactional edit log | Major | ha | Wei-Chiu Chuang | Stephen O'Donnell | +| [HDFS-14692](https://issues.apache.org/jira/browse/HDFS-14692) | Upload button should not encode complete url | Major | . | Lokesh Jain | Lokesh Jain | +| [HADOOP-15908](https://issues.apache.org/jira/browse/HADOOP-15908) | hadoop-build-tools jar is downloaded from remote repository instead of using from local | Minor | build | Oleksandr Shevchenko | Oleksandr Shevchenko | +| [HDFS-14631](https://issues.apache.org/jira/browse/HDFS-14631) | The DirectoryScanner doesn't fix the wrongly placed replica. | Major | . | Jinglun | Jinglun | +| [YARN-9685](https://issues.apache.org/jira/browse/YARN-9685) | NPE when rendering the info table of leaf queue in non-accessible partitions | Major | capacityscheduler | Tao Yang | Tao Yang | +| [HDFS-14459](https://issues.apache.org/jira/browse/HDFS-14459) | ClosedChannelException silently ignored in FsVolumeList.addBlockPool() | Major | datanode | Stephen O'Donnell | Stephen O'Donnell | +| [HDFS-13359](https://issues.apache.org/jira/browse/HDFS-13359) | DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream | Major | datanode | Yiqun Lin | Yiqun Lin | +| [YARN-9451](https://issues.apache.org/jira/browse/YARN-9451) | AggregatedLogsBlock shows wrong NM http port | Minor | nodemanager | Prabhu Joseph | Prabhu Joseph | +| [YARN-9723](https://issues.apache.org/jira/browse/YARN-9723) | ApplicationPlacementContext is not required for terminated jobs during recovery | Major | resourcemanager | Prabhu Joseph | Prabhu Joseph | +| [HDFS-12914](https://issues.apache.org/jira/browse/HDFS-12914) | Block report leases cause missing blocks until next report | Critical | namenode | Daryn Sharp | Santosh Marella | +| [HDFS-14148](https://issues.apache.org/jira/browse/HDFS-14148) | HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory | Major | hdfs | Siyao Meng | Siyao Meng | +| [HDFS-14595](https://issues.apache.org/jira/browse/HDFS-14595) | HDFS-11848 breaks API compatibility | Blocker | . | Wei-Chiu Chuang | Siyao Meng | +| [HDFS-14423](https://issues.apache.org/jira/browse/HDFS-14423) | Percent (%) and plus (+) characters no longer work in WebHDFS | Major | webhdfs | Jing Wang | Masatake Iwasaki | +| [MAPREDUCE-7230](https://issues.apache.org/jira/browse/MAPREDUCE-7230) | TestHSWebApp.testLogsViewSingle fails | Major | jobhistoryserver, test | Prabhu Joseph | Prabhu Joseph | +| [YARN-9749](https://issues.apache.org/jira/browse/YARN-9749) | TestAppLogAggregatorImpl#testDFSQuotaExceeded fails on trunk | Major | log-aggregation, test | Peter Bacsko | Adam Antal | +| [HDFS-14687](https://issues.apache.org/jira/browse/HDFS-14687) | Standby Namenode never come out of safemode when EC files are being written. | Critical | ec, namenode | Surendra Singh Lilhore | Surendra Singh Lilhore | +| [HDFS-13101](https://issues.apache.org/jira/browse/HDFS-13101) | Yet another fsimage corruption related to snapshot | Major | snapshots | Yongjun Zhang | Shashikant Banerjee | +| [HDFS-13201](https://issues.apache.org/jira/browse/HDFS-13201) | Fix prompt message in testPolicyAndStateCantBeNull | Minor | . | chencan | chencan | +| [HDFS-14311](https://issues.apache.org/jira/browse/HDFS-14311) | Multi-threading conflict at layoutVersion when loading block pool storage | Major | rolling upgrades | Yicong Cai | Yicong Cai | +| [HDFS-14582](https://issues.apache.org/jira/browse/HDFS-14582) | Failed to start DN with ArithmeticException when NULL checksum used | Major | datanode | Surendra Singh Lilhore | Surendra Singh Lilhore | +| [YARN-9217](https://issues.apache.org/jira/browse/YARN-9217) | Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing | Major | yarn | Antal Bálint Steinbach | Peter Bacsko | +| [HADOOP-16494](https://issues.apache.org/jira/browse/HADOOP-16494) | Add SHA-256 or SHA-512 checksum to release artifacts to comply with the release distribution policy | Blocker | build | Akira Ajisaka | Akira Ajisaka | +| [YARN-9774](https://issues.apache.org/jira/browse/YARN-9774) | Fix order of arguments for assertEquals in TestSLSUtils | Minor | test | Nikhil Navadiya | Nikhil Navadiya | +| [HDFS-13596](https://issues.apache.org/jira/browse/HDFS-13596) | NN restart fails after RollingUpgrade from 2.x to 3.x | Blocker | hdfs | Hanisha Koneru | Fei Hui | +| [HDFS-14396](https://issues.apache.org/jira/browse/HDFS-14396) | Failed to load image from FSImageFile when downgrade from 3.x to 2.x | Blocker | rolling upgrades | Fei Hui | Fei Hui | +| [YARN-8917](https://issues.apache.org/jira/browse/YARN-8917) | Absolute (maximum) capacity of level3+ queues is wrongly calculated for absolute resource | Critical | capacityscheduler | Tao Yang | Tao Yang | +| [YARN-9642](https://issues.apache.org/jira/browse/YARN-9642) | Fix Memory Leak in AbstractYarnScheduler caused by timer | Blocker | resourcemanager | Bibin A Chundatt | Bibin A Chundatt | +| [HDFS-13977](https://issues.apache.org/jira/browse/HDFS-13977) | NameNode can kill itself if it tries to send too many txns to a QJM simultaneously | Major | namenode, qjm | Erik Krogen | Erik Krogen | +| [HDFS-2470](https://issues.apache.org/jira/browse/HDFS-2470) | NN should automatically set permissions on dfs.namenode.\*.dir | Major | namenode | Aaron T. Myers | Siddharth Wagle | +| [YARN-9438](https://issues.apache.org/jira/browse/YARN-9438) | launchTime not written to state store for running applications | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9640](https://issues.apache.org/jira/browse/YARN-9640) | Slow event processing could cause too many attempt unregister events | Critical | . | Bibin A Chundatt | Bibin A Chundatt | +| [HDFS-12212](https://issues.apache.org/jira/browse/HDFS-12212) | Options.Rename.To\_TRASH is considered even when Options.Rename.NONE is specified | Major | namenode | Vinayakumar B | Vinayakumar B | +| [YARN-9714](https://issues.apache.org/jira/browse/YARN-9714) | ZooKeeper connection in ZKRMStateStore leaks after RM transitioned to standby | Major | resourcemanager | Tao Yang | Tao Yang | +| [HDFS-8178](https://issues.apache.org/jira/browse/HDFS-8178) | QJM doesn't move aside stale inprogress edits files | Major | qjm | Zhe Zhang | Istvan Fajth | +| [HDFS-14706](https://issues.apache.org/jira/browse/HDFS-14706) | Checksums are not checked if block meta file is less than 7 bytes | Major | . | Stephen O'Donnell | Stephen O'Donnell | +| [YARN-9797](https://issues.apache.org/jira/browse/YARN-9797) | LeafQueue#activateApplications should use resourceCalculator#fitsIn | Blocker | . | Bibin A Chundatt | Bilwa S T | +| [YARN-9785](https://issues.apache.org/jira/browse/YARN-9785) | Fix DominantResourceCalculator when one resource is zero | Blocker | . | Bilwa S T | Bilwa S T | +| [YARN-9718](https://issues.apache.org/jira/browse/YARN-9718) | Yarn REST API, services endpoint remote command ejection | Major | . | Eric Yang | Eric Yang | +| [HADOOP-16255](https://issues.apache.org/jira/browse/HADOOP-16255) | ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum | Major | fs | Steve Loughran | Jungtaek Lim | +| [YARN-9817](https://issues.apache.org/jira/browse/YARN-9817) | Fix failing testcases due to not initialized AsyncDispatcher - ArithmeticException: / by zero | Major | test | Prabhu Joseph | Prabhu Joseph | +| [YARN-9813](https://issues.apache.org/jira/browse/YARN-9813) | RM does not start on JDK11 when UIv2 is enabled | Critical | resourcemanager, yarn | Adam Antal | Adam Antal | +| [YARN-9820](https://issues.apache.org/jira/browse/YARN-9820) | RM logs InvalidStateTransitionException when app is submitted | Critical | . | Rohith Sharma K S | Prabhu Joseph | + + +### TESTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [YARN-8907](https://issues.apache.org/jira/browse/YARN-8907) | Modify a logging message in TestCapacityScheduler | Trivial | . | Zhankun Tang | Zhankun Tang | +| [YARN-8904](https://issues.apache.org/jira/browse/YARN-8904) | TestRMDelegationTokens can fail in testRMDTMasterKeyStateOnRollingMasterKey | Minor | test | Wilfred Spiegelenburg | Wilfred Spiegelenburg | +| [YARN-8944](https://issues.apache.org/jira/browse/YARN-8944) | TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896 | Minor | capacity scheduler | Wilfred Spiegelenburg | Wilfred Spiegelenburg | +| [YARN-9263](https://issues.apache.org/jira/browse/YARN-9263) | TestConfigurationNodeAttributesProvider fails after Mockito updated | Minor | . | Weiwei Yang | Weiwei Yang | +| [YARN-9315](https://issues.apache.org/jira/browse/YARN-9315) | TestCapacitySchedulerMetrics fails intermittently | Minor | capacity scheduler | Prabhu Joseph | Prabhu Joseph | +| [YARN-9316](https://issues.apache.org/jira/browse/YARN-9316) | TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently | Minor | capacity scheduler | Prabhu Joseph | Prabhu Joseph | +| [YARN-9324](https://issues.apache.org/jira/browse/YARN-9324) | TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11 | Major | capacity scheduler | Prabhu Joseph | Prabhu Joseph | +| [YARN-9325](https://issues.apache.org/jira/browse/YARN-9325) | TestQueueManagementDynamicEditPolicy fails intermittent | Minor | capacity scheduler | Prabhu Joseph | Prabhu Joseph | +| [HDFS-11950](https://issues.apache.org/jira/browse/HDFS-11950) | Disable libhdfs zerocopy test on Mac | Minor | libhdfs | John Zhuge | Akira Ajisaka | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [YARN-6989](https://issues.apache.org/jira/browse/YARN-6989) | Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way | Major | timelineserver | Vrushali C | Abhishek Modi | +| [YARN-8834](https://issues.apache.org/jira/browse/YARN-8834) | Provide Java client for fetching Yarn specific entities from TimelineReader | Critical | timelinereader | Rohith Sharma K S | Abhishek Modi | +| [YARN-3879](https://issues.apache.org/jira/browse/YARN-3879) | [Storage implementation] Create HDFS backing storage implementation for ATS reads | Major | timelineserver | Tsuyoshi Ozawa | Abhishek Modi | +| [YARN-6098](https://issues.apache.org/jira/browse/YARN-6098) | Add documentation for Delete Queue | Major | capacity scheduler, documentation | Naganarasimha G R | Suma Shivaprasad | +| [YARN-8456](https://issues.apache.org/jira/browse/YARN-8456) | Fix a configuration handling bug when user leave FPGA discover executable path configuration default but set OpenCL SDK path environment variable | Major | yarn | Zhankun Tang | Zhankun Tang | +| [HADOOP-15868](https://issues.apache.org/jira/browse/HADOOP-15868) | AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy | Major | fs/oss | wujinhu | wujinhu | +| [YARN-8871](https://issues.apache.org/jira/browse/YARN-8871) | Document behavior of YARN-5742 | Major | . | Vrushali C | Suma Shivaprasad | +| [YARN-7754](https://issues.apache.org/jira/browse/YARN-7754) | [Atsv2] Update document for running v1 and v2 TS | Major | . | Rohith Sharma K S | Suma Shivaprasad | +| [HDFS-14047](https://issues.apache.org/jira/browse/HDFS-14047) | [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test\_libhdfs\_threaded.c | Major | libhdfs, native | Anatoli Shein | Anatoli Shein | +| [YARN-8988](https://issues.apache.org/jira/browse/YARN-8988) | Reduce the verbose log on RM heartbeat path when distributed node-attributes is enabled | Major | . | Weiwei Yang | Tao Yang | +| [HADOOP-15846](https://issues.apache.org/jira/browse/HADOOP-15846) | ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. | Major | fs/azure | Thomas Marquardt | junhua gu | +| [HADOOP-15812](https://issues.apache.org/jira/browse/HADOOP-15812) | ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console | Major | fs/azure | Da Zhou | Da Zhou | +| [YARN-8987](https://issues.apache.org/jira/browse/YARN-8987) | Usability improvements node-attributes CLI | Critical | . | Weiwei Yang | Bibin A Chundatt | +| [HADOOP-15876](https://issues.apache.org/jira/browse/HADOOP-15876) | Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore | Minor | fs/azure | Ted Yu | Da Zhou | +| [HADOOP-15917](https://issues.apache.org/jira/browse/HADOOP-15917) | AliyunOSS: fix incorrect ReadOps and WriteOps in statistics | Major | fs/oss | wujinhu | wujinhu | +| [YARN-8303](https://issues.apache.org/jira/browse/YARN-8303) | YarnClient should contact TimelineReader for application/attempt/container report | Critical | . | Rohith Sharma K S | Abhishek Modi | +| [HADOOP-15872](https://issues.apache.org/jira/browse/HADOOP-15872) | ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 | Major | fs/azure | Thomas Marquardt | junhua gu | +| [HADOOP-15940](https://issues.apache.org/jira/browse/HADOOP-15940) | ABFS: For HNS account, avoid unnecessary get call when doing Rename | Major | fs/azure | Da Zhou | Da Zhou | +| [YARN-8986](https://issues.apache.org/jira/browse/YARN-8986) | publish all exposed ports to random ports when using bridge network | Minor | yarn | dockerzhang | dockerzhang | +| [HADOOP-15932](https://issues.apache.org/jira/browse/HADOOP-15932) | Oozie unable to create sharelib in s3a filesystem | Critical | fs, fs/s3 | Soumitra Sulav | Steve Loughran | +| [YARN-9034](https://issues.apache.org/jira/browse/YARN-9034) | ApplicationCLI should have option to take clusterId | Major | . | Rohith Sharma K S | Rohith Sharma K S | +| [HDFS-13713](https://issues.apache.org/jira/browse/HDFS-13713) | Add specification of Multipart Upload API to FS specification, with contract tests | Blocker | fs, test | Steve Loughran | Ewan Higgs | +| [HADOOP-15968](https://issues.apache.org/jira/browse/HADOOP-15968) | ABFS: add try catch for UGI failure when initializing ABFS | Major | fs/azure | Da Zhou | Da Zhou | +| [HADOOP-15969](https://issues.apache.org/jira/browse/HADOOP-15969) | ABFS: getNamespaceEnabled can fail blocking user access thru ACLs | Major | fs/azure | Da Zhou | Da Zhou | +| [HADOOP-15972](https://issues.apache.org/jira/browse/HADOOP-15972) | ABFS: reduce list page size to to 500 | Major | fs/azure | Da Zhou | Da Zhou | +| [HADOOP-16004](https://issues.apache.org/jira/browse/HADOOP-16004) | ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException | Major | fs/azure | Da Zhou | Da Zhou | +| [YARN-9126](https://issues.apache.org/jira/browse/YARN-9126) | Container reinit always fails in branch-3.2 and trunk | Major | . | Eric Yang | Chandni Singh | +| [YARN-8925](https://issues.apache.org/jira/browse/YARN-8925) | Updating distributed node attributes only when necessary | Major | resourcemanager | Tao Yang | Tao Yang | +| [HADOOP-16009](https://issues.apache.org/jira/browse/HADOOP-16009) | Replace the url of the repository in Apache Hadoop source code | Major | documentation | Akira Ajisaka | Akira Ajisaka | +| [HADOOP-15323](https://issues.apache.org/jira/browse/HADOOP-15323) | AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore | Major | fs/oss | wujinhu | wujinhu | +| [YARN-6149](https://issues.apache.org/jira/browse/YARN-6149) | Allow port range to be specified while starting NM Timeline collector manager. | Major | timelineserver | Varun Saxena | Abhishek Modi | +| [HADOOP-16040](https://issues.apache.org/jira/browse/HADOOP-16040) | ABFS: Bug fix for tolerateOobAppends configuration | Major | fs/azure | Da Zhou | Da Zhou | +| [HADOOP-15975](https://issues.apache.org/jira/browse/HADOOP-15975) | ABFS: remove timeout check for DELETE and RENAME | Major | fs/azure | Da Zhou | Da Zhou | +| [HADOOP-15662](https://issues.apache.org/jira/browse/HADOOP-15662) | ABFS: Better exception handling of DNS errors | Major | fs/azure | Thomas Marquardt | Da Zhou | +| [HADOOP-16045](https://issues.apache.org/jira/browse/HADOOP-16045) | Don't run TestDU on Windows | Trivial | common, test | Lukas Majercak | Lukas Majercak | +| [HADOOP-16044](https://issues.apache.org/jira/browse/HADOOP-16044) | ABFS: Better exception handling of DNS errors followup | Major | . | Da Zhou | Da Zhou | +| [HADOOP-16048](https://issues.apache.org/jira/browse/HADOOP-16048) | ABFS: Fix Date format parser | Major | fs/azure | Da Zhou | Da Zhou | +| [YARN-8101](https://issues.apache.org/jira/browse/YARN-8101) | Add UT to verify node-attributes in RM nodes rest API | Minor | resourcemanager, restapi | Weiwei Yang | Prabhu Joseph | +| [HADOOP-16041](https://issues.apache.org/jira/browse/HADOOP-16041) | UserAgent string for ABFS | Major | fs/azure | Shweta | Shweta | +| [HADOOP-16079](https://issues.apache.org/jira/browse/HADOOP-16079) | Token.toString faulting if any token listed can't load. | Blocker | security | Steve Loughran | Steve Loughran | +| [YARN-9275](https://issues.apache.org/jira/browse/YARN-9275) | Add link to NodeAttributes doc in PlacementConstraints document | Minor | documentation | Weiwei Yang | Masatake Iwasaki | +| [YARN-6735](https://issues.apache.org/jira/browse/YARN-6735) | Have a way to turn off container metrics from NMs | Major | timelineserver | Vrushali C | Abhishek Modi | +| [HADOOP-15954](https://issues.apache.org/jira/browse/HADOOP-15954) | ABFS: Enable owner and group conversion for MSI and login user using OAuth | Major | fs/azure | junhua gu | Da Zhou | +| [YARN-9253](https://issues.apache.org/jira/browse/YARN-9253) | Add UT to verify Placement Constraint in Distributed Shell | Major | . | Prabhu Joseph | Prabhu Joseph | +| [YARN-9252](https://issues.apache.org/jira/browse/YARN-9252) | Allocation Tag Namespace support in Distributed Shell | Major | distributed-shell | Prabhu Joseph | Prabhu Joseph | +| [YARN-8555](https://issues.apache.org/jira/browse/YARN-8555) | Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC handler options | Minor | . | Weiwei Yang | Prabhu Joseph | +| [YARN-9293](https://issues.apache.org/jira/browse/YARN-9293) | Optimize MockAMLauncher event handling | Major | . | Bibin A Chundatt | Bibin A Chundatt | +| [HADOOP-16104](https://issues.apache.org/jira/browse/HADOOP-16104) | Wasb tests to downgrade to skip when test a/c is namespace enabled | Major | fs/azure, test | Steve Loughran | Masatake Iwasaki | +| [YARN-9258](https://issues.apache.org/jira/browse/YARN-9258) | Support to specify allocation tags without constraint in distributed shell CLI | Major | distributed-shell | Prabhu Joseph | Prabhu Joseph | +| [HADOOP-16136](https://issues.apache.org/jira/browse/HADOOP-16136) | ABFS: Should only transform username to short name | Major | . | Da Zhou | Da Zhou | +| [YARN-5336](https://issues.apache.org/jira/browse/YARN-5336) | Limit the flow name size & consider cleanup for hex chars | Major | timelineserver | Vrushali C | Sushil Ks | +| [YARN-3841](https://issues.apache.org/jira/browse/YARN-3841) | [Storage implementation] Adding retry semantics to HDFS backing storage | Major | timelineserver | Tsuyoshi Ozawa | Abhishek Modi | +| [HADOOP-16169](https://issues.apache.org/jira/browse/HADOOP-16169) | ABFS: Bug fix for getPathProperties | Major | fs/azure | Da Zhou | Da Zhou | +| [HADOOP-16109](https://issues.apache.org/jira/browse/HADOOP-16109) | Parquet reading S3AFileSystem causes EOF | Blocker | fs/s3 | Dave Christianson | Steve Loughran | +| [HADOOP-15625](https://issues.apache.org/jira/browse/HADOOP-15625) | S3A input stream to use etags/version number to detect changed source files | Major | fs/s3 | Brahma Reddy Battula | Ben Roling | +| [HADOOP-16124](https://issues.apache.org/jira/browse/HADOOP-16124) | Extend documentation in testing.md about endpoint constants | Trivial | hadoop-aws | Adam Antal | Adam Antal | +| [HADOOP-16191](https://issues.apache.org/jira/browse/HADOOP-16191) | AliyunOSS: improvements for copyFile/copyDirectory and logging | Major | fs/oss | wujinhu | wujinhu | +| [YARN-9387](https://issues.apache.org/jira/browse/YARN-9387) | Update document for ATS HBase Custom tablenames (-entityTableName) | Critical | ATSv2 | Prabhu Joseph | Prabhu Joseph | +| [YARN-9389](https://issues.apache.org/jira/browse/YARN-9389) | FlowActivity and FlowRun table prefix is wrong | Minor | ATSv2 | Prabhu Joseph | Prabhu Joseph | +| [YARN-9391](https://issues.apache.org/jira/browse/YARN-9391) | Disable PATH variable to be passed to Docker container | Major | . | Eric Yang | Jim Brennan | +| [HADOOP-16058](https://issues.apache.org/jira/browse/HADOOP-16058) | S3A tests to include Terasort | Major | fs/s3, test | Steve Loughran | Steve Loughran | +| [HADOOP-16220](https://issues.apache.org/jira/browse/HADOOP-16220) | Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project | Major | . | Gabor Bota | Gabor Bota | +| [YARN-9418](https://issues.apache.org/jira/browse/YARN-9418) | ATSV2 /apps/appId/entities/YARN\_CONTAINER rest api does not show metrics | Critical | ATSv2 | Prabhu Joseph | Prabhu Joseph | +| [HADOOP-16233](https://issues.apache.org/jira/browse/HADOOP-16233) | S3AFileStatus to declare that isEncrypted() is always true | Minor | fs/s3 | Steve Loughran | Steve Loughran | +| [YARN-9303](https://issues.apache.org/jira/browse/YARN-9303) | Username splits won't help timelineservice.app\_flow table | Major | ATSv2 | Prabhu Joseph | Prabhu Joseph | +| [YARN-9382](https://issues.apache.org/jira/browse/YARN-9382) | Publish container killed, paused and resumed events to ATSv2. | Major | . | Abhishek Modi | Abhishek Modi | +| [YARN-9335](https://issues.apache.org/jira/browse/YARN-9335) | [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls | Major | . | Vrushali C | Abhishek Modi | +| [HADOOP-16269](https://issues.apache.org/jira/browse/HADOOP-16269) | ABFS: add listFileStatus with StartFrom | Major | fs/azure | Da Zhou | Da Zhou | +| [HADOOP-16251](https://issues.apache.org/jira/browse/HADOOP-16251) | ABFS: add FSMainOperationsBaseTest | Major | fs/azure | Da Zhou | Da Zhou | +| [HADOOP-16306](https://issues.apache.org/jira/browse/HADOOP-16306) | AliyunOSS: Remove temporary files when upload small files to OSS | Major | fs/oss | wujinhu | wujinhu | +| [YARN-7537](https://issues.apache.org/jira/browse/YARN-7537) | [Atsv2] load hbase configuration from filesystem rather than URL | Major | . | Rohith Sharma K S | Prabhu Joseph | +| [HDFS-14553](https://issues.apache.org/jira/browse/HDFS-14553) | Make queue size of BlockReportProcessingThread configurable | Major | namenode | He Xiaoqiao | He Xiaoqiao | +| [HADOOP-16211](https://issues.apache.org/jira/browse/HADOOP-16211) | Update guava to 27.0-jre in hadoop-project branch-3.2 | Major | . | Gabor Bota | Gabor Bota | +| [YARN-8499](https://issues.apache.org/jira/browse/YARN-8499) | ATS v2 Generic TimelineStorageMonitor | Major | ATSv2 | Sunil Govindan | Prabhu Joseph | +| [YARN-9374](https://issues.apache.org/jira/browse/YARN-9374) | HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage down | Major | ATSv2 | Prabhu Joseph | Prabhu Joseph | +| [HADOOP-16401](https://issues.apache.org/jira/browse/HADOOP-16401) | ABFS: port Azure doc to 3.2 branch | Major | fs/azure | Da Zhou | Masatake Iwasaki | +| [HDFS-14034](https://issues.apache.org/jira/browse/HDFS-14034) | Support getQuotaUsage API in WebHDFS | Major | fs, webhdfs | Erik Krogen | Chao Sun | +| [YARN-9765](https://issues.apache.org/jira/browse/YARN-9765) | SLS runner crashes when run with metrics turned off. | Major | . | Abhishek Modi | Abhishek Modi | +| [HDFS-14674](https://issues.apache.org/jira/browse/HDFS-14674) | [SBN read] Got an unexpected txid when tail editlog | Blocker | . | wangzhaohui | wangzhaohui | +| [YARN-9775](https://issues.apache.org/jira/browse/YARN-9775) | RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore | Major | restapi | Prabhu Joseph | Prabhu Joseph | +| [HDFS-14779](https://issues.apache.org/jira/browse/HDFS-14779) | Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9804](https://issues.apache.org/jira/browse/YARN-9804) | Update ATSv2 document for latest feature supports | Blocker | . | Rohith Sharma K S | Rohith Sharma K S | +| [YARN-9821](https://issues.apache.org/jira/browse/YARN-9821) | NM hangs at serviceStop when ATSV2 Backend Hbase is Down | Major | ATSv2 | Prabhu Joseph | Prabhu Joseph | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-15851](https://issues.apache.org/jira/browse/HADOOP-15851) | Disable wildfly logs to the console | Major | fs/azure | Vishwajeet Dusane | Vishwajeet Dusane | +| [HDFS-12729](https://issues.apache.org/jira/browse/HDFS-12729) | Document special paths in HDFS | Major | documentation | Chris Douglas | Masatake Iwasaki | +| [YARN-9191](https://issues.apache.org/jira/browse/YARN-9191) | Add cli option in DS to support enforceExecutionType in resource requests. | Major | . | Abhishek Modi | Abhishek Modi | +| [HADOOP-16037](https://issues.apache.org/jira/browse/HADOOP-16037) | DistCp: Document usage of Sync (-diff option) in detail | Major | documentation, tools/distcp | Siyao Meng | Siyao Meng | +| [HADOOP-16263](https://issues.apache.org/jira/browse/HADOOP-16263) | Update BUILDING.txt with macOS native build instructions | Minor | . | Siyao Meng | Siyao Meng | +| [YARN-9559](https://issues.apache.org/jira/browse/YARN-9559) | Create AbstractContainersLauncher for pluggable ContainersLauncher logic | Major | . | Jonathan Hung | Jonathan Hung | +| [YARN-9796](https://issues.apache.org/jira/browse/YARN-9796) | Fix ASF license issue in branch-3.2 | Blocker | . | Rohith Sharma K S | Prabhu Joseph | + + diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/RELEASENOTES.3.2.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/RELEASENOTES.3.2.1.md new file mode 100644 index 0000000000000..0a8862abbbcaa --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/RELEASENOTES.3.2.1.md @@ -0,0 +1,80 @@ + + +# Apache Hadoop 3.2.1 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [YARN-8986](https://issues.apache.org/jira/browse/YARN-8986) | *Minor* | **publish all exposed ports to random ports when using bridge network** + +support -p and -P for bridge type network; + + +--- + +* [YARN-9071](https://issues.apache.org/jira/browse/YARN-9071) | *Critical* | **NM and service AM don't have updated status for reinitialized containers** + +In progress upgrade status may show READY state sooner than actual upgrade operations. External caller to upgrade API is recommended to wait minimum 30 seconds before querying yarn app -status. + + +--- + +* [YARN-9084](https://issues.apache.org/jira/browse/YARN-9084) | *Major* | **Service Upgrade: With default readiness check, the status of upgrade is reported to be successful prematurely** + +Improve transient container status accuracy for upgrade. + + +--- + +* [HADOOP-15922](https://issues.apache.org/jira/browse/HADOOP-15922) | *Major* | **DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL** + +- Fix DelegationTokenAuthentication filter for incorrectly double encode doAs user parameter. + + +--- + +* [YARN-8761](https://issues.apache.org/jira/browse/YARN-8761) | *Major* | **Service AM support for decommissioning component instances** + +- Component instance number is not linear increment when decommission feature is used. Application with assumption of linear increment component instance number maybe impacted by introduction of this feature. + + +--- + +* [HDFS-14305](https://issues.apache.org/jira/browse/HDFS-14305) | *Major* | **Serial number in BlockTokenSecretManager could overlap between different namenodes** + +NameNodes rely on independent block token key ranges to communicate block token identities to DataNodes and clients in a way that does not create conflicts between the tokens issued by multiple NameNodes. HDFS-6440 introduced the potential for overlaps in key ranges; this fixes the issue by creating 64 possible key ranges that NameNodes assign themselves to, allowing for up to 64 NameNodes to run safely. This limitation only applies within a single Namespace; there may be more than 64 NameNodes total spread among multiple federated Namespaces. + + +--- + +* [HDFS-14396](https://issues.apache.org/jira/browse/HDFS-14396) | *Blocker* | **Failed to load image from FSImageFile when downgrade from 3.x to 2.x** + +During a rolling upgrade from Hadoop 2.x to 3.x, NameNode cannot persist erasure coding information, and therefore a user cannot start using erasure coding feature until finalize is done. + + +--- + +* [YARN-7055](https://issues.apache.org/jira/browse/YARN-7055) | *Major* | **YARN Timeline Service v.2: beta 1 / GA** + +Application Timeline Server v2 is ready for production. It is GA from 3.2.1 release on wards. + + + diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java index 64beb7b484296..a6adb9f20a3ef 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java @@ -2437,7 +2437,7 @@ public void testGettingPropertiesWithPrefix() throws Exception { } conf.set("different.prefix" + ".name", "value"); Map prefixedProps = conf.getPropsWithPrefix("prefix."); - assertEquals(prefixedProps.size(), 10); + assertThat(prefixedProps.size(), is(10)); for (int i = 0; i < 10; i++) { assertEquals("value" + i, prefixedProps.get("name" + i)); } @@ -2448,7 +2448,7 @@ public void testGettingPropertiesWithPrefix() throws Exception { conf.set("subprefix." + "subname" + i, "value_${foo}" + i); } prefixedProps = conf.getPropsWithPrefix("subprefix."); - assertEquals(prefixedProps.size(), 10); + assertThat(prefixedProps.size(), is(10)); for (int i = 0; i < 10; i++) { assertEquals("value_bar" + i, prefixedProps.get("subname" + i)); } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/CryptoStreamsTestBase.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/CryptoStreamsTestBase.java index 7463d6c3bc064..53d0939a2d398 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/CryptoStreamsTestBase.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/CryptoStreamsTestBase.java @@ -46,6 +46,8 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.assertj.core.api.Assertions.assertThat; + public abstract class CryptoStreamsTestBase { protected static final Logger LOG = LoggerFactory.getLogger( CryptoStreamsTestBase.class); @@ -198,7 +200,7 @@ private void readCheck(InputStream in) throws Exception { // EOF n = in.read(result, 0, dataLen); - Assert.assertEquals(n, -1); + assertThat(n).isEqualTo(-1); } /** Test crypto writing with different buffer size. */ @@ -388,42 +390,41 @@ private void positionedReadCheckWithByteBuffer(InputStream in, int pos) Assert.assertArrayEquals(readData, expectedData); } - /** Test read fully */ + /** Test read fully. */ @Test(timeout=120000) public void testReadFully() throws Exception { OutputStream out = getOutputStream(defaultBufferSize); writeData(out); - InputStream in = getInputStream(defaultBufferSize); - final int len1 = dataLen / 4; - // Read len1 bytes - byte[] readData = new byte[len1]; - readAll(in, readData, 0, len1); - byte[] expectedData = new byte[len1]; - System.arraycopy(data, 0, expectedData, 0, len1); - Assert.assertArrayEquals(readData, expectedData); - - // Pos: 1/3 dataLen - readFullyCheck(in, dataLen / 3); - - // Read len1 bytes - readData = new byte[len1]; - readAll(in, readData, 0, len1); - expectedData = new byte[len1]; - System.arraycopy(data, len1, expectedData, 0, len1); - Assert.assertArrayEquals(readData, expectedData); - - // Pos: 1/2 dataLen - readFullyCheck(in, dataLen / 2); - - // Read len1 bytes - readData = new byte[len1]; - readAll(in, readData, 0, len1); - expectedData = new byte[len1]; - System.arraycopy(data, 2 * len1, expectedData, 0, len1); - Assert.assertArrayEquals(readData, expectedData); - - in.close(); + try (InputStream in = getInputStream(defaultBufferSize)) { + final int len1 = dataLen / 4; + // Read len1 bytes + byte[] readData = new byte[len1]; + readAll(in, readData, 0, len1); + byte[] expectedData = new byte[len1]; + System.arraycopy(data, 0, expectedData, 0, len1); + Assert.assertArrayEquals(readData, expectedData); + + // Pos: 1/3 dataLen + readFullyCheck(in, dataLen / 3); + + // Read len1 bytes + readData = new byte[len1]; + readAll(in, readData, 0, len1); + expectedData = new byte[len1]; + System.arraycopy(data, len1, expectedData, 0, len1); + Assert.assertArrayEquals(readData, expectedData); + + // Pos: 1/2 dataLen + readFullyCheck(in, dataLen / 2); + + // Read len1 bytes + readData = new byte[len1]; + readAll(in, readData, 0, len1); + expectedData = new byte[len1]; + System.arraycopy(data, 2 * len1, expectedData, 0, len1); + Assert.assertArrayEquals(readData, expectedData); + } } private void readFullyCheck(InputStream in, int pos) throws Exception { @@ -441,6 +442,60 @@ private void readFullyCheck(InputStream in, int pos) throws Exception { } catch (EOFException e) { } } + + /** Test byte byffer read fully. */ + @Test(timeout=120000) + public void testByteBufferReadFully() throws Exception { + OutputStream out = getOutputStream(defaultBufferSize); + writeData(out); + + try (InputStream in = getInputStream(defaultBufferSize)) { + final int len1 = dataLen / 4; + // Read len1 bytes + byte[] readData = new byte[len1]; + readAll(in, readData, 0, len1); + byte[] expectedData = new byte[len1]; + System.arraycopy(data, 0, expectedData, 0, len1); + Assert.assertArrayEquals(readData, expectedData); + + // Pos: 1/3 dataLen + byteBufferReadFullyCheck(in, dataLen / 3); + + // Read len1 bytes + readData = new byte[len1]; + readAll(in, readData, 0, len1); + expectedData = new byte[len1]; + System.arraycopy(data, len1, expectedData, 0, len1); + Assert.assertArrayEquals(readData, expectedData); + + // Pos: 1/2 dataLen + byteBufferReadFullyCheck(in, dataLen / 2); + + // Read len1 bytes + readData = new byte[len1]; + readAll(in, readData, 0, len1); + expectedData = new byte[len1]; + System.arraycopy(data, 2 * len1, expectedData, 0, len1); + Assert.assertArrayEquals(readData, expectedData); + } + } + + private void byteBufferReadFullyCheck(InputStream in, int pos) + throws Exception { + ByteBuffer result = ByteBuffer.allocate(dataLen - pos); + ((ByteBufferPositionedReadable) in).readFully(pos, result); + + byte[] expectedData = new byte[dataLen - pos]; + System.arraycopy(data, pos, expectedData, 0, dataLen - pos); + Assert.assertArrayEquals(result.array(), expectedData); + + result = ByteBuffer.allocate(dataLen); // Exceeds maximum length + try { + ((ByteBufferPositionedReadable) in).readFully(pos, result); + Assert.fail("Read fully exceeds maximum length should fail."); + } catch (EOFException e) { + } + } /** Test seek to different position. */ @Test(timeout=120000) @@ -559,7 +614,7 @@ public void testSkip() throws Exception { // Skip after EOF skipped = in.skip(3); - Assert.assertEquals(skipped, 0); + assertThat(skipped).isZero(); in.close(); } @@ -791,7 +846,7 @@ public void testCombinedOp() throws Exception { ((Seekable) in).seek(dataLen); buf.clear(); n = ((ByteBufferReadable) in).read(buf); - Assert.assertEquals(n, -1); + assertThat(n).isEqualTo(-1); in.close(); } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreams.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreams.java index 8bcf46eacf041..73c6249612387 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreams.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreams.java @@ -330,6 +330,30 @@ public int read(long position, ByteBuffer buf) throws IOException { return -1; } + @Override + public void readFully(long position, ByteBuffer buf) throws IOException { + if (buf == null) { + throw new NullPointerException(); + } else if (!buf.hasRemaining()) { + return; + } + + if (position > length) { + throw new IOException("Cannot read after EOF."); + } + if (position < 0) { + throw new IOException("Cannot read to negative offset."); + } + + checkStream(); + + if (position + buf.remaining() > length) { + throw new EOFException("Reach the end of stream."); + } + + buf.put(data, (int) position, buf.remaining()); + } + @Override public void readFully(long position, byte[] b, int off, int len) throws IOException { diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsForLocalFS.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsForLocalFS.java index e7d922e78a64e..8453889b53a5a 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsForLocalFS.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsForLocalFS.java @@ -95,6 +95,11 @@ public void testByteBufferRead() throws Exception {} @Override @Test(timeout=10000) public void testPositionedReadWithByteBuffer() throws IOException {} + + @Ignore("Wrapped stream doesn't support ByteBufferPositionedReadable") + @Override + @Test(timeout=10000) + public void testByteBufferReadFully() throws Exception {} @Ignore("ChecksumFSOutputSummer doesn't support Syncable") @Override diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsNormal.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsNormal.java index 036706f435a60..1bf1dd3e0d6a3 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsNormal.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsNormal.java @@ -96,6 +96,11 @@ public void testPositionedRead() throws IOException {} @Test(timeout=10000) public void testPositionedReadWithByteBuffer() throws IOException {} + @Ignore("Wrapped stream doesn't support ByteBufferPositionedReadable") + @Override + @Test(timeout=10000) + public void testByteBufferReadFully() throws Exception {} + @Ignore("Wrapped stream doesn't support ReadFully") @Override @Test(timeout=10000) diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithJceAesCtrCryptoCodec.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithJceAesCtrCryptoCodec.java index 76c39d694b04c..d47dd307574f8 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithJceAesCtrCryptoCodec.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithJceAesCtrCryptoCodec.java @@ -19,20 +19,21 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.CommonConfigurationKeysPublic; -import org.junit.Assert; import org.junit.BeforeClass; +import static org.assertj.core.api.Assertions.assertThat; + public class TestCryptoStreamsWithJceAesCtrCryptoCodec extends TestCryptoStreams { @BeforeClass - public static void init() throws Exception { + public static void init() { Configuration conf = new Configuration(); conf.set( CommonConfigurationKeysPublic.HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_AES_CTR_NOPADDING_KEY, JceAesCtrCryptoCodec.class.getName()); codec = CryptoCodec.getInstance(conf); - Assert.assertEquals(JceAesCtrCryptoCodec.class.getCanonicalName(), - codec.getClass().getCanonicalName()); + assertThat(JceAesCtrCryptoCodec.class.getCanonicalName()) + .isEqualTo(codec.getClass().getCanonicalName()); } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/key/TestValueQueue.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/key/TestValueQueue.java index abc4ebf9b4dbc..55a9280d6260a 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/key/TestValueQueue.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/key/TestValueQueue.java @@ -20,6 +20,7 @@ import java.io.IOException; import java.util.Queue; import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.TimeoutException; import java.util.concurrent.TimeUnit; import org.slf4j.Logger; @@ -31,7 +32,6 @@ import org.junit.Assert; import org.junit.Test; -import com.google.common.base.Supplier; import com.google.common.collect.Sets; public class TestValueQueue { @@ -62,6 +62,18 @@ public FillInfo getTop() throws InterruptedException { } } + private void waitForRefill(ValueQueue valueQueue, String queueName, int queueSize) + throws TimeoutException, InterruptedException { + GenericTestUtils.waitFor(() -> { + int size = valueQueue.getSize(queueName); + if (size != queueSize) { + LOG.info("Current ValueQueue size is " + size); + return false; + } + return true; + }, 100, 3000); + } + /** * Verifies that Queue is initially filled to "numInitValues" */ @@ -69,7 +81,7 @@ public FillInfo getTop() throws InterruptedException { public void testInitFill() throws Exception { MockFiller filler = new MockFiller(); ValueQueue vq = - new ValueQueue(10, 0.1f, 300, 1, + new ValueQueue(10, 0.1f, 30000, 1, SyncGenerationPolicy.ALL, filler); Assert.assertEquals("test", vq.getNext("k1")); Assert.assertEquals(1, filler.getTop().num); @@ -83,7 +95,7 @@ public void testInitFill() throws Exception { public void testWarmUp() throws Exception { MockFiller filler = new MockFiller(); ValueQueue vq = - new ValueQueue(10, 0.5f, 300, 1, + new ValueQueue(10, 0.5f, 30000, 1, SyncGenerationPolicy.ALL, filler); vq.initializeQueuesForKeys("k1", "k2", "k3"); FillInfo[] fillInfos = @@ -106,14 +118,17 @@ public void testWarmUp() throws Exception { public void testRefill() throws Exception { MockFiller filler = new MockFiller(); ValueQueue vq = - new ValueQueue(10, 0.1f, 300, 1, + new ValueQueue(100, 0.1f, 30000, 1, SyncGenerationPolicy.ALL, filler); + // Trigger a prefill (10) and an async refill (91) Assert.assertEquals("test", vq.getNext("k1")); - Assert.assertEquals(1, filler.getTop().num); - // Trigger refill - vq.getNext("k1"); - Assert.assertEquals(1, filler.getTop().num); Assert.assertEquals(10, filler.getTop().num); + + // Wait for the async task to finish + waitForRefill(vq, "k1", 100); + // Refill task should add 91 values to get to a full queue (10 produced by + // the prefill to the low watermark, 1 consumed by getNext()) + Assert.assertEquals(91, filler.getTop().num); vq.shutdown(); } @@ -125,10 +140,27 @@ public void testRefill() throws Exception { public void testNoRefill() throws Exception { MockFiller filler = new MockFiller(); ValueQueue vq = - new ValueQueue(10, 0.5f, 300, 1, + new ValueQueue(10, 0.5f, 30000, 1, SyncGenerationPolicy.ALL, filler); + // Trigger a prefill (5) and an async refill (6) Assert.assertEquals("test", vq.getNext("k1")); Assert.assertEquals(5, filler.getTop().num); + + // Wait for the async task to finish + waitForRefill(vq, "k1", 10); + // Refill task should add 6 values to get to a full queue (5 produced by + // the prefill to the low watermark, 1 consumed by getNext()) + Assert.assertEquals(6, filler.getTop().num); + + // Take another value, queue is still above the watermark + Assert.assertEquals("test", vq.getNext("k1")); + + // Wait a while to make sure that no async refills are triggered + try { + waitForRefill(vq, "k1", 10); + } catch (TimeoutException ignored) { + // This is the correct outcome - no refill is expected + } Assert.assertEquals(null, filler.getTop()); vq.shutdown(); } @@ -140,11 +172,29 @@ public void testNoRefill() throws Exception { public void testgetAtMostPolicyALL() throws Exception { MockFiller filler = new MockFiller(); final ValueQueue vq = - new ValueQueue(10, 0.1f, 300, 1, + new ValueQueue(10, 0.1f, 30000, 1, SyncGenerationPolicy.ALL, filler); + // Trigger a prefill (1) and an async refill (10) Assert.assertEquals("test", vq.getNext("k1")); Assert.assertEquals(1, filler.getTop().num); + // Wait for the async task to finish + waitForRefill(vq, "k1", 10); + // Refill task should add 10 values to get to a full queue (1 produced by + // the prefill to the low watermark, 1 consumed by getNext()) + Assert.assertEquals(10, filler.getTop().num); + + // Drain completely, no further refills triggered + vq.drain("k1"); + + // Wait a while to make sure that no async refills are triggered + try { + waitForRefill(vq, "k1", 10); + } catch (TimeoutException ignored) { + // This is the correct outcome - no refill is expected + } + Assert.assertNull(filler.getTop()); + // Synchronous call: // 1. Synchronously fill returned list // 2. Start another async task to fill the queue in the cache @@ -154,23 +204,16 @@ public void testgetAtMostPolicyALL() throws Exception { filler.getTop().num); // Wait for the async task to finish - GenericTestUtils.waitFor(new Supplier() { - @Override - public Boolean get() { - int size = vq.getSize("k1"); - if (size != 10) { - LOG.info("Current ValueQueue size is " + size); - return false; - } - return true; - } - }, 100, 3000); + waitForRefill(vq, "k1", 10); + // Refill task should add 10 values to get to a full queue Assert.assertEquals("Failed in async call.", 10, filler.getTop().num); // Drain completely after filled by the async thread - Assert.assertEquals("Failed to drain completely after async.", 10, - vq.getAtMost("k1", 10).size()); - // Synchronous call (No Async call since num > lowWatermark) + vq.drain("k1"); + Assert.assertEquals("Failed to drain completely after async.", 0, + vq.getSize("k1")); + + // Synchronous call Assert.assertEquals("Failed to get all 19.", 19, vq.getAtMost("k1", 19).size()); Assert.assertEquals("Failed in sync call.", 19, filler.getTop().num); @@ -184,14 +227,29 @@ public Boolean get() { public void testgetAtMostPolicyATLEAST_ONE() throws Exception { MockFiller filler = new MockFiller(); ValueQueue vq = - new ValueQueue(10, 0.3f, 300, 1, + new ValueQueue(10, 0.3f, 30000, 1, SyncGenerationPolicy.ATLEAST_ONE, filler); + // Trigger a prefill (3) and an async refill (8) Assert.assertEquals("test", vq.getNext("k1")); Assert.assertEquals(3, filler.getTop().num); - // Drain completely - Assert.assertEquals(2, vq.getAtMost("k1", 10).size()); - // Asynch Refill call - Assert.assertEquals(10, filler.getTop().num); + + // Wait for the async task to finish + waitForRefill(vq, "k1", 10); + // Refill task should add 8 values to get to a full queue (3 produced by + // the prefill to the low watermark, 1 consumed by getNext()) + Assert.assertEquals("Failed in async call.", 8, filler.getTop().num); + + // Drain completely, no further refills triggered + vq.drain("k1"); + + // Queue is empty, sync will return a single value and trigger a refill + Assert.assertEquals(1, vq.getAtMost("k1", 10).size()); + Assert.assertEquals(1, filler.getTop().num); + + // Wait for the async task to finish + waitForRefill(vq, "k1", 10); + // Refill task should add 10 values to get to a full queue + Assert.assertEquals("Failed in async call.", 10, filler.getTop().num); vq.shutdown(); } @@ -202,16 +260,29 @@ public void testgetAtMostPolicyATLEAST_ONE() throws Exception { public void testgetAtMostPolicyLOW_WATERMARK() throws Exception { MockFiller filler = new MockFiller(); ValueQueue vq = - new ValueQueue(10, 0.3f, 300, 1, + new ValueQueue(10, 0.3f, 30000, 1, SyncGenerationPolicy.LOW_WATERMARK, filler); + // Trigger a prefill (3) and an async refill (8) Assert.assertEquals("test", vq.getNext("k1")); Assert.assertEquals(3, filler.getTop().num); - // Drain completely + + // Wait for the async task to finish + waitForRefill(vq, "k1", 10); + // Refill task should add 8 values to get to a full queue (3 produced by + // the prefill to the low watermark, 1 consumed by getNext()) + Assert.assertEquals("Failed in async call.", 8, filler.getTop().num); + + // Drain completely, no further refills triggered + vq.drain("k1"); + + // Queue is empty, sync will return 3 values and trigger a refill Assert.assertEquals(3, vq.getAtMost("k1", 10).size()); - // Synchronous call - Assert.assertEquals(1, filler.getTop().num); - // Asynch Refill call - Assert.assertEquals(10, filler.getTop().num); + Assert.assertEquals(3, filler.getTop().num); + + // Wait for the async task to finish + waitForRefill(vq, "k1", 10); + // Refill task should add 10 values to get to a full queue + Assert.assertEquals("Failed in async call.", 10, filler.getTop().num); vq.shutdown(); } @@ -219,11 +290,27 @@ public void testgetAtMostPolicyLOW_WATERMARK() throws Exception { public void testDrain() throws Exception { MockFiller filler = new MockFiller(); ValueQueue vq = - new ValueQueue(10, 0.1f, 300, 1, + new ValueQueue(10, 0.1f, 30000, 1, SyncGenerationPolicy.ALL, filler); + // Trigger a prefill (1) and an async refill (10) Assert.assertEquals("test", vq.getNext("k1")); Assert.assertEquals(1, filler.getTop().num); + + // Wait for the async task to finish + waitForRefill(vq, "k1", 10); + // Refill task should add 10 values to get to a full queue (1 produced by + // the prefill to the low watermark, 1 consumed by getNext()) + Assert.assertEquals(10, filler.getTop().num); + + // Drain completely, no further refills triggered vq.drain("k1"); + + // Wait a while to make sure that no async refills are triggered + try { + waitForRefill(vq, "k1", 10); + } catch (TimeoutException ignored) { + // This is the correct outcome - no refill is expected + } Assert.assertNull(filler.getTop()); vq.shutdown(); } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextURIBase.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextURIBase.java index a99f7625e0c9e..abb6d4f901591 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextURIBase.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextURIBase.java @@ -130,7 +130,7 @@ public void testCreateFileWithNullName() throws IOException { @Test public void testCreateExistingFile() throws IOException { - String fileName = "testFile"; + String fileName = "testCreateExistingFile"; Path testPath = qualifiedPath(fileName, fc2); // Ensure file does not exist @@ -153,7 +153,7 @@ public void testCreateExistingFile() throws IOException { @Test public void testCreateFileInNonExistingDirectory() throws IOException { - String fileName = "testDir/testFile"; + String fileName = "testCreateFileInNonExistingDirectory/testFile"; Path testPath = qualifiedPath(fileName, fc2); @@ -165,7 +165,8 @@ public void testCreateFileInNonExistingDirectory() throws IOException { // Ensure using fc2 that file is created Assert.assertTrue(isDir(fc2, testPath.getParent())); - Assert.assertEquals("testDir", testPath.getParent().getName()); + Assert.assertEquals("testCreateFileInNonExistingDirectory", + testPath.getParent().getName()); Assert.assertTrue(exists(fc2, testPath)); } @@ -293,7 +294,7 @@ public void testIsDirectory() throws IOException { @Test public void testDeleteFile() throws IOException { - Path testPath = qualifiedPath("testFile", fc2); + Path testPath = qualifiedPath("testDeleteFile", fc2); // Ensure file does not exist Assert.assertFalse(exists(fc2, testPath)); @@ -314,7 +315,7 @@ public void testDeleteFile() throws IOException { @Test public void testDeleteNonExistingFile() throws IOException { - String testFileName = "testFile"; + String testFileName = "testDeleteNonExistingFile"; Path testPath = qualifiedPath(testFileName, fc2); // TestCase1 : Test delete on file never existed @@ -341,7 +342,7 @@ public void testDeleteNonExistingFile() throws IOException { @Test public void testDeleteNonExistingFileInDir() throws IOException { - String testFileInDir = "testDir/testDir/TestFile"; + String testFileInDir = "testDeleteNonExistingFileInDir/testDir/TestFile"; Path testPath = qualifiedPath(testFileInDir, fc2); // TestCase1 : Test delete on file never existed @@ -418,7 +419,7 @@ public void testDeleteDirectory() throws IOException { @Test public void testDeleteNonExistingDirectory() throws IOException { - String testDirName = "testFile"; + String testDirName = "testDeleteNonExistingDirectory"; Path testPath = qualifiedPath(testDirName, fc2); // TestCase1 : Test delete on directory never existed @@ -445,7 +446,7 @@ public void testDeleteNonExistingDirectory() throws IOException { @Test public void testModificationTime() throws IOException { - String testFile = "file1"; + String testFile = "testModificationTime"; long fc2ModificationTime, fc1ModificationTime; Path testPath = qualifiedPath(testFile, fc2); @@ -461,7 +462,7 @@ public void testModificationTime() throws IOException { @Test public void testFileStatus() throws IOException { - String fileName = "file1"; + String fileName = "testModificationTime"; Path path2 = fc2.makeQualified(new Path(BASE, fileName)); // Create a file on fc2's file system using fc1 diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestChecksumFs.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestChecksumFs.java new file mode 100644 index 0000000000000..0959845963000 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestChecksumFs.java @@ -0,0 +1,135 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs; + +import java.io.IOException; +import java.util.EnumSet; + +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.permission.FsPermission; +import org.apache.hadoop.test.GenericTestUtils; +import org.apache.hadoop.test.HadoopTestBase; + +import static org.apache.hadoop.fs.CreateFlag.*; + +/** + * This class tests the functionality of ChecksumFs. + */ +public class TestChecksumFs extends HadoopTestBase { + private Configuration conf; + private Path testRootDirPath; + private FileContext fc; + + @Before + public void setUp() throws Exception { + conf = getTestConfiguration(); + fc = FileContext.getFileContext(conf); + testRootDirPath = new Path(GenericTestUtils.getRandomizedTestDir() + .getAbsolutePath()); + mkdirs(testRootDirPath); + } + + @After + public void tearDown() throws Exception { + if (fc != null) { + fc.delete(testRootDirPath, true); + } + } + + @Test + public void testRenameFileToFile() throws Exception { + Path srcPath = new Path(testRootDirPath, "testRenameSrc"); + Path dstPath = new Path(testRootDirPath, "testRenameDst"); + verifyRename(srcPath, dstPath, false); + } + + @Test + public void testRenameFileToFileWithOverwrite() throws Exception { + Path srcPath = new Path(testRootDirPath, "testRenameSrc"); + Path dstPath = new Path(testRootDirPath, "testRenameDst"); + verifyRename(srcPath, dstPath, true); + } + + @Test + public void testRenameFileIntoDirFile() throws Exception { + Path srcPath = new Path(testRootDirPath, "testRenameSrc"); + Path dstPath = new Path(testRootDirPath, "testRenameDir/testRenameDst"); + mkdirs(dstPath); + verifyRename(srcPath, dstPath, false); + } + + @Test + public void testRenameFileIntoDirFileWithOverwrite() throws Exception { + Path srcPath = new Path(testRootDirPath, "testRenameSrc"); + Path dstPath = new Path(testRootDirPath, "testRenameDir/testRenameDst"); + mkdirs(dstPath); + verifyRename(srcPath, dstPath, true); + } + + private void verifyRename(Path srcPath, Path dstPath, + boolean overwrite) throws Exception { + ChecksumFs fs = (ChecksumFs) fc.getDefaultFileSystem(); + + fs.delete(srcPath, true); + fs.delete(dstPath, true); + + Options.Rename renameOpt = Options.Rename.NONE; + if (overwrite) { + renameOpt = Options.Rename.OVERWRITE; + createTestFile(fs, dstPath, 2); + } + + // ensure file + checksum are moved + createTestFile(fs, srcPath, 1); + assertTrue("Checksum file doesn't exist for source file - " + srcPath, + fc.util().exists(fs.getChecksumFile(srcPath))); + fs.rename(srcPath, dstPath, renameOpt); + assertTrue("Checksum file doesn't exist for dest file - " + srcPath, + fc.util().exists(fs.getChecksumFile(dstPath))); + try (FSDataInputStream is = fs.open(dstPath)) { + assertEquals(1, is.readInt()); + } + } + + private static Configuration getTestConfiguration() { + Configuration conf = new Configuration(false); + conf.set("fs.defaultFS", "file:///"); + conf.setClass("fs.AbstractFileSystem.file.impl", + org.apache.hadoop.fs.local.LocalFs.class, + org.apache.hadoop.fs.AbstractFileSystem.class); + return conf; + } + + private void createTestFile(ChecksumFs fs, Path path, int content) + throws IOException { + try (FSDataOutputStream fout = fs.create(path, + EnumSet.of(CREATE, OVERWRITE), + Options.CreateOpts.perms(FsPermission.getDefault()))) { + fout.writeInt(content); + } + } + + private void mkdirs(Path dirPath) throws IOException { + fc.mkdir(dirPath, FileContext.DEFAULT_PERM, true); + } +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java index f5571038a6b5a..5d22a6a2a4896 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java @@ -1493,4 +1493,10 @@ public void testReadSymlinkWithAFileAsInput() throws IOException { file.delete(); } + /** + * The size of FileSystem cache. + */ + public static int getCacheSize() { + return FileSystem.cacheSize(); + } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java index e7f42ff2b5603..f0057a6c6d902 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java @@ -77,7 +77,6 @@ public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException; - public boolean mkdirs(Path f); public FSDataInputStream open(Path f); public FSDataInputStream open(PathHandle f); public FSDataOutputStream create(Path f); @@ -135,6 +134,8 @@ public Token[] addDelegationTokens(String renewer, Credentials creds) public Path fixRelativePart(Path p); public ContentSummary getContentSummary(Path f); public QuotaUsage getQuotaUsage(Path f); + void setQuota(Path f, long namespaceQuota, long storagespaceQuota); + void setQuotaByStorageType(Path f, StorageType type, long quota); StorageStatistics getStorageStatistics(); } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java index f9b2420067029..72ae296c957b5 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java @@ -189,7 +189,7 @@ private void checkPut(Path srcPath, Path targetDir, boolean useWindowsPath) // copy to new file, then again prepPut(dstPath, false, false); checkPut(0, srcPath, dstPath, useWindowsPath); - if (lfs.isFile(srcPath)) { + if (lfs.getFileStatus(srcPath).isFile()) { checkPut(1, srcPath, dstPath, useWindowsPath); } else { // directory works because it copies into the dir // clear contents so the check won't think there are extra paths @@ -228,11 +228,11 @@ private void prepPut(Path dst, boolean create, if (create) { if (isDir) { lfs.mkdirs(dst); - assertTrue(lfs.isDirectory(dst)); + assertTrue(lfs.getFileStatus(dst).isDirectory()); } else { lfs.mkdirs(new Path(dst.getName())); lfs.create(dst).close(); - assertTrue(lfs.isFile(dst)); + assertTrue(lfs.getFileStatus(dst).isFile()); } } } @@ -253,7 +253,7 @@ private void checkPut(int exitCode, Path src, Path dest, Path target; if (lfs.exists(dest)) { - if (lfs.isDirectory(dest)) { + if (lfs.getFileStatus(dest).isDirectory()) { target = new Path(pathAsString(dest), src.getName()); } else { target = dest; @@ -276,7 +276,8 @@ private void checkPut(int exitCode, Path src, Path dest, if (exitCode == 0) { assertTrue(lfs.exists(target)); - assertTrue(lfs.isFile(src) == lfs.isFile(target)); + assertTrue(lfs.getFileStatus(src).isFile() == + lfs.getFileStatus(target).isFile()); assertEquals(1, lfs.listStatus(lfs.makeQualified(target).getParent()).length); } else { assertEquals(targetExists, lfs.exists(target)); @@ -293,7 +294,7 @@ public void testRepresentsDir() throws Exception { argv = new String[]{ "-put", srcPath.toString(), dstPath.toString() }; assertEquals(0, shell.run(argv)); - assertTrue(lfs.exists(dstPath) && lfs.isFile(dstPath)); + assertTrue(lfs.exists(dstPath) && lfs.getFileStatus(dstPath).isFile()); lfs.delete(dstPath, true); assertFalse(lfs.exists(dstPath)); @@ -319,7 +320,7 @@ public void testRepresentsDir() throws Exception { "-put", srcPath.toString(), dstPath.toString()+suffix }; assertEquals(0, shell.run(argv)); assertTrue(lfs.exists(subdirDstPath)); - assertTrue(lfs.isFile(subdirDstPath)); + assertTrue(lfs.getFileStatus(subdirDstPath).isFile()); } // ensure .. is interpreted as a dir @@ -329,7 +330,7 @@ public void testRepresentsDir() throws Exception { argv = new String[]{ "-put", srcPath.toString(), dotdotDst }; assertEquals(0, shell.run(argv)); assertTrue(lfs.exists(subdirDstPath)); - assertTrue(lfs.isFile(subdirDstPath)); + assertTrue(lfs.getFileStatus(subdirDstPath).isFile()); } @Test @@ -442,9 +443,33 @@ public void testMoveFileFromLocal() throws Exception { assertEquals(0, exit); assertFalse(lfs.exists(srcFile)); assertTrue(lfs.exists(target)); - assertTrue(lfs.isFile(target)); + assertTrue(lfs.getFileStatus(target).isFile()); } - + + @Test + public void testMoveFileFromLocalDestExists() throws Exception{ + Path testRoot = new Path(testRootDir, "testPutFile"); + lfs.delete(testRoot, true); + lfs.mkdirs(testRoot); + + Path target = new Path(testRoot, "target"); + Path srcFile = new Path(testRoot, new Path("srcFile")); + lfs.createNewFile(srcFile); + + int exit = shell.run(new String[]{ + "-moveFromLocal", srcFile.toString(), target.toString()}); + assertEquals(0, exit); + assertFalse(lfs.exists(srcFile)); + assertTrue(lfs.exists(target)); + assertTrue(lfs.getFileStatus(target).isFile()); + + lfs.createNewFile(srcFile); + exit = shell.run(new String[]{ + "-moveFromLocal", srcFile.toString(), target.toString()}); + assertEquals(1, exit); + assertTrue(lfs.exists(srcFile)); + } + @Test public void testMoveDirFromLocal() throws Exception { Path testRoot = new Path(testRootDir, "testPutDir"); @@ -502,7 +527,7 @@ public void testMoveFromWindowsLocalPath() throws Exception { shellRun(0, "-moveFromLocal", winSrcFile, target.toString()); assertFalse(lfs.exists(srcFile)); assertTrue(lfs.exists(target)); - assertTrue(lfs.isFile(target)); + assertTrue(lfs.getFileStatus(target).isFile()); } @Test diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java index 57798c2c8b982..3b923e05bd3a5 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java @@ -46,8 +46,7 @@ import static org.apache.hadoop.fs.Options.ChecksumOpt; import static org.apache.hadoop.fs.Options.CreateOpts; import static org.apache.hadoop.fs.Options.Rename; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; +import static org.assertj.core.api.Assertions.assertThat; @SuppressWarnings("deprecation") public class TestHarFileSystem { @@ -118,6 +117,8 @@ public FSDataOutputStream create(Path f, FsPermission permission, public void processDeleteOnExit(); public ContentSummary getContentSummary(Path f); public QuotaUsage getQuotaUsage(Path f); + void setQuota(Path f, long namespaceQuota, long storagespaceQuota); + void setQuotaByStorageType(Path f, StorageType type, long quota); public FsStatus getStatus(); public FileStatus[] listStatus(Path f, PathFilter filter); public FileStatus[] listStatusBatch(Path f, byte[] token); @@ -277,13 +278,8 @@ static void checkInvalidPath(String s, Configuration conf) { @Test public void testFileChecksum() throws Exception { final Path p = new Path("har://file-localhost/foo.har/file1"); - final HarFileSystem harfs = new HarFileSystem(); - try { - Assert.assertEquals(null, harfs.getFileChecksum(p)); - } finally { - if (harfs != null) { - harfs.close(); - } + try (HarFileSystem harfs = new HarFileSystem()) { + assertThat(harfs.getFileChecksum(p)).isNull(); } } @@ -297,30 +293,30 @@ public void testFixBlockLocations() { // case 1: range starts before current har block and ends after BlockLocation[] b = { new BlockLocation(null, null, 10, 10) }; HarFileSystem.fixBlockLocations(b, 0, 20, 5); - assertEquals(b[0].getOffset(), 5); - assertEquals(b[0].getLength(), 10); + assertThat(b[0].getOffset()).isEqualTo(5); + assertThat(b[0].getLength()).isEqualTo(10); } { // case 2: range starts in current har block and ends after BlockLocation[] b = { new BlockLocation(null, null, 10, 10) }; HarFileSystem.fixBlockLocations(b, 0, 20, 15); - assertEquals(b[0].getOffset(), 0); - assertEquals(b[0].getLength(), 5); + assertThat(b[0].getOffset()).isZero(); + assertThat(b[0].getLength()).isEqualTo(5); } { // case 3: range starts before current har block and ends in // current har block BlockLocation[] b = { new BlockLocation(null, null, 10, 10) }; HarFileSystem.fixBlockLocations(b, 0, 10, 5); - assertEquals(b[0].getOffset(), 5); - assertEquals(b[0].getLength(), 5); + assertThat(b[0].getOffset()).isEqualTo(5); + assertThat(b[0].getLength()).isEqualTo(5); } { // case 4: range starts and ends in current har block BlockLocation[] b = { new BlockLocation(null, null, 10, 10) }; HarFileSystem.fixBlockLocations(b, 0, 6, 12); - assertEquals(b[0].getOffset(), 0); - assertEquals(b[0].getLength(), 6); + assertThat(b[0].getOffset()).isZero(); + assertThat(b[0].getLength()).isEqualTo(6); } // now try a range where start == 3 @@ -328,30 +324,30 @@ public void testFixBlockLocations() { // case 5: range starts before current har block and ends after BlockLocation[] b = { new BlockLocation(null, null, 10, 10) }; HarFileSystem.fixBlockLocations(b, 3, 20, 5); - assertEquals(b[0].getOffset(), 5); - assertEquals(b[0].getLength(), 10); + assertThat(b[0].getOffset()).isEqualTo(5); + assertThat(b[0].getLength()).isEqualTo(10); } { // case 6: range starts in current har block and ends after BlockLocation[] b = { new BlockLocation(null, null, 10, 10) }; HarFileSystem.fixBlockLocations(b, 3, 20, 15); - assertEquals(b[0].getOffset(), 3); - assertEquals(b[0].getLength(), 2); + assertThat(b[0].getOffset()).isEqualTo(3); + assertThat(b[0].getLength()).isEqualTo(2); } { // case 7: range starts before current har block and ends in // current har block BlockLocation[] b = { new BlockLocation(null, null, 10, 10) }; HarFileSystem.fixBlockLocations(b, 3, 7, 5); - assertEquals(b[0].getOffset(), 5); - assertEquals(b[0].getLength(), 5); + assertThat(b[0].getOffset()).isEqualTo(5); + assertThat(b[0].getLength()).isEqualTo(5); } { // case 8: range starts and ends in current har block BlockLocation[] b = { new BlockLocation(null, null, 10, 10) }; HarFileSystem.fixBlockLocations(b, 3, 3, 12); - assertEquals(b[0].getOffset(), 3); - assertEquals(b[0].getLength(), 3); + assertThat(b[0].getOffset()).isEqualTo(3); + assertThat(b[0].getLength()).isEqualTo(3); } // test case from JIRA MAPREDUCE-1752 @@ -359,10 +355,10 @@ public void testFixBlockLocations() { BlockLocation[] b = { new BlockLocation(null, null, 512, 512), new BlockLocation(null, null, 1024, 512) }; HarFileSystem.fixBlockLocations(b, 0, 512, 896); - assertEquals(b[0].getOffset(), 0); - assertEquals(b[0].getLength(), 128); - assertEquals(b[1].getOffset(), 128); - assertEquals(b[1].getLength(), 384); + assertThat(b[0].getOffset()).isZero(); + assertThat(b[0].getLength()).isEqualTo(128); + assertThat(b[1].getOffset()).isEqualTo(128); + assertThat(b[1].getLength()).isEqualTo(384); } } @@ -394,7 +390,9 @@ public void testInheritedMethodsImplemented() throws Exception { } } } - assertTrue((errors + " methods were not overridden correctly - see log"), - errors <= 0); + assertThat(errors) + .withFailMessage(errors + + " methods were not overridden correctly - see log") + .isLessThanOrEqualTo(0); } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java index bffcfa76f207e..517f6ce016544 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java @@ -44,7 +44,11 @@ import static org.apache.hadoop.test.PlatformAssumptions.assumeNotWindows; import static org.apache.hadoop.test.PlatformAssumptions.assumeWindows; -import static org.junit.Assert.*; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; import static org.mockito.Mockito.*; import org.junit.After; @@ -56,6 +60,7 @@ import javax.annotation.Nonnull; +import static org.assertj.core.api.Assertions.assertThat; /** * This class tests the local file system via the FileSystem abstraction. @@ -692,27 +697,33 @@ public void testFSOutputStreamBuilder() throws Exception { FSDataOutputStreamBuilder builder = fileSys.createFile(path); try (FSDataOutputStream stream = builder.build()) { - Assert.assertEquals("Should be default block size", - builder.getBlockSize(), fileSys.getDefaultBlockSize()); - Assert.assertEquals("Should be default replication factor", - builder.getReplication(), fileSys.getDefaultReplication()); - Assert.assertEquals("Should be default buffer size", - builder.getBufferSize(), - fileSys.getConf().getInt(IO_FILE_BUFFER_SIZE_KEY, + assertThat(builder.getBlockSize()) + .withFailMessage("Should be default block size") + .isEqualTo(fileSys.getDefaultBlockSize()); + assertThat(builder.getReplication()) + .withFailMessage("Should be default replication factor") + .isEqualTo(fileSys.getDefaultReplication()); + assertThat(builder.getBufferSize()) + .withFailMessage("Should be default buffer size") + .isEqualTo(fileSys.getConf().getInt(IO_FILE_BUFFER_SIZE_KEY, IO_FILE_BUFFER_SIZE_DEFAULT)); - Assert.assertEquals("Should be default permission", - builder.getPermission(), FsPermission.getFileDefault()); + assertThat(builder.getPermission()) + .withFailMessage("Should be default permission") + .isEqualTo(FsPermission.getFileDefault()); } // Test set 0 to replication, block size and buffer size builder = fileSys.createFile(path); builder.bufferSize(0).blockSize(0).replication((short) 0); - Assert.assertEquals("Block size should be 0", - builder.getBlockSize(), 0); - Assert.assertEquals("Replication factor should be 0", - builder.getReplication(), 0); - Assert.assertEquals("Buffer size should be 0", - builder.getBufferSize(), 0); + assertThat(builder.getBlockSize()) + .withFailMessage("Block size should be 0") + .isZero(); + assertThat(builder.getReplication()) + .withFailMessage("Replication factor should be 0") + .isZero(); + assertThat(builder.getBufferSize()) + .withFailMessage("Buffer size should be 0") + .isZero(); } /** diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestRawLocalFileSystemContract.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestRawLocalFileSystemContract.java index 2e514c4648ea9..b51419d8c53f9 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestRawLocalFileSystemContract.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestRawLocalFileSystemContract.java @@ -24,12 +24,14 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.test.GenericTestUtils; import org.apache.hadoop.test.StatUtils; +import org.apache.hadoop.util.NativeCodeLoader; import org.apache.hadoop.util.Shell; import org.junit.Before; import org.junit.Test; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; +import static org.junit.Assume.assumeTrue; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -127,6 +129,8 @@ protected boolean filesystemIsCaseSensitive() { @Test @SuppressWarnings("deprecation") public void testPermission() throws Exception { + assumeTrue("No native library", + NativeCodeLoader.isNativeCodeLoaded()); Path testDir = getTestBaseDir(); String testFilename = "teststat2File"; Path path = new Path(testDir, testFilename); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java index cf22f3b10b58d..e8e028732b2a8 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java @@ -38,7 +38,10 @@ import org.junit.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import static org.junit.Assert.*; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotEquals; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.TrashPolicyDefault.Emptier; @@ -112,7 +115,7 @@ static void checkNotInTrash(FileSystem fs, Path trashRoot, String pathname) * @throws IOException */ public static void trashShell(final FileSystem fs, final Path base) - throws IOException { + throws Exception { Configuration conf = new Configuration(); conf.set("fs.defaultFS", fs.getUri().toString()); trashShell(conf, base, null, null); @@ -127,8 +130,7 @@ public static void trashShell(final FileSystem fs, final Path base) * @throws IOException */ public static void trashShell(final Configuration conf, final Path base, - FileSystem trashRootFs, Path trashRoot) - throws IOException { + FileSystem trashRootFs, Path trashRoot) throws Exception { FileSystem fs = FileSystem.get(conf); conf.setLong(FS_TRASH_INTERVAL_KEY, 0); // disabled @@ -163,13 +165,9 @@ public static void trashShell(final Configuration conf, final Path base, String[] args = new String[1]; args[0] = "-expunge"; int val = -1; - try { - val = shell.run(args); - } catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } - assertTrue(val == 0); + val = shell.run(args); + + assertEquals("Expunge should return zero", 0, val); } // Verify that we succeed in removing the file we created. @@ -179,15 +177,10 @@ public static void trashShell(final Configuration conf, final Path base, args[0] = "-rm"; args[1] = myFile.toString(); int val = -1; - try { - val = shell.run(args); - } catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } - assertTrue(val == 0); + val = shell.run(args); + + assertEquals("Remove should return zero", 0, val); - checkTrash(trashRootFs, trashRoot, fs.makeQualified(myFile)); } @@ -200,13 +193,9 @@ public static void trashShell(final Configuration conf, final Path base, args[0] = "-rm"; args[1] = new Path(base, "test/mkdirs/myFile").toString(); int val = -1; - try { - val = shell.run(args); - } catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } - assertTrue(val == 0); + val = shell.run(args); + + assertEquals("Remove should return zero", 0, val); } // Verify that we can recreate the file @@ -219,13 +208,9 @@ public static void trashShell(final Configuration conf, final Path base, args[0] = "-rmr"; args[1] = new Path(base, "test/mkdirs").toString(); int val = -1; - try { - val = shell.run(args); - } catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } - assertTrue(val == 0); + val = shell.run(args); + + assertEquals("Recursive Remove should return zero", 0, val); } // recreate directory @@ -237,29 +222,22 @@ public static void trashShell(final Configuration conf, final Path base, args[0] = "-rmr"; args[1] = new Path(base, "test/mkdirs").toString(); int val = -1; - try { - val = shell.run(args); - } catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } - assertTrue(val == 0); + val = shell.run(args); + + assertEquals("Recursive Remove should return zero", 0, val); } // Check that we can delete a file from the trash { - Path toErase = new Path(trashRoot, "toErase"); - int retVal = -1; - writeFile(trashRootFs, toErase, 10); - try { - retVal = shell.run(new String[] {"-rm", toErase.toString()}); - } catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } - assertTrue(retVal == 0); - checkNotInTrash (trashRootFs, trashRoot, toErase.toString()); - checkNotInTrash (trashRootFs, trashRoot, toErase.toString()+".1"); + Path toErase = new Path(trashRoot, "toErase"); + int val = -1; + writeFile(trashRootFs, toErase, 10); + + val = shell.run(new String[] {"-rm", toErase.toString()}); + + assertEquals("Recursive Remove should return zero", 0, val); + checkNotInTrash(trashRootFs, trashRoot, toErase.toString()); + checkNotInTrash(trashRootFs, trashRoot, toErase.toString()+".1"); } // simulate Trash removal @@ -267,17 +245,14 @@ public static void trashShell(final Configuration conf, final Path base, String[] args = new String[1]; args[0] = "-expunge"; int val = -1; - try { - val = shell.run(args); - } catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } - assertTrue(val == 0); + val = shell.run(args); + + assertEquals("Expunge should return zero", 0, val); } // verify that after expunging the Trash, it really goes away - checkNotInTrash(trashRootFs, trashRoot, new Path(base, "test/mkdirs/myFile").toString()); + checkNotInTrash(trashRootFs, trashRoot, new Path( + base, "test/mkdirs/myFile").toString()); // recreate directory and file mkdir(fs, myPath); @@ -289,26 +264,18 @@ public static void trashShell(final Configuration conf, final Path base, args[0] = "-rm"; args[1] = myFile.toString(); int val = -1; - try { - val = shell.run(args); - } catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } - assertTrue(val == 0); + val = shell.run(args); + + assertEquals("Remove should return zero", 0, val); checkTrash(trashRootFs, trashRoot, myFile); args = new String[2]; args[0] = "-rmr"; args[1] = myPath.toString(); val = -1; - try { - val = shell.run(args); - } catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } - assertTrue(val == 0); + val = shell.run(args); + + assertEquals("Recursive Remove should return zero", 0, val); checkTrash(trashRootFs, trashRoot, myPath); } @@ -318,13 +285,9 @@ public static void trashShell(final Configuration conf, final Path base, args[0] = "-rmr"; args[1] = trashRoot.getParent().getParent().toString(); int val = -1; - try { - val = shell.run(args); - } catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } - assertEquals("exit code", 1, val); + val = shell.run(args); + + assertEquals("Recursive Remove should return exit code 1", 1, val); assertTrue(trashRootFs.exists(trashRoot)); } @@ -341,23 +304,18 @@ public static void trashShell(final Configuration conf, final Path base, args[1] = "-skipTrash"; args[2] = myFile.toString(); int val = -1; - try { - // Clear out trash - assertEquals("-expunge failed", - 0, shell.run(new String [] { "-expunge" } )); - - val = shell.run(args); - - }catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } + // Clear out trash + assertEquals("-expunge failed", + 0, shell.run(new String[] {"-expunge" })); + + val = shell.run(args); + assertFalse("Expected TrashRoot (" + trashRoot + ") to exist in file system:" + trashRootFs.getUri(), trashRootFs.exists(trashRoot)); // No new Current should be created assertFalse(fs.exists(myFile)); - assertTrue(val == 0); + assertEquals("Remove with skipTrash should return zero", 0, val); } // recreate directory and file @@ -372,64 +330,52 @@ public static void trashShell(final Configuration conf, final Path base, args[2] = myPath.toString(); int val = -1; - try { - // Clear out trash - assertEquals(0, shell.run(new String [] { "-expunge" } )); - - val = shell.run(args); + // Clear out trash + assertEquals(0, shell.run(new String[] {"-expunge" })); - }catch (Exception e) { - System.err.println("Exception raised from Trash.run " + - e.getLocalizedMessage()); - } + val = shell.run(args); assertFalse(trashRootFs.exists(trashRoot)); // No new Current should be created assertFalse(fs.exists(myPath)); assertFalse(fs.exists(myFile)); - assertTrue(val == 0); + assertEquals("Remove with skipTrash should return zero", 0, val); } // deleting same file multiple times { int val = -1; mkdir(fs, myPath); - - try { - assertEquals(0, shell.run(new String [] { "-expunge" } )); - } catch (Exception e) { - System.err.println("Exception raised from fs expunge " + - e.getLocalizedMessage()); - } + assertEquals("Expunge should return zero", + 0, shell.run(new String[] {"-expunge" })); + // create a file in that directory. myFile = new Path(base, "test/mkdirs/myFile"); - String [] args = new String[] {"-rm", myFile.toString()}; + String[] args = new String[] {"-rm", myFile.toString()}; int num_runs = 10; - for(int i=0;i getFileSystem().concat(target, new Path[]{target}))); } + @Test + public void testFileSystemDeclaresCapability() throws Throwable { + assertHasPathCapabilities(getFileSystem(), target, FS_CONCAT); + } + } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractGetFileStatusTest.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractGetFileStatusTest.java index eb1fd619ca21e..85bd137813f66 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractGetFileStatusTest.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractGetFileStatusTest.java @@ -112,9 +112,11 @@ public void testListFilesEmptyDirectoryRecursive() throws IOException { private void listFilesOnEmptyDir(boolean recursive) throws IOException { describe("Invoke listFiles(recursive=" + recursive + ")" + " on empty directories, expect nothing found"); - Path subfolder = createDirWithEmptySubFolder(); FileSystem fs = getFileSystem(); - new TreeScanResults(fs.listFiles(getContract().getTestPath(), recursive)) + Path path = getContract().getTestPath(); + fs.delete(path, true); + Path subfolder = createDirWithEmptySubFolder(); + new TreeScanResults(fs.listFiles(path, recursive)) .assertSizeEquals("listFiles(test dir, " + recursive + ")", 0, 0, 0); describe("Test on empty subdirectory"); new TreeScanResults(fs.listFiles(subfolder, recursive)) @@ -126,9 +128,11 @@ private void listFilesOnEmptyDir(boolean recursive) throws IOException { public void testListLocatedStatusEmptyDirectory() throws IOException { describe("Invoke listLocatedStatus() on empty directories;" + " expect directories to be found"); - Path subfolder = createDirWithEmptySubFolder(); FileSystem fs = getFileSystem(); - new TreeScanResults(fs.listLocatedStatus(getContract().getTestPath())) + Path path = getContract().getTestPath(); + fs.delete(path, true); + Path subfolder = createDirWithEmptySubFolder(); + new TreeScanResults(fs.listLocatedStatus(path)) .assertSizeEquals("listLocatedStatus(test dir)", 0, 1, 0); describe("Test on empty subdirectory"); new TreeScanResults(fs.listLocatedStatus(subfolder)) diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRenameTest.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRenameTest.java index 5b76a753de170..2751294beb92c 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRenameTest.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRenameTest.java @@ -268,7 +268,7 @@ public void testRenamePopulatesFileAncestors() throws IOException { * @param dst the destination root to move * @param nestedPath the nested path to move */ - private void validateAncestorsMoved(Path src, Path dst, String nestedPath) + protected void validateAncestorsMoved(Path src, Path dst, String nestedPath) throws IOException { assertIsDirectory(dst); assertPathDoesNotExist("src path should not exist", path(src + nestedPath)); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRootDirectoryTest.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRootDirectoryTest.java index 5fba4bfc2786b..27c6933ae1885 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRootDirectoryTest.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRootDirectoryTest.java @@ -31,8 +31,10 @@ import java.util.concurrent.atomic.AtomicInteger; import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.RemoteIterator; import org.apache.hadoop.test.LambdaTestUtils; +import static org.apache.commons.lang3.StringUtils.join; import static org.apache.hadoop.fs.contract.ContractTestUtils.createFile; import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset; import static org.apache.hadoop.fs.contract.ContractTestUtils.deleteChildren; @@ -149,14 +151,18 @@ public void testRmRootRecursive() throws Throwable { Path root = new Path("/"); assertIsDirectory(root); Path file = new Path("/testRmRootRecursive"); - ContractTestUtils.touch(getFileSystem(), file); - boolean deleted = getFileSystem().delete(root, true); - assertIsDirectory(root); - LOG.info("rm -rf / result is {}", deleted); - if (deleted) { - assertPathDoesNotExist("expected file to be deleted", file); - } else { - assertPathExists("expected file to be preserved", file);; + try { + ContractTestUtils.touch(getFileSystem(), file); + boolean deleted = getFileSystem().delete(root, true); + assertIsDirectory(root); + LOG.info("rm -rf / result is {}", deleted); + if (deleted) { + assertPathDoesNotExist("expected file to be deleted", file); + } else { + assertPathExists("expected file to be preserved", file); + } + } finally{ + getFileSystem().delete(file, false); } } @@ -183,30 +189,59 @@ public void testListEmptyRootDirectory() throws IOException { Path root = new Path("/"); FileStatus[] statuses = fs.listStatus(root); for (FileStatus status : statuses) { - ContractTestUtils.assertDeleted(fs, status.getPath(), true); + ContractTestUtils.assertDeleted(fs, status.getPath(), false, true, false); } - assertEquals("listStatus on empty root-directory returned a non-empty list", - 0, fs.listStatus(root).length); - assertFalse("listFiles(/, false).hasNext", - fs.listFiles(root, false).hasNext()); - assertFalse("listFiles(/, true).hasNext", - fs.listFiles(root, true).hasNext()); - assertFalse("listLocatedStatus(/).hasNext", - fs.listLocatedStatus(root).hasNext()); + FileStatus[] rootListStatus = fs.listStatus(root); + assertEquals("listStatus on empty root-directory returned found: " + + join("\n", rootListStatus), + 0, rootListStatus.length); + assertNoElements("listFiles(/, false)", + fs.listFiles(root, false)); + assertNoElements("listFiles(/, true)", + fs.listFiles(root, true)); + assertNoElements("listLocatedStatus(/)", + fs.listLocatedStatus(root)); assertIsDirectory(root); } + /** + * Assert that an iterator has no elements; the raised exception + * will include the element list. + * @param operation operation for assertion text. + * @param iter iterator + * @throws IOException failure retrieving the values. + */ + protected void assertNoElements(String operation, + RemoteIterator iter) throws IOException { + List resultList = toList(iter); + if (!resultList.isEmpty()) { + fail("Expected no results from " + operation + ", but got " + + resultList.size() + " elements:\n" + + join(resultList, "\n")); + } + } + @Test public void testSimpleRootListing() throws IOException { describe("test the nonrecursive root listing calls"); FileSystem fs = getFileSystem(); Path root = new Path("/"); FileStatus[] statuses = fs.listStatus(root); + String listStatusResult = join(statuses, "\n"); List locatedStatusList = toList( fs.listLocatedStatus(root)); - assertEquals(statuses.length, locatedStatusList.size()); + String locatedStatusResult = join(locatedStatusList, "\n"); + + assertEquals("listStatus(/) vs listLocatedStatus(/) with \n" + + "listStatus =" + listStatusResult + +" listLocatedStatus = " + locatedStatusResult, + statuses.length, locatedStatusList.size()); List fileList = toList(fs.listFiles(root, false)); - assertTrue(fileList.size() <= statuses.length); + String listFilesResult = join(fileList, "\n"); + assertTrue("listStatus(/) vs listFiles(/, false) with \n" + + "listStatus = " + listStatusResult + + "listFiles = " + listFilesResult, + fileList.size() <= statuses.length); } @Test diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java index b4db3a5803ad8..f61634943bb7f 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java @@ -25,11 +25,12 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.LocatedFileStatus; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.PathCapabilities; import org.apache.hadoop.fs.RemoteIterator; import org.apache.hadoop.fs.StreamCapabilities; import org.apache.hadoop.io.IOUtils; import org.junit.Assert; -import org.junit.internal.AssumptionViolatedException; +import org.junit.AssumptionViolatedException; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -457,8 +458,10 @@ public static void rejectRootOperation(Path path) throws IOException { public static FileStatus[] deleteChildren(FileSystem fileSystem, Path path, boolean recursive) throws IOException { + LOG.debug("Deleting children of {} (recursive={})", path, recursive); FileStatus[] children = listChildren(fileSystem, path); for (FileStatus entry : children) { + LOG.debug("Deleting {}", entry); fileSystem.delete(entry.getPath(), recursive); } return children; @@ -556,7 +559,8 @@ public static void assertFileHasLength(FileSystem fs, Path path, */ public static void assertIsDirectory(FileSystem fs, Path path) throws IOException { - FileStatus fileStatus = fs.getFileStatus(path); + FileStatus fileStatus = verifyPathExists(fs, + "Expected to find a directory", path); assertIsDirectory(fileStatus); } @@ -672,7 +676,8 @@ public static void touch(FileSystem fs, /** * Delete a file/dir and assert that delete() returned true * and that the path no longer exists. This variant rejects - * all operations on root directories. + * all operations on root directories and requires the target path + * to exist before the deletion operation. * @param fs filesystem * @param file path to delete * @param recursive flag to enable recursive delete @@ -686,20 +691,41 @@ public static void assertDeleted(FileSystem fs, /** * Delete a file/dir and assert that delete() returned true - * and that the path no longer exists. This variant rejects - * all operations on root directories + * and that the path no longer exists. + * This variant requires the target path + * to exist before the deletion operation. + * @param fs filesystem + * @param file path to delete + * @param recursive flag to enable recursive delete + * @param allowRootOperations can the root dir be deleted? + * @throws IOException IO problems + */ + public static void assertDeleted(FileSystem fs, + Path file, + boolean recursive, + boolean allowRootOperations) throws IOException { + assertDeleted(fs, file, true, recursive, allowRootOperations); + } + + /** + * Delete a file/dir and assert that delete() returned true + * and that the path no longer exists. * @param fs filesystem * @param file path to delete + * @param requirePathToExist check for the path existing first? * @param recursive flag to enable recursive delete * @param allowRootOperations can the root dir be deleted? * @throws IOException IO problems */ public static void assertDeleted(FileSystem fs, Path file, + boolean requirePathToExist, boolean recursive, boolean allowRootOperations) throws IOException { rejectRootOperation(file, allowRootOperations); - assertPathExists(fs, "about to be deleted file", file); + if (requirePathToExist) { + assertPathExists(fs, "about to be deleted file", file); + } boolean deleted = fs.delete(file, recursive); String dir = ls(fs, file.getParent()); assertTrue("Delete failed on " + file + ": " + dir, deleted); @@ -1466,22 +1492,61 @@ public static void assertCapabilities( assertTrue("Stream should be instanceof StreamCapabilities", stream instanceof StreamCapabilities); - if (shouldHaveCapabilities!=null) { + StreamCapabilities source = (StreamCapabilities) stream; + if (shouldHaveCapabilities != null) { for (String shouldHaveCapability : shouldHaveCapabilities) { assertTrue("Should have capability: " + shouldHaveCapability, - ((StreamCapabilities) stream).hasCapability(shouldHaveCapability)); + source.hasCapability(shouldHaveCapability)); } } - if (shouldNotHaveCapabilities!=null) { + if (shouldNotHaveCapabilities != null) { for (String shouldNotHaveCapability : shouldNotHaveCapabilities) { assertFalse("Should not have capability: " + shouldNotHaveCapability, - ((StreamCapabilities) stream) - .hasCapability(shouldNotHaveCapability)); + source.hasCapability(shouldNotHaveCapability)); } } } + /** + * Custom assert to test {@link PathCapabilities}. + * + * @param source source (FS, FC, etc) + * @param path path to check + * @param capabilities The array of unexpected capabilities + */ + public static void assertHasPathCapabilities( + final PathCapabilities source, + final Path path, + final String...capabilities) throws IOException { + + for (String shouldHaveCapability: capabilities) { + assertTrue("Should have capability: " + shouldHaveCapability + + " under " + path, + source.hasPathCapability(path, shouldHaveCapability)); + } + } + + /** + * Custom assert to test that the named {@link PathCapabilities} + * are not supported. + * + * @param source source (FS, FC, etc) + * @param path path to check + * @param capabilities The array of unexpected capabilities + */ + public static void assertLacksPathCapabilities( + final PathCapabilities source, + final Path path, + final String...capabilities) throws IOException { + + for (String shouldHaveCapability: capabilities) { + assertFalse("Path must not support capability: " + shouldHaveCapability + + " under " + path, + source.hasPathCapability(path, shouldHaveCapability)); + } + } + /** * Function which calls {@code InputStream.read()} and * downgrades an IOE to a runtime exception. @@ -1603,6 +1668,22 @@ public String toString() { getFileCount() == 1 ? "" : "s"); } + /** + * Dump the files and directories to a multi-line string for error + * messages and assertions. + * @return a dump of the internal state + */ + private String dump() { + StringBuilder sb = new StringBuilder(toString()); + sb.append("\nFiles:"); + directories.forEach(p -> + sb.append("\n \"").append(p.toString())); + sb.append("\nDirectories:"); + files.forEach(p -> + sb.append("\n \"").append(p.toString())); + return sb.toString(); + } + /** * Equality check compares files and directory counts. * As these are non-final fields, this class cannot be used in @@ -1643,7 +1724,7 @@ public int hashCode() { * @param o expected other entries. */ public void assertSizeEquals(String text, long f, long d, long o) { - String self = toString(); + String self = dump(); Assert.assertEquals(text + ": file count in " + self, f, getFileCount()); Assert.assertEquals(text + ": directory count in " + self, diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopy.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopy.java index 74fb34d9fddb6..f73e83d858bc7 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopy.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopy.java @@ -18,6 +18,7 @@ package org.apache.hadoop.fs.shell; +import static org.apache.hadoop.test.GenericTestUtils.assertExceptionContains; import static org.junit.Assert.*; import static org.mockito.Mockito.*; @@ -77,10 +78,19 @@ public void testCopyStreamTarget() throws Exception { when(in.read(any(byte[].class), anyInt(), anyInt())).thenReturn(-1); tryCopyStream(in, true); + verify(in).close(); + verify(out, times(2)).close(); + // no data was written. + verify(out, never()).write(any(byte[].class), anyInt(), anyInt()); verify(mockFs, never()).delete(eq(path), anyBoolean()); verify(mockFs).rename(eq(tmpPath), eq(path)); verify(mockFs, never()).delete(eq(tmpPath), anyBoolean()); verify(mockFs, never()).close(); + // temp path never had is existence checked. This is critical for S3 as it + // avoids the successful path accidentally getting a 404 into the S3 load + // balancer cache + verify(mockFs, never()).exists(eq(tmpPath)); + verify(mockFs, never()).exists(eq(path)); } @Test @@ -110,6 +120,31 @@ public void testInterruptedCreate() throws Exception { FSDataInputStream in = mock(FSDataInputStream.class); tryCopyStream(in, false); + verify(mockFs, never()).rename(any(Path.class), any(Path.class)); + verify(mockFs, never()).delete(eq(tmpPath), anyBoolean()); + verify(mockFs, never()).delete(eq(path), anyBoolean()); + verify(mockFs, never()).close(); + } + + /** + * Create a file but fail in the write. + * The copy operation should attempt to clean up by + * closing the output stream then deleting it. + */ + @Test + public void testFailedWrite() throws Exception { + FSDataOutputStream out = mock(FSDataOutputStream.class); + doThrow(new IOException("mocked")) + .when(out).write(any(byte[].class), anyInt(), anyInt()); + whenFsCreate().thenReturn(out); + when(mockFs.getFileStatus(eq(tmpPath))).thenReturn(fileStat); + FSInputStream in = mock(FSInputStream.class); + doReturn(0) + .when(in).read(any(byte[].class), anyInt(), anyInt()); + Throwable thrown = tryCopyStream(in, false); + assertExceptionContains("mocked", thrown); + verify(in).close(); + verify(out, times(2)).close(); verify(mockFs).delete(eq(tmpPath), anyBoolean()); verify(mockFs, never()).rename(any(Path.class), any(Path.class)); verify(mockFs, never()).delete(eq(path), anyBoolean()); @@ -155,14 +190,21 @@ private OngoingStubbing whenFsCreate() throws IOException { anyBoolean(), anyInt(), anyShort(), anyLong(), any())); } - private void tryCopyStream(InputStream in, boolean shouldPass) { + private Throwable tryCopyStream(InputStream in, boolean shouldPass) { try { cmd.copyStreamToTarget(new FSDataInputStream(in), target); + return null; } catch (InterruptedIOException e) { - assertFalse("copy failed", shouldPass); + if (shouldPass) { + throw new AssertionError("copy failed", e); + } + return e; } catch (Throwable e) { - assertFalse(e.getMessage(), shouldPass); - } + if (shouldPass) { + throw new AssertionError(e.getMessage(), e); + } + return e; + } } static class MockFileSystem extends FilterFileSystem { @@ -183,4 +225,4 @@ public Configuration getConf() { return conf; } } -} \ No newline at end of file +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestChRootedFileSystem.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestChRootedFileSystem.java index 780f40974b0ee..8267b214d53bc 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestChRootedFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestChRootedFileSystem.java @@ -22,6 +22,7 @@ import java.net.URI; import java.util.Collections; import java.util.List; +import java.util.Objects; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; @@ -32,7 +33,6 @@ import org.apache.hadoop.fs.FsConstants; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.permission.AclEntry; -import org.apache.hadoop.fs.viewfs.ChRootedFileSystem; import org.junit.After; import org.junit.Assert; import org.junit.Before; @@ -396,7 +396,7 @@ public void testAclMethodsPathTranslation() throws IOException { } @Test - public void testListLocatedFileStatus() throws IOException { + public void testListLocatedFileStatus() throws Exception { final Path mockMount = new Path("mockfs://foo/user"); final Path mockPath = new Path("/usermock"); final Configuration conf = new Configuration(); @@ -404,17 +404,35 @@ public void testListLocatedFileStatus() throws IOException { ConfigUtil.addLink(conf, mockPath.toString(), mockMount.toUri()); FileSystem vfs = FileSystem.get(URI.create("viewfs:///"), conf); vfs.listLocatedStatus(mockPath); - final FileSystem mockFs = ((MockFileSystem)mockMount.getFileSystem(conf)) - .getRawFileSystem(); + final FileSystem mockFs = + ((MockFileSystem) getChildFileSystem((ViewFileSystem) vfs, + new URI("mockfs://foo/"))).getRawFileSystem(); verify(mockFs).listLocatedStatus(new Path(mockMount.toUri().getPath())); } + static FileSystem getChildFileSystem(ViewFileSystem viewFs, URI uri) { + for (FileSystem fs : viewFs.getChildFileSystems()) { + if (Objects.equals(fs.getUri().getScheme(), uri.getScheme()) && Objects + .equals(fs.getUri().getAuthority(), uri.getAuthority())) { + return fs; + } + } + return null; + } + static class MockFileSystem extends FilterFileSystem { + private URI uri; MockFileSystem() { super(mock(FileSystem.class)); } @Override - public void initialize(URI name, Configuration conf) throws IOException {} + public URI getUri() { + return uri; + } + @Override + public void initialize(URI name, Configuration conf) throws IOException { + uri = name; + } } @Test(timeout = 30000) diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegation.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegation.java index cb1f413bda42a..d8c39f79d0454 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegation.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegation.java @@ -20,6 +20,7 @@ import java.io.IOException; import java.net.URI; +import java.net.URISyntaxException; import java.util.Collections; import java.util.List; import org.apache.hadoop.conf.Configuration; @@ -31,6 +32,8 @@ import org.apache.hadoop.fs.permission.AclEntry; import org.apache.hadoop.fs.viewfs.TestChRootedFileSystem.MockFileSystem; import org.junit.*; + +import static org.apache.hadoop.fs.viewfs.TestChRootedFileSystem.getChildFileSystem; import static org.junit.Assert.*; import static org.mockito.Mockito.*; @@ -46,12 +49,16 @@ public class TestViewFileSystemDelegation { //extends ViewFileSystemTestSetup { @BeforeClass public static void setup() throws Exception { conf = ViewFileSystemTestSetup.createConfig(); - fs1 = setupFileSystem(new URI("fs1:/"), FakeFileSystem.class); - fs2 = setupFileSystem(new URI("fs2:/"), FakeFileSystem.class); + setupFileSystem(new URI("fs1:/"), FakeFileSystem.class); + setupFileSystem(new URI("fs2:/"), FakeFileSystem.class); viewFs = FileSystem.get(FsConstants.VIEWFS_URI, conf); + fs1 = (FakeFileSystem) getChildFileSystem((ViewFileSystem) viewFs, + new URI("fs1:/")); + fs2 = (FakeFileSystem) getChildFileSystem((ViewFileSystem) viewFs, + new URI("fs2:/")); } - static FakeFileSystem setupFileSystem(URI uri, Class clazz) + static void setupFileSystem(URI uri, Class clazz) throws Exception { String scheme = uri.getScheme(); conf.set("fs."+scheme+".impl", clazz.getName()); @@ -59,22 +66,21 @@ static FakeFileSystem setupFileSystem(URI uri, Class clazz) assertEquals(uri, fs.getUri()); Path targetPath = new FileSystemTestHelper().getAbsoluteTestRootPath(fs); ConfigUtil.addLink(conf, "/mounts/"+scheme, targetPath.toUri()); - return fs; } - private static FileSystem setupMockFileSystem(Configuration conf, URI uri) + private static void setupMockFileSystem(Configuration config, URI uri) throws Exception { String scheme = uri.getScheme(); - conf.set("fs." + scheme + ".impl", MockFileSystem.class.getName()); - FileSystem fs = FileSystem.get(uri, conf); - ConfigUtil.addLink(conf, "/mounts/" + scheme, uri); - return ((MockFileSystem)fs).getRawFileSystem(); + config.set("fs." + scheme + ".impl", MockFileSystem.class.getName()); + ConfigUtil.addLink(config, "/mounts/" + scheme, uri); } @Test - public void testSanity() { - assertEquals("fs1:/", fs1.getUri().toString()); - assertEquals("fs2:/", fs2.getUri().toString()); + public void testSanity() throws URISyntaxException { + assertEquals(new URI("fs1:/").getScheme(), fs1.getUri().getScheme()); + assertEquals(new URI("fs1:/").getAuthority(), fs1.getUri().getAuthority()); + assertEquals(new URI("fs2:/").getScheme(), fs2.getUri().getScheme()); + assertEquals(new URI("fs2:/").getAuthority(), fs2.getUri().getAuthority()); } @Test @@ -91,9 +97,15 @@ public void testVerifyChecksum() throws Exception { @Test public void testAclMethods() throws Exception { Configuration conf = ViewFileSystemTestSetup.createConfig(); - FileSystem mockFs1 = setupMockFileSystem(conf, new URI("mockfs1:/")); - FileSystem mockFs2 = setupMockFileSystem(conf, new URI("mockfs2:/")); + setupMockFileSystem(conf, new URI("mockfs1:/")); + setupMockFileSystem(conf, new URI("mockfs2:/")); FileSystem viewFs = FileSystem.get(FsConstants.VIEWFS_URI, conf); + FileSystem mockFs1 = + ((MockFileSystem) getChildFileSystem((ViewFileSystem) viewFs, + new URI("mockfs1:/"))).getRawFileSystem(); + FileSystem mockFs2 = + ((MockFileSystem) getChildFileSystem((ViewFileSystem) viewFs, + new URI("mockfs2:/"))).getRawFileSystem(); Path viewFsPath1 = new Path("/mounts/mockfs1/a/b/c"); Path mockFsPath1 = new Path("/a/b/c"); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java index 735dfcf3cf627..239f47d1da6f3 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java @@ -18,6 +18,7 @@ package org.apache.hadoop.fs.viewfs; import static org.junit.Assert.*; +import static org.apache.hadoop.fs.viewfs.TestChRootedFileSystem.getChildFileSystem; import java.io.IOException; import java.net.URI; @@ -54,12 +55,16 @@ public class TestViewFileSystemDelegationTokenSupport { @BeforeClass public static void setup() throws Exception { conf = ViewFileSystemTestSetup.createConfig(); - fs1 = setupFileSystem(new URI("fs1:///"), FakeFileSystem.class); - fs2 = setupFileSystem(new URI("fs2:///"), FakeFileSystem.class); + setupFileSystem(new URI("fs1:///"), FakeFileSystem.class); + setupFileSystem(new URI("fs2:///"), FakeFileSystem.class); viewFs = FileSystem.get(FsConstants.VIEWFS_URI, conf); + fs1 = (FakeFileSystem) getChildFileSystem((ViewFileSystem) viewFs, + new URI("fs1:///")); + fs2 = (FakeFileSystem) getChildFileSystem((ViewFileSystem) viewFs, + new URI("fs2:///")); } - static FakeFileSystem setupFileSystem(URI uri, Class clazz) + static void setupFileSystem(URI uri, Class clazz) throws Exception { String scheme = uri.getScheme(); conf.set("fs."+scheme+".impl", clazz.getName()); @@ -67,7 +72,6 @@ static FakeFileSystem setupFileSystem(URI uri, Class clazz // mount each fs twice, will later ensure 1 token/fs ConfigUtil.addLink(conf, "/mounts/"+scheme+"-one", fs.getUri()); ConfigUtil.addLink(conf, "/mounts/"+scheme+"-two", fs.getUri()); - return fs; } /** diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsTrash.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsTrash.java index 62ef9d146384e..94c3262eaae92 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsTrash.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsTrash.java @@ -70,7 +70,7 @@ public void tearDown() throws Exception { } @Test - public void testTrash() throws IOException { + public void testTrash() throws Exception { TestTrash.trashShell(conf, fileSystemTestHelper.getTestRootPath(fsView), fsTarget, new Path(fsTarget.getHomeDirectory(), ".Trash/Current")); } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java index b9453df5699f2..4902d733e954b 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java @@ -27,6 +27,7 @@ import java.util.ArrayList; import java.util.Map; import java.util.Map.Entry; +import java.util.Random; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.BlockLocation; @@ -36,9 +37,11 @@ import org.apache.hadoop.fs.FileSystemTestHelper; import org.apache.hadoop.fs.FsConstants; import org.apache.hadoop.fs.FsStatus; +import org.apache.hadoop.fs.LocalFileSystem; import org.apache.hadoop.fs.LocatedFileStatus; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.RemoteIterator; +import org.apache.hadoop.fs.TestFileUtil; import org.apache.hadoop.fs.Trash; import org.apache.hadoop.fs.UnsupportedFileSystemException; import org.apache.hadoop.fs.contract.ContractTestUtils; @@ -64,10 +67,11 @@ import org.junit.Before; import org.junit.Test; +import static org.apache.hadoop.test.GenericTestUtils.assertExceptionContains; import static org.hamcrest.CoreMatchers.containsString; +import static org.hamcrest.CoreMatchers.equalTo; import static org.junit.Assert.*; - /** *

    * A collection of tests for the {@link ViewFileSystem}. @@ -473,10 +477,10 @@ void compareBLs(BlockLocation[] viewBL, BlockLocation[] targetBL) { Assert.assertEquals(targetBL.length, viewBL.length); int i = 0; for (BlockLocation vbl : viewBL) { - Assert.assertEquals(vbl.toString(), targetBL[i].toString()); - Assert.assertEquals(targetBL[i].getOffset(), vbl.getOffset()); - Assert.assertEquals(targetBL[i].getLength(), vbl.getLength()); - i++; + assertThat(vbl.toString(), equalTo(targetBL[i].toString())); + assertThat(vbl.getOffset(), equalTo(targetBL[i].getOffset())); + assertThat(vbl.getLength(), equalTo(targetBL[i].getLength())); + i++; } } @@ -1272,4 +1276,96 @@ public void testLinkTarget() throws Exception { containsString("File does not exist:")); } } + + @Test + public void testViewFileSystemInnerCache() throws Exception { + ViewFileSystem.InnerCache cache = new ViewFileSystem.InnerCache(); + FileSystem fs = cache.get(fsTarget.getUri(), conf); + + // InnerCache caches filesystem. + assertSame(fs, cache.get(fsTarget.getUri(), conf)); + + // InnerCache and FileSystem.CACHE are independent. + assertNotSame(fs, FileSystem.get(fsTarget.getUri(), conf)); + + // close InnerCache. + cache.closeAll(); + try { + fs.exists(new Path("/")); + if (!(fs instanceof LocalFileSystem)) { + // Ignore LocalFileSystem because it can still be used after close. + fail("Expect Filesystem closed exception"); + } + } catch (IOException e) { + assertExceptionContains("Filesystem closed", e); + } + } + + @Test + public void testCloseChildrenFileSystem() throws Exception { + final String clusterName = "cluster" + new Random().nextInt(); + Configuration config = new Configuration(conf); + ConfigUtil.addLink(config, clusterName, "/user", + new Path(targetTestRoot, "user").toUri()); + config.setBoolean("fs.viewfs.impl.disable.cache", false); + URI uri = new URI("viewfs://" + clusterName + "/"); + + ViewFileSystem viewFs = (ViewFileSystem) FileSystem.get(uri, config); + assertTrue("viewfs should have at least one child fs.", + viewFs.getChildFileSystems().length > 0); + // viewFs is cached in FileSystem.CACHE + assertSame(viewFs, FileSystem.get(uri, config)); + + // child fs is not cached in FileSystem.CACHE + FileSystem child = viewFs.getChildFileSystems()[0]; + assertNotSame(child, FileSystem.get(child.getUri(), config)); + + viewFs.close(); + for (FileSystem childfs : viewFs.getChildFileSystems()) { + try { + childfs.exists(new Path("/")); + if (!(childfs instanceof LocalFileSystem)) { + // Ignore LocalFileSystem because it can still be used after close. + fail("Expect Filesystem closed exception"); + } + } catch (IOException e) { + assertExceptionContains("Filesystem closed", e); + } + } + } + + @Test + public void testChildrenFileSystemLeak() throws Exception { + final String clusterName = "cluster" + new Random().nextInt(); + Configuration config = new Configuration(conf); + ConfigUtil.addLink(config, clusterName, "/user", + new Path(targetTestRoot, "user").toUri()); + + final int cacheSize = TestFileUtil.getCacheSize(); + ViewFileSystem viewFs = (ViewFileSystem) FileSystem + .get(new URI("viewfs://" + clusterName + "/"), config); + assertEquals(cacheSize + 1, TestFileUtil.getCacheSize()); + viewFs.close(); + assertEquals(cacheSize, TestFileUtil.getCacheSize()); + } + + @Test + public void testDeleteOnExit() throws Exception { + final String clusterName = "cluster" + new Random().nextInt(); + Configuration config = new Configuration(conf); + ConfigUtil.addLink(config, clusterName, "/user", + new Path(targetTestRoot, "user").toUri()); + + Path testDir = new Path("/user/testDeleteOnExit"); + Path realTestPath = new Path(targetTestRoot, "user/testDeleteOnExit"); + ViewFileSystem viewFs = (ViewFileSystem) FileSystem + .get(new URI("viewfs://" + clusterName + "/"), config); + viewFs.mkdirs(testDir); + assertTrue(viewFs.exists(testDir)); + assertTrue(fsTarget.exists(realTestPath)); + + viewFs.deleteOnExit(testDir); + viewFs.close(); + assertFalse(fsTarget.exists(realTestPath)); + } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java index e080cd6715e0e..d96cdb172b702 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java @@ -17,6 +17,7 @@ */ package org.apache.hadoop.fs.viewfs; +import static org.assertj.core.api.Assertions.assertThat; import static org.apache.hadoop.fs.FileContextTestHelper.checkFileLinkStatus; import static org.apache.hadoop.fs.FileContextTestHelper.checkFileStatus; import static org.apache.hadoop.fs.FileContextTestHelper.exists; @@ -459,9 +460,9 @@ void compareBLs(BlockLocation[] viewBL, BlockLocation[] targetBL) { Assert.assertEquals(targetBL.length, viewBL.length); int i = 0; for (BlockLocation vbl : viewBL) { - Assert.assertEquals(vbl.toString(), targetBL[i].toString()); - Assert.assertEquals(targetBL[i].getOffset(), vbl.getOffset()); - Assert.assertEquals(targetBL[i].getLength(), vbl.getLength()); + assertThat(vbl.toString()).isEqualTo(targetBL[i].toString()); + assertThat(vbl.getOffset()).isEqualTo(targetBL[i].getOffset()); + assertThat(vbl.getLength()).isEqualTo(targetBL[i].getLength()); i++; } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java index 7396694ce95f8..be6181157c0dd 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java @@ -65,6 +65,15 @@ public abstract class ClientBaseWithFixes extends ZKTestCase { public static int CONNECTION_TIMEOUT = 30000; static final File BASETEST = GenericTestUtils.getTestDir(); + static { + // The 4-letter-words commands are simple diagnostics telnet commands in + // ZooKeeper. Since ZooKeeper 3.5, these are disabled by default due to + // security concerns: https://issues.apache.org/jira/browse/ZOOKEEPER-2693 + // We are enabling them for the tests here, as some tests in hadoop or in + // other projects might still use them + System.setProperty("zookeeper.4lw.commands.whitelist", "*"); + } + protected final String hostPort = initHostPort(); protected int maxCnxns = 0; protected ServerCnxnFactory serverFactory = null; diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java index 51112bedefa87..064527c3fed6d 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java @@ -39,6 +39,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.apache.hadoop.fs.CommonConfigurationKeys.HA_HM_RPC_CONNECT_MAX_RETRIES_DEFAULT; import static org.apache.hadoop.fs.CommonConfigurationKeys.HA_HM_RPC_TIMEOUT_DEFAULT; /** @@ -63,6 +64,7 @@ class DummyHAService extends HAServiceTarget { public int fenceCount = 0; public int activeTransitionCount = 0; boolean testWithProtoBufRPC = false; + int rpcTimeout; static ArrayList instances = Lists.newArrayList(); int index; @@ -82,7 +84,8 @@ class DummyHAService extends HAServiceTarget { } Configuration conf = new Configuration(); this.proxy = makeMock(conf, HA_HM_RPC_TIMEOUT_DEFAULT); - this.healthMonitorProxy = makeHealthMonitorMock(conf, HA_HM_RPC_TIMEOUT_DEFAULT); + this.healthMonitorProxy = makeHealthMonitorMock(conf, + HA_HM_RPC_TIMEOUT_DEFAULT, HA_HM_RPC_CONNECT_MAX_RETRIES_DEFAULT); try { conf.set(DUMMY_FENCE_KEY, DummyFencer.class.getName()); this.fencer = Mockito.spy( @@ -149,13 +152,13 @@ private HAServiceProtocol makeMock(Configuration conf, int timeoutMs) { } private HAServiceProtocol makeHealthMonitorMock(Configuration conf, - int timeoutMs) { + int timeoutMs, int retries) { HAServiceProtocol service; if (!testWithProtoBufRPC) { service = new MockHAProtocolImpl(); } else { try { - service = super.getHealthMonitorProxy(conf, timeoutMs); + service = super.getHealthMonitorProxy(conf, timeoutMs, retries); } catch (IOException e) { return null; } @@ -189,9 +192,9 @@ public HAServiceProtocol getProxy(Configuration conf, int timeout) @Override public HAServiceProtocol getHealthMonitorProxy(Configuration conf, - int timeout) throws IOException { + int timeout, int retries) throws IOException { if (testWithProtoBufRPC) { - proxy = makeHealthMonitorMock(conf, timeout); + proxy = makeHealthMonitorMock(conf, timeout, retries); } return proxy; } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestGlobalFilter.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestGlobalFilter.java index 70361752633c3..ade383883f10e 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestGlobalFilter.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestGlobalFilter.java @@ -142,6 +142,8 @@ public void testServletFilter() throws Exception { for(int i = 0; i < urls.length; i++) { assertTrue(RECORDS.remove(urls[i])); } - assertTrue(RECORDS.isEmpty()); + assertTrue(RECORDS.size()==1); + // Accesing "/" will redirect to /index.html + assertTrue(RECORDS.contains("/index.html")); } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpRequestLog.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpRequestLog.java index 212807f78ef65..d0123e32039c9 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpRequestLog.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpRequestLog.java @@ -18,7 +18,7 @@ package org.apache.hadoop.http; import org.apache.log4j.Logger; -import org.eclipse.jetty.server.NCSARequestLog; +import org.eclipse.jetty.server.CustomRequestLog; import org.eclipse.jetty.server.RequestLog; import org.junit.Test; @@ -42,6 +42,7 @@ public void testAppenderDefined() { RequestLog requestLog = HttpRequestLog.getRequestLog("test"); Logger.getLogger("http.requests.test").removeAppender(requestLogAppender); assertNotNull("RequestLog should not be null", requestLog); - assertEquals("Class mismatch", NCSARequestLog.class, requestLog.getClass()); + assertEquals("Class mismatch", + CustomRequestLog.class, requestLog.getClass()); } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServerWithSpnego.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServerWithSpnego.java new file mode 100644 index 0000000000000..ea7c8cd4e6864 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServerWithSpnego.java @@ -0,0 +1,238 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

    + * http://www.apache.org/licenses/LICENSE-2.0 + *

    + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.http; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.CommonConfigurationKeys; +import org.apache.hadoop.minikdc.MiniKdc; +import org.apache.hadoop.net.NetUtils; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.authentication.KerberosTestUtils; +import org.apache.hadoop.security.authentication.client.AuthenticatedURL; +import org.apache.hadoop.security.authentication.server.AuthenticationFilter; +import org.apache.hadoop.security.authentication.server.AuthenticationToken; +import org.apache.hadoop.security.authentication.server.ProxyUserAuthenticationFilterInitializer; +import org.apache.hadoop.security.authentication.util.Signer; +import org.apache.hadoop.security.authentication.util.SignerSecretProvider; +import org.apache.hadoop.security.authentication.util.StringSignerSecretProviderCreator; +import org.apache.hadoop.security.authorize.AccessControlList; +import org.apache.hadoop.security.authorize.ProxyUsers; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.Assert; + +import java.io.File; +import java.io.FileWriter; +import java.io.Writer; +import java.net.HttpURLConnection; +import java.net.URI; +import java.net.URL; +import java.util.Properties; +import static org.junit.Assert.assertTrue; + +/** + * This class is tested for http server with SPNEGO authentication. + */ +public class TestHttpServerWithSpnego { + + static final Log LOG = LogFactory.getLog(TestHttpServerWithSpnego.class); + + private static final String SECRET_STR = "secret"; + private static final String HTTP_USER = "HTTP"; + private static final String PREFIX = "hadoop.http.authentication."; + private static final long TIMEOUT = 20000; + + private static File httpSpnegoKeytabFile = new File( + KerberosTestUtils.getKeytabFile()); + private static String httpSpnegoPrincipal = + KerberosTestUtils.getServerPrincipal(); + private static String realm = KerberosTestUtils.getRealm(); + + private static File testRootDir = new File("target", + TestHttpServerWithSpnego.class.getName() + "-root"); + private static MiniKdc testMiniKDC; + private static File secretFile = new File(testRootDir, SECRET_STR); + + @BeforeClass + public static void setUp() throws Exception { + try { + testMiniKDC = new MiniKdc(MiniKdc.createConf(), testRootDir); + testMiniKDC.start(); + testMiniKDC.createPrincipal( + httpSpnegoKeytabFile, HTTP_USER + "/localhost"); + } catch (Exception e) { + assertTrue("Couldn't setup MiniKDC", false); + } + Writer w = new FileWriter(secretFile); + w.write("secret"); + w.close(); + } + + @AfterClass + public static void tearDown() { + if (testMiniKDC != null) { + testMiniKDC.stop(); + } + } + + /** + * groupA + * - userA + * groupB + * - userA, userB + * groupC + * - userC + * SPNEGO filter has been enabled. + * userA has the privilege to impersonate users in groupB. + * userA has admin access to all default servlets, but userB + * and userC don't have. So "/logs" can only be accessed by userA. + * @throws Exception + */ + @Test + public void testAuthenticationWithProxyUser() throws Exception { + Configuration spengoConf = getSpengoConf(new Configuration()); + + //setup logs dir + System.setProperty("hadoop.log.dir", testRootDir.getAbsolutePath()); + + // Setup user group + UserGroupInformation.createUserForTesting("userA", + new String[]{"groupA", "groupB"}); + UserGroupInformation.createUserForTesting("userB", + new String[]{"groupB"}); + UserGroupInformation.createUserForTesting("userC", + new String[]{"groupC"}); + + // Make userA impersonate users in groupB + spengoConf.set("hadoop.proxyuser.userA.hosts", "*"); + spengoConf.set("hadoop.proxyuser.userA.groups", "groupB"); + ProxyUsers.refreshSuperUserGroupsConfiguration(spengoConf); + + HttpServer2 httpServer = null; + try { + // Create http server to test. + httpServer = getCommonBuilder() + .setConf(spengoConf) + .setACL(new AccessControlList("userA groupA")) + .build(); + httpServer.start(); + + // Get signer to encrypt token + Signer signer = getSignerToEncrypt(); + + // setup auth token for userA + AuthenticatedURL.Token token = getEncryptedAuthToken(signer, "userA"); + + String serverURL = "http://" + + NetUtils.getHostPortString(httpServer.getConnectorAddress(0)) + "/"; + + // The default authenticator is kerberos. + AuthenticatedURL authUrl = new AuthenticatedURL(); + + // userA impersonates userB, it's allowed. + for (String servlet : + new String[]{"stacks", "jmx", "conf"}) { + HttpURLConnection conn = authUrl + .openConnection(new URL(serverURL + servlet + "?doAs=userB"), + token); + Assert.assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode()); + } + + // userA cannot impersonate userC, it fails. + for (String servlet : + new String[]{"stacks", "jmx", "conf"}){ + HttpURLConnection conn = authUrl + .openConnection(new URL(serverURL + servlet + "?doAs=userC"), + token); + Assert.assertEquals(HttpURLConnection.HTTP_FORBIDDEN, + conn.getResponseCode()); + } + + + // "/logs" and "/logLevel" require admin authorization, + // only userA has the access. + for (String servlet : + new String[]{"logLevel", "logs"}) { + HttpURLConnection conn = authUrl + .openConnection(new URL(serverURL + servlet), token); + Assert.assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode()); + } + + // Setup token for userB + token = getEncryptedAuthToken(signer, "userB"); + + // userB cannot access these servlets. + for (String servlet : + new String[]{"logLevel", "logs"}) { + HttpURLConnection conn = authUrl + .openConnection(new URL(serverURL + servlet), token); + Assert.assertEquals(HttpURLConnection.HTTP_FORBIDDEN, + conn.getResponseCode()); + } + + } finally { + if (httpServer != null) { + httpServer.stop(); + } + } + } + + private AuthenticatedURL.Token getEncryptedAuthToken(Signer signer, + String user) throws Exception { + AuthenticationToken token = + new AuthenticationToken(user, user, "kerberos"); + token.setExpires(System.currentTimeMillis() + TIMEOUT); + return new AuthenticatedURL.Token(signer.sign(token.toString())); + } + + private Signer getSignerToEncrypt() throws Exception { + SignerSecretProvider secretProvider = + StringSignerSecretProviderCreator.newStringSignerSecretProvider(); + Properties secretProviderProps = new Properties(); + secretProviderProps.setProperty( + AuthenticationFilter.SIGNATURE_SECRET, SECRET_STR); + secretProvider.init(secretProviderProps, null, TIMEOUT); + return new Signer(secretProvider); + } + + private Configuration getSpengoConf(Configuration conf) { + conf = new Configuration(); + conf.set(HttpServer2.FILTER_INITIALIZER_PROPERTY, + ProxyUserAuthenticationFilterInitializer.class.getName()); + conf.set(PREFIX + "type", "kerberos"); + conf.setBoolean(PREFIX + "simple.anonymous.allowed", false); + conf.set(PREFIX + "signature.secret.file", + secretFile.getAbsolutePath()); + conf.set(PREFIX + "kerberos.keytab", + httpSpnegoKeytabFile.getAbsolutePath()); + conf.set(PREFIX + "kerberos.principal", httpSpnegoPrincipal); + conf.set(PREFIX + "cookie.domain", realm); + conf.setBoolean(CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, + true); + return conf; + } + + private HttpServer2.Builder getCommonBuilder() throws Exception { + return new HttpServer2.Builder().setName("test") + .addEndpoint(new URI("http://localhost:0")) + .setFindPort(true); + } +} \ No newline at end of file diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestPathFilter.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestPathFilter.java index 4c35b391c39d6..d54503a0086e9 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestPathFilter.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestPathFilter.java @@ -35,7 +35,6 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.net.NetUtils; -import org.junit.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -102,7 +101,6 @@ static void access(String urlstring) throws IOException { } } - @Test public void testPathSpecFilters() throws Exception { Configuration conf = new Configuration(); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java index 3f6ee7b79f40b..5f7a264190953 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java @@ -98,6 +98,8 @@ public class TestSSLHttpServer extends HttpServerFunctionalTest { + "TLS_DHE_RSA_WITH_AES_128_CBC_SHA,\t\n " + "TLS_DHE_DSS_WITH_AES_128_CBC_SHA"; + private static final String INCLUDED_PROTOCOLS = "SSLv2Hello,TLSv1.1"; + @BeforeClass public static void setup() throws Exception { turnOnSSLDebugLogging(); @@ -128,6 +130,8 @@ public static void setup() throws Exception { private static void setupServer(Configuration conf, Configuration sslConf) throws IOException, URISyntaxException { + conf.set(SSLFactory.SSL_ENABLED_PROTOCOLS_KEY, INCLUDED_PROTOCOLS); + sslConf.set(SSLFactory.SSL_ENABLED_PROTOCOLS_KEY, INCLUDED_PROTOCOLS); server = new HttpServer2.Builder().setName("test") .addEndpoint(new URI("https://localhost")).setConf(conf) .keyPassword( @@ -214,6 +218,22 @@ private HttpsURLConnection getConnectionWithSSLSocketFactory(URL url, return conn; } + private HttpsURLConnection + getConnectionWithPreferredProtocolSSLSocketFactory(URL url, + String protocols) throws IOException, GeneralSecurityException { + HttpsURLConnection conn = (HttpsURLConnection) url.openConnection(); + SSLSocketFactory sslSocketFactory = clientSslFactory + .createSSLSocketFactory(); + LOG.info("Creating " + + PreferredProtocolSSLSocketFactory.class.getCanonicalName() + + " with protocols: " + protocols); + PreferredProtocolSSLSocketFactory cipherSSLSocketFactory + = new PreferredProtocolSSLSocketFactory(sslSocketFactory, + StringUtils.getTrimmedStrings(protocols)); + conn.setSSLSocketFactory(cipherSSLSocketFactory); + return conn; + } + @Test public void testEcho() throws Exception { assertEquals("a:b\nc:d\n", @@ -269,6 +289,18 @@ public void testExcludedCiphers() throws Exception { } } + @Test + public void testIncludedProtocols() throws Exception { + URL url = new URL(baseUrl, SERVLET_PATH_ECHO + "?a=b&c=d"); + HttpsURLConnection conn = + getConnectionWithPreferredProtocolSSLSocketFactory(url, + INCLUDED_PROTOCOLS); + assertFalse("included protocol list is empty", + INCLUDED_PROTOCOLS.isEmpty()); + + readFromConnection(conn); + } + /** Test that verified that additionally included cipher * TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA is only available cipher for working * TLS connection from client to server disabled for all other common ciphers. @@ -370,4 +402,78 @@ private void setEnabledCipherSuites(SSLSocket sslSocket) { } } } + + private class PreferredProtocolSSLSocketFactory extends SSLSocketFactory { + private final SSLSocketFactory delegateSocketFactory; + private final String[] enabledProtocols; + + PreferredProtocolSSLSocketFactory(SSLSocketFactory sslSocketFactory, + String[] enabledProtocols) { + delegateSocketFactory = sslSocketFactory; + if (null != enabledProtocols && enabledProtocols.length > 0) { + this.enabledProtocols = enabledProtocols; + } else { + this.enabledProtocols = null; + } + } + + @Override + public String[] getDefaultCipherSuites() { + return delegateSocketFactory.getDefaultCipherSuites(); + } + + @Override + public String[] getSupportedCipherSuites() { + return delegateSocketFactory.getSupportedCipherSuites(); + } + + @Override + public Socket createSocket(Socket socket, String string, int i, boolean bln) + throws IOException { + SSLSocket sslSocket = (SSLSocket) delegateSocketFactory.createSocket( + socket, string, i, bln); + setEnabledProtocols(sslSocket); + return sslSocket; + } + + @Override + public Socket createSocket(String string, int i) throws IOException { + SSLSocket sslSocket = (SSLSocket) delegateSocketFactory.createSocket( + string, i); + setEnabledProtocols(sslSocket); + return sslSocket; + } + + @Override + public Socket createSocket(String string, int i, InetAddress ia, int i1) + throws IOException { + SSLSocket sslSocket = (SSLSocket) delegateSocketFactory.createSocket( + string, i, ia, i1); + setEnabledProtocols(sslSocket); + return sslSocket; + } + + @Override + public Socket createSocket(InetAddress ia, int i) throws IOException { + SSLSocket sslSocket = (SSLSocket) delegateSocketFactory.createSocket(ia, + i); + setEnabledProtocols(sslSocket); + return sslSocket; + } + + @Override + public Socket createSocket(InetAddress ia, int i, InetAddress ia1, int i1) + throws IOException { + SSLSocket sslSocket = (SSLSocket) delegateSocketFactory.createSocket(ia, + i, ia1, i1); + setEnabledProtocols(sslSocket); + return sslSocket; + } + + private void setEnabledProtocols(SSLSocket sslSocket) { + if (null != enabledProtocols) { + sslSocket.setEnabledProtocols(enabledProtocols); + } + } + } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestServletFilter.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestServletFilter.java index eafd0ae9ccae3..a8ecbd4fe28ef 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestServletFilter.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestServletFilter.java @@ -100,7 +100,6 @@ static void access(String urlstring) throws IOException { } } - @Test public void testServletFilter() throws Exception { Configuration conf = new Configuration(); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestArrayFile.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestArrayFile.java index 722e9de595823..2f69093d2654e 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestArrayFile.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestArrayFile.java @@ -30,9 +30,9 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.assertj.core.api.Assertions.assertThat; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertNotNull; -import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; import static org.junit.Assert.assertFalse; import static org.junit.Assert.fail; @@ -146,15 +146,16 @@ public void testArrayFileIteration() { for (int i = 0; i < SIZE; i++) { nextWritable = (LongWritable)reader.next(nextWritable); - assertEquals(nextWritable.get(), i); + assertThat(nextWritable.get()).isEqualTo(i); } assertTrue("testArrayFileIteration seek error !!!", reader.seek(new LongWritable(6))); nextWritable = (LongWritable) reader.next(nextWritable); - assertTrue("testArrayFileIteration error !!!", reader.key() == 7); - assertTrue("testArrayFileIteration error !!!", - nextWritable.equals(new LongWritable(7))); + assertThat(reader.key()).withFailMessage( + "testArrayFileIteration error !!!").isEqualTo(7); + assertThat(nextWritable).withFailMessage( + "testArrayFileIteration error !!!").isEqualTo(new LongWritable(7)); assertFalse("testArrayFileIteration error !!!", reader.seek(new LongWritable(SIZE + 5))); reader.close(); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSequenceFileSync.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSequenceFileSync.java index 363177b46464b..5fbb083189e8a 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSequenceFileSync.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSequenceFileSync.java @@ -18,8 +18,6 @@ package org.apache.hadoop.io; -import static org.junit.Assert.assertEquals; - import java.io.IOException; import java.util.Random; @@ -31,6 +29,8 @@ import org.apache.hadoop.test.GenericTestUtils; import org.junit.Test; +import static org.assertj.core.api.Assertions.assertThat; + /** Tests sync based seek reads/write intervals inside SequenceFiles. */ public class TestSequenceFileSync { private static final int NUMRECORDS = 2000; @@ -46,12 +46,10 @@ private static void forOffset(SequenceFile.Reader reader, val.clear(); reader.sync(off); reader.next(key, val); - assertEquals(key.get(), expectedRecord); + assertThat(key.get()).isEqualTo(expectedRecord); final String test = String.format(REC_FMT, expectedRecord, expectedRecord); - assertEquals( - "Invalid value in iter " + iter + ": " + val, - 0, - val.find(test, 0)); + assertThat(val.find(test, 0)).withFailMessage( + "Invalid value in iter " + iter + ": " + val).isZero(); } @Test @@ -124,7 +122,7 @@ public void testLowSyncpoint() throws IOException { SequenceFile.Writer.syncInterval(20*100) ); // Ensure the custom sync interval value is set - assertEquals(writer.syncInterval, 20*100); + assertThat(writer.syncInterval).isEqualTo(20*100); try { writeSequenceFile(writer, NUMRECORDS); for (int i = 0; i < 5; i++) { diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java index 9771fd1a96623..59856a4de11f9 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java @@ -26,6 +26,8 @@ import com.google.common.base.Charsets; import com.google.common.primitives.Bytes; import org.junit.Test; + +import static org.assertj.core.api.Assertions.assertThat; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; @@ -232,10 +234,10 @@ public void testCompare() throws Exception { @Test public void testFind() throws Exception { Text text = new Text("abcd\u20acbdcd\u20ac"); - assertTrue(text.find("abd")==-1); - assertTrue(text.find("ac") ==-1); - assertTrue(text.find("\u20ac") == 4); - assertTrue(text.find("\u20ac", 5)==11); + assertThat(text.find("abd")).isEqualTo(-1); + assertThat(text.find("ac")).isEqualTo(-1); + assertThat(text.find("\u20ac")).isEqualTo(4); + assertThat(text.find("\u20ac", 5)).isEqualTo(11); } @Test diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/file/tfile/TestTFileUnsortedByteArrays.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/file/tfile/TestTFileUnsortedByteArrays.java index f243b2a94065a..f849d538d6d61 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/file/tfile/TestTFileUnsortedByteArrays.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/file/tfile/TestTFileUnsortedByteArrays.java @@ -20,7 +20,6 @@ import java.io.IOException; import org.junit.After; -import org.junit.Assert; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataOutputStream; @@ -33,6 +32,9 @@ import org.junit.Before; import org.junit.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.Assert.fail; + public class TestTFileUnsortedByteArrays { private static String ROOT = GenericTestUtils.getTestDir().getAbsolutePath(); @@ -84,152 +86,119 @@ public void tearDown() throws IOException { // we still can scan records in an unsorted TFile @Test public void testFailureScannerWithKeys() throws IOException { - Reader reader = - new Reader(fs.open(path), fs.getFileStatus(path).getLen(), conf); - Assert.assertFalse(reader.isSorted()); - Assert.assertEquals((int) reader.getEntryCount(), 4); - - try { - Scanner scanner = - reader.createScannerByKey("aaa".getBytes(), "zzz".getBytes()); - Assert - .fail("Failed to catch creating scanner with keys on unsorted file."); - } - catch (RuntimeException e) { - } - finally { - reader.close(); + try (Reader reader = + new Reader(fs.open(path), fs.getFileStatus(path).getLen(), conf)) { + assertThat(reader.isSorted()).isFalse(); + assertThat(reader.getEntryCount()).isEqualTo(4); + try { + reader.createScannerByKey("aaa".getBytes(), "zzz".getBytes()); + fail("Failed to catch creating scanner with keys on unsorted file."); + } catch (RuntimeException expected) { + } } } // we still can scan records in an unsorted TFile @Test public void testScan() throws IOException { - Reader reader = - new Reader(fs.open(path), fs.getFileStatus(path).getLen(), conf); - Assert.assertFalse(reader.isSorted()); - Assert.assertEquals((int) reader.getEntryCount(), 4); - - Scanner scanner = reader.createScanner(); - - try { - - // read key and value - byte[] kbuf = new byte[BUF_SIZE]; - int klen = scanner.entry().getKeyLength(); - scanner.entry().getKey(kbuf); - Assert.assertEquals(new String(kbuf, 0, klen), "keyZ"); - - byte[] vbuf = new byte[BUF_SIZE]; - int vlen = scanner.entry().getValueLength(); - scanner.entry().getValue(vbuf); - Assert.assertEquals(new String(vbuf, 0, vlen), "valueZ"); - - scanner.advance(); - - // now try get value first - vbuf = new byte[BUF_SIZE]; - vlen = scanner.entry().getValueLength(); - scanner.entry().getValue(vbuf); - Assert.assertEquals(new String(vbuf, 0, vlen), "valueM"); - - kbuf = new byte[BUF_SIZE]; - klen = scanner.entry().getKeyLength(); - scanner.entry().getKey(kbuf); - Assert.assertEquals(new String(kbuf, 0, klen), "keyM"); - } - finally { - scanner.close(); - reader.close(); + try (Reader reader = + new Reader(fs.open(path), fs.getFileStatus(path).getLen(), conf)) { + assertThat(reader.isSorted()).isFalse(); + assertThat(reader.getEntryCount()).isEqualTo(4); + try (Scanner scanner = reader.createScanner()) { + // read key and value + byte[] kbuf = new byte[BUF_SIZE]; + int klen = scanner.entry().getKeyLength(); + scanner.entry().getKey(kbuf); + assertThat(new String(kbuf, 0, klen)).isEqualTo("keyZ"); + + byte[] vbuf = new byte[BUF_SIZE]; + int vlen = scanner.entry().getValueLength(); + scanner.entry().getValue(vbuf); + assertThat(new String(vbuf, 0, vlen)).isEqualTo("valueZ"); + + scanner.advance(); + + // now try get value first + vbuf = new byte[BUF_SIZE]; + vlen = scanner.entry().getValueLength(); + scanner.entry().getValue(vbuf); + assertThat(new String(vbuf, 0, vlen)).isEqualTo("valueM"); + + kbuf = new byte[BUF_SIZE]; + klen = scanner.entry().getKeyLength(); + scanner.entry().getKey(kbuf); + assertThat(new String(kbuf, 0, klen)).isEqualTo("keyM"); + } } } // we still can scan records in an unsorted TFile @Test public void testScanRange() throws IOException { - Reader reader = - new Reader(fs.open(path), fs.getFileStatus(path).getLen(), conf); - Assert.assertFalse(reader.isSorted()); - Assert.assertEquals((int) reader.getEntryCount(), 4); - - Scanner scanner = reader.createScanner(); - - try { - - // read key and value - byte[] kbuf = new byte[BUF_SIZE]; - int klen = scanner.entry().getKeyLength(); - scanner.entry().getKey(kbuf); - Assert.assertEquals(new String(kbuf, 0, klen), "keyZ"); - - byte[] vbuf = new byte[BUF_SIZE]; - int vlen = scanner.entry().getValueLength(); - scanner.entry().getValue(vbuf); - Assert.assertEquals(new String(vbuf, 0, vlen), "valueZ"); - - scanner.advance(); - - // now try get value first - vbuf = new byte[BUF_SIZE]; - vlen = scanner.entry().getValueLength(); - scanner.entry().getValue(vbuf); - Assert.assertEquals(new String(vbuf, 0, vlen), "valueM"); - - kbuf = new byte[BUF_SIZE]; - klen = scanner.entry().getKeyLength(); - scanner.entry().getKey(kbuf); - Assert.assertEquals(new String(kbuf, 0, klen), "keyM"); - } - finally { - scanner.close(); - reader.close(); + try (Reader reader = + new Reader(fs.open(path), fs.getFileStatus(path).getLen(), conf)) { + assertThat(reader.isSorted()).isFalse(); + assertThat(reader.getEntryCount()).isEqualTo(4); + + try (Scanner scanner = reader.createScanner()) { + + // read key and value + byte[] kbuf = new byte[BUF_SIZE]; + int klen = scanner.entry().getKeyLength(); + scanner.entry().getKey(kbuf); + assertThat(new String(kbuf, 0, klen)).isEqualTo("keyZ"); + + byte[] vbuf = new byte[BUF_SIZE]; + int vlen = scanner.entry().getValueLength(); + scanner.entry().getValue(vbuf); + assertThat(new String(vbuf, 0, vlen)).isEqualTo("valueZ"); + + scanner.advance(); + + // now try get value first + vbuf = new byte[BUF_SIZE]; + vlen = scanner.entry().getValueLength(); + scanner.entry().getValue(vbuf); + assertThat(new String(vbuf, 0, vlen)).isEqualTo("valueM"); + + kbuf = new byte[BUF_SIZE]; + klen = scanner.entry().getKeyLength(); + scanner.entry().getKey(kbuf); + assertThat(new String(kbuf, 0, klen)).isEqualTo("keyM"); + } } } @Test public void testFailureSeek() throws IOException { - Reader reader = - new Reader(fs.open(path), fs.getFileStatus(path).getLen(), conf); - Scanner scanner = reader.createScanner(); - - try { + try (Reader reader = new Reader(fs.open(path), + fs.getFileStatus(path).getLen(), conf); + Scanner scanner = reader.createScanner()) { // can't find ceil try { scanner.lowerBound("keyN".getBytes()); - Assert.fail("Cannot search in a unsorted TFile!"); - } - catch (Exception e) { - // noop, expecting excetions + fail("Cannot search in a unsorted TFile!"); } - finally { + catch (Exception expected) { } // can't find higher try { scanner.upperBound("keyA".getBytes()); - Assert.fail("Cannot search higher in a unsorted TFile!"); + fail("Cannot search higher in a unsorted TFile!"); } - catch (Exception e) { - // noop, expecting excetions - } - finally { + catch (Exception expected) { } // can't seek try { scanner.seekTo("keyM".getBytes()); - Assert.fail("Cannot search a unsorted TFile!"); - } - catch (Exception e) { - // noop, expecting excetions + fail("Cannot search a unsorted TFile!"); } - finally { + catch (Exception expected) { } } - finally { - scanner.close(); - reader.close(); - } } private void closeOutput() throws IOException { diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/file/tfile/TestVLong.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/file/tfile/TestVLong.java index 69e6eb8741270..b7550f9d584d2 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/file/tfile/TestVLong.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/file/tfile/TestVLong.java @@ -33,6 +33,8 @@ import org.junit.Before; import org.junit.Test; +import static org.assertj.core.api.Assertions.assertThat; + public class TestVLong { private static String ROOT = GenericTestUtils.getTestDir().getAbsolutePath(); private Configuration conf; @@ -70,8 +72,7 @@ public void testVLongByte() throws IOException { FSDataInputStream in = fs.open(path); for (int i = Byte.MIN_VALUE; i <= Byte.MAX_VALUE; ++i) { - long n = Utils.readVLong(in); - Assert.assertEquals(n, i); + assertThat(Utils.readVLong(in)).isEqualTo(i); } in.close(); fs.delete(path, false); @@ -85,8 +86,7 @@ private long writeAndVerify(int shift) throws IOException { out.close(); FSDataInputStream in = fs.open(path); for (int i = Short.MIN_VALUE; i <= Short.MAX_VALUE; ++i) { - long n = Utils.readVLong(in); - Assert.assertEquals(n, ((long) i) << shift); + assertThat(Utils.readVLong(in)).isEqualTo(((long) i) << shift); } in.close(); long ret = fs.getFileStatus(path).getLen(); @@ -165,7 +165,7 @@ public void testVLongRandom() throws IOException { FSDataInputStream in = fs.open(path); for (int i = 0; i < data.length; ++i) { - Assert.assertEquals(Utils.readVLong(in), data[i]); + assertThat(Utils.readVLong(in)).isEqualTo(data[i]); } in.close(); fs.delete(path, false); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java index 6b3c2325d8ff6..a14928c7b4e24 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java @@ -25,6 +25,8 @@ import java.io.FileWriter; import java.io.IOException; import java.io.RandomAccessFile; +import java.nio.file.Files; +import java.nio.file.Paths; import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel; import java.nio.channels.FileChannel.MapMode; @@ -782,4 +784,155 @@ public void testNativeFadviseConsts() { assertTrue("Native POSIX_FADV_NOREUSE const not set", POSIX_FADV_NOREUSE >= 0); } + + + @Test (timeout=10000) + public void testPmemCheckParameters() { + assumeNotWindows("Native PMDK not supported on Windows"); + // Skip testing while the build or environment does not support PMDK + assumeTrue(NativeIO.POSIX.isPmdkAvailable()); + + // Please make sure /mnt/pmem0 is a persistent memory device with total + // volume size 'volumeSize' + String filePath = "/$:"; + long length = 0; + long volumnSize = 16 * 1024 * 1024 * 1024L; + + // Incorrect file length + try { + NativeIO.POSIX.Pmem.mapBlock(filePath, length); + fail("Illegal length parameter should be detected"); + } catch (Exception e) { + LOG.info(e.getMessage()); + } + + // Incorrect file length + filePath = "/mnt/pmem0/test_native_io"; + length = -1L; + try { + NativeIO.POSIX.Pmem.mapBlock(filePath, length); + fail("Illegal length parameter should be detected"); + }catch (Exception e) { + LOG.info(e.getMessage()); + } + } + + @Test (timeout=10000) + public void testPmemMapMultipleFiles() { + assumeNotWindows("Native PMDK not supported on Windows"); + // Skip testing while the build or environment does not support PMDK + assumeTrue(NativeIO.POSIX.isPmdkAvailable()); + + // Please make sure /mnt/pmem0 is a persistent memory device with total + // volume size 'volumeSize' + String filePath = "/mnt/pmem0/test_native_io"; + long length = 0; + long volumnSize = 16 * 1024 * 1024 * 1024L; + + // Multiple files, each with 128MB size, aggregated size exceeds volume + // limit 16GB + length = 128 * 1024 * 1024L; + long fileNumber = volumnSize / length; + LOG.info("File number = " + fileNumber); + for (int i = 0; i < fileNumber; i++) { + String path = filePath + i; + LOG.info("File path = " + path); + NativeIO.POSIX.Pmem.mapBlock(path, length); + } + try { + NativeIO.POSIX.Pmem.mapBlock(filePath, length); + fail("Request map extra file when persistent memory is all occupied"); + } catch (Exception e) { + LOG.info(e.getMessage()); + } + } + + @Test (timeout=10000) + public void testPmemMapBigFile() { + assumeNotWindows("Native PMDK not supported on Windows"); + // Skip testing while the build or environment does not support PMDK + assumeTrue(NativeIO.POSIX.isPmdkAvailable()); + + // Please make sure /mnt/pmem0 is a persistent memory device with total + // volume size 'volumeSize' + String filePath = "/mnt/pmem0/test_native_io_big"; + long length = 0; + long volumeSize = 16 * 1024 * 1024 * 1024L; + + // One file length exceeds persistent memory volume 16GB. + length = volumeSize + 1024L; + try { + LOG.info("File length = " + length); + NativeIO.POSIX.Pmem.mapBlock(filePath, length); + fail("File length exceeds persistent memory total volume size"); + }catch (Exception e) { + LOG.info(e.getMessage()); + deletePmemMappedFile(filePath); + } + } + + @Test (timeout=10000) + public void testPmemCopy() throws IOException { + assumeNotWindows("Native PMDK not supported on Windows"); + // Skip testing while the build or environment does not support PMDK + assumeTrue(NativeIO.POSIX.isPmdkAvailable()); + + // Create and map a block file. Please make sure /mnt/pmem0 is a persistent + // memory device. + String filePath = "/mnt/pmem0/copy"; + long length = 4096; + PmemMappedRegion region = NativeIO.POSIX.Pmem.mapBlock(filePath, length); + assertTrue(NativeIO.POSIX.Pmem.isPmem(region.getAddress(), length)); + assertFalse(NativeIO.POSIX.Pmem.isPmem(region.getAddress(), length + 100)); + assertFalse(NativeIO.POSIX.Pmem.isPmem(region.getAddress() + 100, length)); + assertFalse(NativeIO.POSIX.Pmem.isPmem(region.getAddress() - 100, length)); + + // Copy content to mapped file + byte[] data = generateSequentialBytes(0, (int) length); + NativeIO.POSIX.Pmem.memCopy(data, region.getAddress(), region.isPmem(), + length); + + // Read content before pmemSync + byte[] readBuf1 = new byte[(int)length]; + IOUtils.readFully(new FileInputStream(filePath), readBuf1, 0, (int)length); + assertArrayEquals(data, readBuf1); + + byte[] readBuf2 = new byte[(int)length]; + // Sync content to persistent memory twice + NativeIO.POSIX.Pmem.memSync(region); + NativeIO.POSIX.Pmem.memSync(region); + // Read content after pmemSync twice + IOUtils.readFully(new FileInputStream(filePath), readBuf2, 0, (int)length); + assertArrayEquals(data, readBuf2); + + //Read content after unmap twice + NativeIO.POSIX.Pmem.unmapBlock(region.getAddress(), length); + NativeIO.POSIX.Pmem.unmapBlock(region.getAddress(), length); + byte[] readBuf3 = new byte[(int)length]; + IOUtils.readFully(new FileInputStream(filePath), readBuf3, 0, (int)length); + assertArrayEquals(data, readBuf3); + } + + private static byte[] generateSequentialBytes(int start, int length) { + byte[] result = new byte[length]; + + for (int i = 0; i < length; i++) { + result[i] = (byte) ((start + i) % 127); + } + return result; + } + + private static void deletePmemMappedFile(String filePath) { + try { + if (filePath != null) { + boolean result = Files.deleteIfExists(Paths.get(filePath)); + if (!result) { + throw new IOException(); + } + } + } catch (Throwable e) { + LOG.error("Failed to delete the mapped file " + filePath + + " from persistent memory", e); + } + } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java index 241e21046d9d4..3b42bb46e828c 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java @@ -25,6 +25,7 @@ import org.apache.hadoop.io.retry.UnreliableInterface.UnreliableException; import org.apache.hadoop.ipc.ProtocolTranslator; import org.apache.hadoop.ipc.RemoteException; +import org.apache.hadoop.security.AccessControlException; import org.junit.Before; import org.junit.Test; import org.mockito.invocation.InvocationOnMock; @@ -48,6 +49,14 @@ import static org.mockito.ArgumentMatchers.anyInt; import static org.mockito.Mockito.*; +/** + * TestRetryProxy tests the behaviour of the {@link RetryPolicy} class using + * a certain method of {@link UnreliableInterface} implemented by + * {@link UnreliableImplementation}. + * + * Some methods may be sensitive to the {@link Idempotent} annotation + * (annotated in {@link UnreliableInterface}). + */ public class TestRetryProxy { private UnreliableImplementation unreliableImpl; @@ -348,4 +357,24 @@ public void testNoRetryOnSaslError() throws Exception { assertEquals(RetryDecision.FAIL, caughtRetryAction.action); } } + + @Test + public void testNoRetryOnAccessControlException() throws Exception { + RetryPolicy policy = mock(RetryPolicy.class); + RetryPolicy realPolicy = RetryPolicies.failoverOnNetworkException(5); + setupMockPolicy(policy, realPolicy); + + UnreliableInterface unreliable = (UnreliableInterface) RetryProxy.create( + UnreliableInterface.class, unreliableImpl, policy); + + try { + unreliable.failsWithAccessControlExceptionEightTimes(); + fail("Should fail"); + } catch (AccessControlException e) { + // expected + verify(policy, times(1)).shouldRetry(any(Exception.class), anyInt(), + anyInt(), anyBoolean()); + assertEquals(RetryDecision.FAIL, caughtRetryAction.action); + } + } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java index 465dc6f94e499..a20d898988400 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java @@ -23,7 +23,14 @@ import org.apache.hadoop.ipc.RemoteException; import org.apache.hadoop.ipc.StandbyException; +import org.apache.hadoop.security.AccessControlException; +/** + * For the usage and purpose of this class see {@link UnreliableInterface} + * which this class implements. + * + * @see UnreliableInterface + */ class UnreliableImplementation implements UnreliableInterface { private int failsOnceInvocationCount, @@ -32,6 +39,7 @@ class UnreliableImplementation implements UnreliableInterface { failsOnceRemoteExceptionInvocationCount, failsTenTimesInvocationCount, failsWithSASLExceptionTenTimesInvocationCount, + failsWithAccessControlExceptionInvocationCount, succeedsOnceThenFailsCount, succeedsOnceThenFailsIdempotentCount, succeedsTenTimesThenFailsCount; @@ -123,6 +131,14 @@ public void failsWithSASLExceptionTenTimes() throws SaslException { } } + @Override + public void failsWithAccessControlExceptionEightTimes() + throws AccessControlException { + if (failsWithAccessControlExceptionInvocationCount++ < 8) { + throw new AccessControlException(); + } + } + @Override public String succeedsOnceThenFailsReturningString() throws UnreliableException, IOException, StandbyException { diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java index d334542ee69c5..738a76086bae2 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java @@ -24,7 +24,20 @@ import org.apache.hadoop.ipc.RemoteException; import org.apache.hadoop.ipc.StandbyException; +import org.apache.hadoop.security.AccessControlException; +/** + * The methods of UnreliableInterface could throw exceptions in a + * predefined way. It is currently used for testing {@link RetryPolicy} + * and {@link FailoverProxyProvider} classes, but can be potentially used + * to test any class's behaviour where an underlying interface or class + * may throw exceptions. + * + * Some methods may be annotated with the {@link Idempotent} annotation. + * In order to test those some methods of UnreliableInterface are annotated, + * but they are not actually Idempotent functions. + * + */ public interface UnreliableInterface { public static class UnreliableException extends Exception { @@ -66,6 +79,10 @@ public static class FatalException extends UnreliableException { void failsWithSASLExceptionTenTimes() throws SaslException; + @Idempotent + void failsWithAccessControlExceptionEightTimes() + throws AccessControlException; + public String succeedsOnceThenFailsReturningString() throws UnreliableException, StandbyException, IOException; @Idempotent diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java index 2b739966d8406..bb4717ed36d35 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java @@ -384,9 +384,22 @@ public void testCallQueueOverflowExceptions() throws Exception { RpcScheduler scheduler = Mockito.mock(RpcScheduler.class); BlockingQueue queue = Mockito.mock(BlockingQueue.class); CallQueueManager cqm = - Mockito.spy(new CallQueueManager<>(queue, scheduler, false)); + Mockito.spy(new CallQueueManager<>(queue, scheduler, false, false)); + CallQueueManager cqmTriggerFailover = + Mockito.spy(new CallQueueManager<>(queue, scheduler, false, true)); Schedulable call = new FakeCall(0); + // call queue exceptions that trigger failover + cqmTriggerFailover.setClientBackoffEnabled(true); + doReturn(Boolean.TRUE).when(cqmTriggerFailover).shouldBackOff(call); + try { + cqmTriggerFailover.put(call); + fail("didn't fail"); + } catch (Exception ex) { + assertEquals(CallQueueOverflowException.FAILOVER.getCause().getMessage(), + ex.getCause().getMessage()); + } + // call queue exceptions passed threw as-is doThrow(CallQueueOverflowException.KEEPALIVE).when(queue).add(call); try { diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestDecayRpcScheduler.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestDecayRpcScheduler.java index 10ab40ace1f67..7bdc6b5e96d0c 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestDecayRpcScheduler.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestDecayRpcScheduler.java @@ -26,6 +26,7 @@ import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; +import org.apache.hadoop.fs.CommonConfigurationKeys; import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.conf.Configuration; @@ -36,6 +37,7 @@ import java.io.ByteArrayOutputStream; import java.io.PrintStream; import java.lang.management.ManagementFactory; +import java.util.concurrent.TimeUnit; public class TestDecayRpcScheduler { private Schedulable mockCall(String id) { @@ -131,67 +133,69 @@ public void testAccumulate() { conf.set("ns." + DecayRpcScheduler.IPC_FCQ_DECAYSCHEDULER_PERIOD_KEY, "99999999"); // Never flush scheduler = new DecayRpcScheduler(1, "ns", conf); - assertEquals(0, scheduler.getCallCountSnapshot().size()); // empty first + assertEquals(0, scheduler.getCallCostSnapshot().size()); // empty first - scheduler.getPriorityLevel(mockCall("A")); - assertEquals(1, scheduler.getCallCountSnapshot().get("A").longValue()); - assertEquals(1, scheduler.getCallCountSnapshot().get("A").longValue()); + getPriorityIncrementCallCount("A"); + assertEquals(1, scheduler.getCallCostSnapshot().get("A").longValue()); + assertEquals(1, scheduler.getCallCostSnapshot().get("A").longValue()); - scheduler.getPriorityLevel(mockCall("A")); - scheduler.getPriorityLevel(mockCall("B")); - scheduler.getPriorityLevel(mockCall("A")); + getPriorityIncrementCallCount("A"); + getPriorityIncrementCallCount("B"); + getPriorityIncrementCallCount("A"); - assertEquals(3, scheduler.getCallCountSnapshot().get("A").longValue()); - assertEquals(1, scheduler.getCallCountSnapshot().get("B").longValue()); + assertEquals(3, scheduler.getCallCostSnapshot().get("A").longValue()); + assertEquals(1, scheduler.getCallCostSnapshot().get("B").longValue()); } @Test @SuppressWarnings("deprecation") public void testDecay() throws Exception { Configuration conf = new Configuration(); - conf.set("ns." + DecayRpcScheduler.IPC_FCQ_DECAYSCHEDULER_PERIOD_KEY, "999999999"); // Never - conf.set("ns." + DecayRpcScheduler.IPC_FCQ_DECAYSCHEDULER_FACTOR_KEY, "0.5"); + conf.setLong("ns." // Never decay + + DecayRpcScheduler.IPC_SCHEDULER_DECAYSCHEDULER_PERIOD_KEY, 999999999); + conf.setDouble("ns." + + DecayRpcScheduler.IPC_SCHEDULER_DECAYSCHEDULER_FACTOR_KEY, 0.5); scheduler = new DecayRpcScheduler(1, "ns", conf); assertEquals(0, scheduler.getTotalCallSnapshot()); for (int i = 0; i < 4; i++) { - scheduler.getPriorityLevel(mockCall("A")); + getPriorityIncrementCallCount("A"); } sleep(1000); for (int i = 0; i < 8; i++) { - scheduler.getPriorityLevel(mockCall("B")); + getPriorityIncrementCallCount("B"); } assertEquals(12, scheduler.getTotalCallSnapshot()); - assertEquals(4, scheduler.getCallCountSnapshot().get("A").longValue()); - assertEquals(8, scheduler.getCallCountSnapshot().get("B").longValue()); + assertEquals(4, scheduler.getCallCostSnapshot().get("A").longValue()); + assertEquals(8, scheduler.getCallCostSnapshot().get("B").longValue()); scheduler.forceDecay(); assertEquals(6, scheduler.getTotalCallSnapshot()); - assertEquals(2, scheduler.getCallCountSnapshot().get("A").longValue()); - assertEquals(4, scheduler.getCallCountSnapshot().get("B").longValue()); + assertEquals(2, scheduler.getCallCostSnapshot().get("A").longValue()); + assertEquals(4, scheduler.getCallCostSnapshot().get("B").longValue()); scheduler.forceDecay(); assertEquals(3, scheduler.getTotalCallSnapshot()); - assertEquals(1, scheduler.getCallCountSnapshot().get("A").longValue()); - assertEquals(2, scheduler.getCallCountSnapshot().get("B").longValue()); + assertEquals(1, scheduler.getCallCostSnapshot().get("A").longValue()); + assertEquals(2, scheduler.getCallCostSnapshot().get("B").longValue()); scheduler.forceDecay(); assertEquals(1, scheduler.getTotalCallSnapshot()); - assertEquals(null, scheduler.getCallCountSnapshot().get("A")); - assertEquals(1, scheduler.getCallCountSnapshot().get("B").longValue()); + assertEquals(null, scheduler.getCallCostSnapshot().get("A")); + assertEquals(1, scheduler.getCallCostSnapshot().get("B").longValue()); scheduler.forceDecay(); assertEquals(0, scheduler.getTotalCallSnapshot()); - assertEquals(null, scheduler.getCallCountSnapshot().get("A")); - assertEquals(null, scheduler.getCallCountSnapshot().get("B")); + assertEquals(null, scheduler.getCallCostSnapshot().get("A")); + assertEquals(null, scheduler.getCallCostSnapshot().get("B")); } @Test @@ -205,16 +209,16 @@ public void testPriority() throws Exception { .IPC_FCQ_DECAYSCHEDULER_THRESHOLDS_KEY, "25, 50, 75"); scheduler = new DecayRpcScheduler(4, namespace, conf); - assertEquals(0, scheduler.getPriorityLevel(mockCall("A"))); - assertEquals(2, scheduler.getPriorityLevel(mockCall("A"))); - assertEquals(0, scheduler.getPriorityLevel(mockCall("B"))); - assertEquals(1, scheduler.getPriorityLevel(mockCall("B"))); - assertEquals(0, scheduler.getPriorityLevel(mockCall("C"))); - assertEquals(0, scheduler.getPriorityLevel(mockCall("C"))); - assertEquals(1, scheduler.getPriorityLevel(mockCall("A"))); - assertEquals(1, scheduler.getPriorityLevel(mockCall("A"))); - assertEquals(1, scheduler.getPriorityLevel(mockCall("A"))); - assertEquals(2, scheduler.getPriorityLevel(mockCall("A"))); + assertEquals(0, getPriorityIncrementCallCount("A")); // 0 out of 0 calls + assertEquals(3, getPriorityIncrementCallCount("A")); // 1 out of 1 calls + assertEquals(0, getPriorityIncrementCallCount("B")); // 0 out of 2 calls + assertEquals(1, getPriorityIncrementCallCount("B")); // 1 out of 3 calls + assertEquals(0, getPriorityIncrementCallCount("C")); // 0 out of 4 calls + assertEquals(0, getPriorityIncrementCallCount("C")); // 1 out of 5 calls + assertEquals(1, getPriorityIncrementCallCount("A")); // 2 out of 6 calls + assertEquals(1, getPriorityIncrementCallCount("A")); // 3 out of 7 calls + assertEquals(2, getPriorityIncrementCallCount("A")); // 4 out of 8 calls + assertEquals(2, getPriorityIncrementCallCount("A")); // 5 out of 9 calls MBeanServer mbs = ManagementFactory.getPlatformMBeanServer(); ObjectName mxbeanName = new ObjectName( @@ -243,7 +247,7 @@ public void testPeriodic() throws InterruptedException { assertEquals(0, scheduler.getTotalCallSnapshot()); for (int i = 0; i < 64; i++) { - scheduler.getPriorityLevel(mockCall("A")); + getPriorityIncrementCallCount("A"); } // It should eventually decay to zero @@ -272,6 +276,108 @@ public void testNPEatInitialization() throws InterruptedException { //set systout back System.setOut(output); } + } + + @Test + public void testUsingWeightedTimeCostProvider() { + scheduler = getSchedulerWithWeightedTimeCostProvider(3); + + // 3 details in increasing order of cost. Although medium has a longer + // duration, the shared lock is weighted less than the exclusive lock + ProcessingDetails callDetailsLow = + new ProcessingDetails(TimeUnit.MILLISECONDS); + callDetailsLow.set(ProcessingDetails.Timing.LOCKFREE, 1); + ProcessingDetails callDetailsMedium = + new ProcessingDetails(TimeUnit.MILLISECONDS); + callDetailsMedium.set(ProcessingDetails.Timing.LOCKSHARED, 500); + ProcessingDetails callDetailsHigh = + new ProcessingDetails(TimeUnit.MILLISECONDS); + callDetailsHigh.set(ProcessingDetails.Timing.LOCKEXCLUSIVE, 100); + + for (int i = 0; i < 10; i++) { + scheduler.addResponseTime("ignored", mockCall("LOW"), callDetailsLow); + } + scheduler.addResponseTime("ignored", mockCall("MED"), callDetailsMedium); + scheduler.addResponseTime("ignored", mockCall("HIGH"), callDetailsHigh); + + assertEquals(0, scheduler.getPriorityLevel(mockCall("LOW"))); + assertEquals(1, scheduler.getPriorityLevel(mockCall("MED"))); + assertEquals(2, scheduler.getPriorityLevel(mockCall("HIGH"))); + + assertEquals(3, scheduler.getUniqueIdentityCount()); + long totalCallInitial = scheduler.getTotalRawCallVolume(); + assertEquals(totalCallInitial, scheduler.getTotalCallVolume()); + + scheduler.forceDecay(); + + // Relative priorities should stay the same after a single decay + assertEquals(0, scheduler.getPriorityLevel(mockCall("LOW"))); + assertEquals(1, scheduler.getPriorityLevel(mockCall("MED"))); + assertEquals(2, scheduler.getPriorityLevel(mockCall("HIGH"))); + + assertEquals(3, scheduler.getUniqueIdentityCount()); + assertEquals(totalCallInitial, scheduler.getTotalRawCallVolume()); + assertTrue(scheduler.getTotalCallVolume() < totalCallInitial); + + for (int i = 0; i < 100; i++) { + scheduler.forceDecay(); + } + // After enough decay cycles, all callers should be high priority again + assertEquals(0, scheduler.getPriorityLevel(mockCall("LOW"))); + assertEquals(0, scheduler.getPriorityLevel(mockCall("MED"))); + assertEquals(0, scheduler.getPriorityLevel(mockCall("HIGH"))); + } + + @Test + public void testUsingWeightedTimeCostProviderWithZeroCostCalls() { + scheduler = getSchedulerWithWeightedTimeCostProvider(2); + + ProcessingDetails emptyDetails = + new ProcessingDetails(TimeUnit.MILLISECONDS); + + for (int i = 0; i < 1000; i++) { + scheduler.addResponseTime("ignored", mockCall("MANY"), emptyDetails); + } + scheduler.addResponseTime("ignored", mockCall("FEW"), emptyDetails); + + // Since the calls are all "free", they should have the same priority + assertEquals(0, scheduler.getPriorityLevel(mockCall("MANY"))); + assertEquals(0, scheduler.getPriorityLevel(mockCall("FEW"))); + } + + @Test + public void testUsingWeightedTimeCostProviderNoRequests() { + scheduler = getSchedulerWithWeightedTimeCostProvider(2); + + assertEquals(0, scheduler.getPriorityLevel(mockCall("A"))); + } + + /** + * Get a scheduler that uses {@link WeightedTimeCostProvider} and has + * normal decaying disabled. + */ + private static DecayRpcScheduler getSchedulerWithWeightedTimeCostProvider( + int priorityLevels) { + Configuration conf = new Configuration(); + conf.setClass("ns." + CommonConfigurationKeys.IPC_COST_PROVIDER_KEY, + WeightedTimeCostProvider.class, CostProvider.class); + conf.setLong("ns." + + DecayRpcScheduler.IPC_SCHEDULER_DECAYSCHEDULER_PERIOD_KEY, 999999); + return new DecayRpcScheduler(priorityLevels, "ns", conf); + } + /** + * Get the priority and increment the call count, assuming that + * {@link DefaultCostProvider} is in use. + */ + private int getPriorityIncrementCallCount(String callId) { + Schedulable mockCall = mockCall(callId); + int priority = scheduler.getPriorityLevel(mockCall); + // The DefaultCostProvider uses a cost of 1 for all calls, ignoring + // the processing details, so an empty one is fine + ProcessingDetails emptyProcessingDetails = + new ProcessingDetails(TimeUnit.MILLISECONDS); + scheduler.addResponseTime("ignored", mockCall, emptyProcessingDetails); + return priority; } } \ No newline at end of file diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestFairCallQueue.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestFairCallQueue.java index 21e2018854cde..e6a5f5e564c1f 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestFairCallQueue.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestFairCallQueue.java @@ -28,9 +28,16 @@ import static org.mockito.Mockito.when; import static org.mockito.Mockito.times; +import org.apache.hadoop.fs.CommonConfigurationKeys; import org.junit.Before; import org.junit.Test; -import static org.junit.Assert.*; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.fail; import javax.management.MBeanServer; import javax.management.ObjectName; @@ -49,6 +56,8 @@ import org.apache.hadoop.ipc.CallQueueManager.CallQueueOverflowException; import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcResponseHeaderProto.RpcStatusProto; +import static org.assertj.core.api.Assertions.assertThat; + public class TestFairCallQueue { private FairCallQueue fcq; @@ -84,17 +93,17 @@ public void testTotalCapacityOfSubQueues() { Configuration conf = new Configuration(); FairCallQueue fairCallQueue; fairCallQueue = new FairCallQueue(1, 1000, "ns", conf); - assertEquals(fairCallQueue.remainingCapacity(), 1000); + assertThat(fairCallQueue.remainingCapacity()).isEqualTo(1000); fairCallQueue = new FairCallQueue(4, 1000, "ns", conf); - assertEquals(fairCallQueue.remainingCapacity(), 1000); + assertThat(fairCallQueue.remainingCapacity()).isEqualTo(1000); fairCallQueue = new FairCallQueue(7, 1000, "ns", conf); - assertEquals(fairCallQueue.remainingCapacity(), 1000); + assertThat(fairCallQueue.remainingCapacity()).isEqualTo(1000); fairCallQueue = new FairCallQueue(1, 1025, "ns", conf); - assertEquals(fairCallQueue.remainingCapacity(), 1025); + assertThat(fairCallQueue.remainingCapacity()).isEqualTo(1025); fairCallQueue = new FairCallQueue(4, 1025, "ns", conf); - assertEquals(fairCallQueue.remainingCapacity(), 1025); + assertThat(fairCallQueue.remainingCapacity()).isEqualTo(1025); fairCallQueue = new FairCallQueue(7, 1025, "ns", conf); - assertEquals(fairCallQueue.remainingCapacity(), 1025); + assertThat(fairCallQueue.remainingCapacity()).isEqualTo(1025); } @Test @@ -148,6 +157,65 @@ public int getAndAdvanceCurrentIndex() { assertNull(fcq.poll()); } + @SuppressWarnings("unchecked") + @Test + public void testInsertionWithFailover() { + Configuration conf = new Configuration(); + // Config for server to throw StandbyException instead of the + // regular RetriableException if call queue is full. + conf.setBoolean( + "ns." + CommonConfigurationKeys.IPC_CALLQUEUE_SERVER_FAILOVER_ENABLE, + true); + // 3 queues, 2 slots each. + fcq = Mockito.spy(new FairCallQueue<>(3, 6, "ns", conf)); + + Schedulable p0 = mockCall("a", 0); + Schedulable p1 = mockCall("b", 1); + + // add to first queue. + addToQueueAndVerify(p0, 1, 0, 0); + // 0:x- 1:-- 2:-- + + // add to second queue. + addToQueueAndVerify(p1, 0, 1, 0); + // 0:x- 1:x- 2:-- + + // add to first queue. + addToQueueAndVerify(p0, 1, 0, 0); + // 0:xx 1:x- 2:-- + + // add to first full queue spills over to second. + addToQueueAndVerify(p0, 1, 1, 0); + // 0:xx 1:xx 2:-- + + // add to second full queue spills over to third. + addToQueueAndVerify(p1, 0, 1, 1); + // 0:xx 1:xx 2:x- + + // add to first and second full queue spills over to third. + addToQueueAndVerify(p0, 1, 1, 1); + // 0:xx 1:xx 2:xx + + // adding non-lowest priority with all queues full throws a + // standby exception for client to try another server. + Mockito.reset(fcq); + try { + fcq.add(p0); + fail("didn't fail"); + } catch (IllegalStateException ise) { + checkOverflowException(ise, RpcStatusProto.FATAL, true); + } + } + + private void addToQueueAndVerify(Schedulable call, int expectedQueue0, + int expectedQueue1, int expectedQueue2) { + Mockito.reset(fcq); + fcq.add(call); + Mockito.verify(fcq, times(expectedQueue0)).offerQueue(0, call); + Mockito.verify(fcq, times(expectedQueue1)).offerQueue(1, call); + Mockito.verify(fcq, times(expectedQueue2)).offerQueue(2, call); + } + @SuppressWarnings("unchecked") // for mock reset. @Test public void testInsertion() throws Exception { @@ -215,7 +283,7 @@ public void testInsertion() throws Exception { fcq.add(p0); fail("didn't fail"); } catch (IllegalStateException ise) { - checkOverflowException(ise, RpcStatusProto.ERROR); + checkOverflowException(ise, RpcStatusProto.ERROR, false); } Mockito.verify(fcq, times(1)).offerQueue(0, p0); Mockito.verify(fcq, times(1)).offerQueue(1, p0); @@ -228,7 +296,7 @@ public void testInsertion() throws Exception { fcq.add(p1); fail("didn't fail"); } catch (IllegalStateException ise) { - checkOverflowException(ise, RpcStatusProto.ERROR); + checkOverflowException(ise, RpcStatusProto.ERROR, false); } Mockito.verify(fcq, times(0)).offerQueue(0, p1); Mockito.verify(fcq, times(1)).offerQueue(1, p1); @@ -241,7 +309,7 @@ public void testInsertion() throws Exception { fcq.add(p2); fail("didn't fail"); } catch (IllegalStateException ise) { - checkOverflowException(ise, RpcStatusProto.FATAL); + checkOverflowException(ise, RpcStatusProto.FATAL, false); } Mockito.verify(fcq, times(0)).offerQueue(0, p2); Mockito.verify(fcq, times(0)).offerQueue(1, p2); @@ -280,7 +348,8 @@ public void testInsertion() throws Exception { Mockito.verify(fcq, times(1)).putQueue(2, p2); } - private void checkOverflowException(Exception ex, RpcStatusProto status) { + private void checkOverflowException(Exception ex, RpcStatusProto status, + boolean failOverTriggered) { // should be an overflow exception assertTrue(ex.getClass().getName() + " != CallQueueOverflowException", ex instanceof CallQueueOverflowException); @@ -289,10 +358,15 @@ private void checkOverflowException(Exception ex, RpcStatusProto status) { assertTrue(ioe.getClass().getName() + " != RpcServerException", ioe instanceof RpcServerException); RpcServerException rse = (RpcServerException)ioe; - // check error/fatal status and if it embeds a retriable ex. + // check error/fatal status and if it embeds a retriable ex or standby ex. assertEquals(status, rse.getRpcStatusProto()); - assertTrue(rse.getClass().getName() + " != RetriableException", - rse.getCause() instanceof RetriableException); + if (failOverTriggered) { + assertTrue(rse.getClass().getName() + " != RetriableException", + rse.getCause() instanceof StandbyException); + } else { + assertTrue(rse.getClass().getName() + " != RetriableException", + rse.getCause() instanceof RetriableException); + } } // diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java index 1c1ad00451d9a..82540637a2004 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java @@ -24,7 +24,6 @@ import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; import static org.mockito.ArgumentMatchers.any; -import static org.mockito.ArgumentMatchers.anyInt; import static org.mockito.Mockito.doAnswer; import static org.mockito.Mockito.doThrow; import static org.mockito.Mockito.mock; @@ -106,6 +105,8 @@ import org.slf4j.LoggerFactory; import org.slf4j.event.Level; +import static org.assertj.core.api.Assertions.assertThat; + /** Unit tests for IPC. */ public class TestIPC { public static final Logger LOG = LoggerFactory.getLogger(TestIPC.class); @@ -1274,7 +1275,7 @@ public void testNoRetryOnInvalidToken() throws IOException { retryProxy.dummyRun(); } finally { // Check if dummyRun called only once - Assert.assertEquals(handler.invocations, 1); + assertThat(handler.invocations).isOne(); Client.setCallIdAndRetryCount(0, 0, null); client.stop(); server.stop(); @@ -1455,7 +1456,7 @@ public void run() { @Test public void testClientGetTimeout() throws IOException { Configuration config = new Configuration(); - assertEquals(Client.getTimeout(config), -1); + assertThat(Client.getTimeout(config)).isEqualTo(-1); } @Test(timeout=60000) @@ -1582,11 +1583,10 @@ public void testRpcResponseLimit() throws Throwable { try { call(client, 0, addr, conf); } catch (IOException ioe) { - Throwable t = ioe.getCause(); - Assert.assertNotNull(t); - Assert.assertEquals(RpcException.class, t.getClass()); + Assert.assertNotNull(ioe); + Assert.assertEquals(RpcException.class, ioe.getClass()); Assert.assertEquals("RPC response exceeds maximum data length", - t.getMessage()); + ioe.getMessage()); return; } Assert.fail("didn't get limit exceeded"); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProcessingDetails.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProcessingDetails.java new file mode 100644 index 0000000000000..0ecc741b014b3 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProcessingDetails.java @@ -0,0 +1,61 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ipc; + +import org.junit.Test; + +import java.util.concurrent.TimeUnit; + +import static org.apache.hadoop.ipc.ProcessingDetails.Timing; +import static org.junit.Assert.assertEquals; + +/** + * Unit tests for ProcessingDetails time unit conversion and output. + */ +public class TestProcessingDetails { + + /** + * Test that the conversion of time values in various units in and out of the + * details are done properly. + */ + @Test + public void testTimeConversion() { + ProcessingDetails details = new ProcessingDetails(TimeUnit.MICROSECONDS); + + details.set(Timing.ENQUEUE, 10); + assertEquals(10, details.get(Timing.ENQUEUE)); + assertEquals(10_000, details.get(Timing.ENQUEUE, TimeUnit.NANOSECONDS)); + + details.set(Timing.QUEUE, 20, TimeUnit.MILLISECONDS); + details.add(Timing.QUEUE, 20, TimeUnit.MICROSECONDS); + assertEquals(20_020, details.get(Timing.QUEUE)); + assertEquals(0, details.get(Timing.QUEUE, TimeUnit.SECONDS)); + } + + @Test + public void testToString() { + ProcessingDetails details = new ProcessingDetails(TimeUnit.MICROSECONDS); + details.set(Timing.ENQUEUE, 10); + details.set(Timing.QUEUE, 20, TimeUnit.MILLISECONDS); + + assertEquals("enqueueTime=10 queueTime=20000 handlerTime=0 " + + "processingTime=0 lockfreeTime=0 lockwaitTime=0 locksharedTime=0 " + + "lockexclusiveTime=0 responseTime=0", details.toString()); + } +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProtoBufRpc.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProtoBufRpc.java index 5fbd957312072..3053f87511885 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProtoBufRpc.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProtoBufRpc.java @@ -34,18 +34,19 @@ import org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos.TestProtobufRpcProto; import org.apache.hadoop.metrics2.MetricsRecordBuilder; import org.apache.hadoop.net.NetUtils; +import org.apache.hadoop.test.GenericTestUtils; import org.junit.After; -import org.junit.Assert; import org.junit.Before; import org.junit.Test; import java.io.IOException; import java.net.URISyntaxException; +import java.util.concurrent.TimeoutException; +import static org.assertj.core.api.Assertions.assertThat; import static org.apache.hadoop.test.MetricsAsserts.assertCounterGt; import static org.apache.hadoop.test.MetricsAsserts.getMetrics; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; /** * Test for testing protocol buffer based RPC mechanism. @@ -141,19 +142,19 @@ public static void testProtoBufRpc(TestRpcService client) throws Exception { EchoRequestProto echoRequest = EchoRequestProto.newBuilder() .setMessage("hello").build(); EchoResponseProto echoResponse = client.echo(null, echoRequest); - Assert.assertEquals(echoResponse.getMessage(), "hello"); + assertThat(echoResponse.getMessage()).isEqualTo("hello"); // Test error method - error should be thrown as RemoteException try { client.error(null, newEmptyRequest()); - Assert.fail("Expected exception is not thrown"); + fail("Expected exception is not thrown"); } catch (ServiceException e) { RemoteException re = (RemoteException)e.getCause(); RpcServerException rse = (RpcServerException) re .unwrapRemoteException(RpcServerException.class); - Assert.assertNotNull(rse); - Assert.assertTrue(re.getErrorCode().equals( - RpcErrorCodeProto.ERROR_RPC_SERVER)); + assertThat(rse).isNotNull(); + assertThat(re.getErrorCode()) + .isEqualTo(RpcErrorCodeProto.ERROR_RPC_SERVER); } } @@ -167,7 +168,7 @@ public void testProtoBufRpc2() throws Exception { // Test echo method EchoResponseProto echoResponse = client.echo2(null, newEchoRequest("hello")); - Assert.assertEquals(echoResponse.getMessage(), "hello"); + assertThat(echoResponse.getMessage()).isEqualTo("hello"); // Ensure RPC metrics are updated MetricsRecordBuilder rpcMetrics = getMetrics(server.getRpcMetrics().name()); @@ -186,13 +187,13 @@ public void testProtoBufRandomException() throws Exception { try { client.error2(null, newEmptyRequest()); } catch (ServiceException se) { - Assert.assertTrue(se.getCause() instanceof RemoteException); + assertThat(se.getCause()).isInstanceOf(RemoteException.class); RemoteException re = (RemoteException) se.getCause(); - Assert.assertTrue(re.getClassName().equals( - URISyntaxException.class.getName())); - Assert.assertTrue(re.getMessage().contains("testException")); - Assert.assertTrue( - re.getErrorCode().equals(RpcErrorCodeProto.ERROR_APPLICATION)); + assertThat(re.getClassName()) + .isEqualTo(URISyntaxException.class.getName()); + assertThat(re.getMessage()).contains("testException"); + assertThat(re.getErrorCode()) + .isEqualTo(RpcErrorCodeProto.ERROR_APPLICATION); } } @@ -203,19 +204,20 @@ public void testExtraLongRpc() throws Exception { // short message goes through EchoResponseProto echoResponse = client.echo2(null, newEchoRequest(shortString)); - Assert.assertEquals(shortString, echoResponse.getMessage()); + assertThat(echoResponse.getMessage()).isEqualTo(shortString); final String longString = StringUtils.repeat("X", 4096); try { client.echo2(null, newEchoRequest(longString)); - Assert.fail("expected extra-long RPC to fail"); + fail("expected extra-long RPC to fail"); } catch (ServiceException se) { // expected } } @Test(timeout = 12000) - public void testLogSlowRPC() throws IOException, ServiceException { + public void testLogSlowRPC() throws IOException, ServiceException, + TimeoutException, InterruptedException { TestRpcService2 client = getClient2(); // make 10 K fast calls for (int x = 0; x < 10000; x++) { @@ -228,15 +230,15 @@ public void testLogSlowRPC() throws IOException, ServiceException { // Ensure RPC metrics are updated RpcMetrics rpcMetrics = server.getRpcMetrics(); - assertTrue(rpcMetrics.getProcessingSampleCount() > 999L); + assertThat(rpcMetrics.getProcessingSampleCount()).isGreaterThan(999L); long before = rpcMetrics.getRpcSlowCalls(); // make a really slow call. Sleep sleeps for 1000ms client.sleep(null, newSleepRequest(SLEEP_DURATION * 3)); - long after = rpcMetrics.getRpcSlowCalls(); // Ensure slow call is logged. - Assert.assertEquals(before + 1L, after); + GenericTestUtils.waitFor(() + -> rpcMetrics.getRpcSlowCalls() == before + 1L, 10, 1000); } @Test(timeout = 12000) @@ -252,7 +254,7 @@ public void testEnsureNoLogIfDisabled() throws IOException, ServiceException { // Ensure RPC metrics are updated RpcMetrics rpcMetrics = server.getRpcMetrics(); - assertTrue(rpcMetrics.getProcessingSampleCount() > 999L); + assertThat(rpcMetrics.getProcessingSampleCount()).isGreaterThan(999L); long before = rpcMetrics.getRpcSlowCalls(); // make a really slow call. Sleep sleeps for 1000ms @@ -261,6 +263,6 @@ public void testEnsureNoLogIfDisabled() throws IOException, ServiceException { long after = rpcMetrics.getRpcSlowCalls(); // make sure we never called into Log slow RPC routine. - assertEquals(before, after); + assertThat(before).isEqualTo(after); } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java index 36a8885c9cfc8..0da0b47529f99 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java @@ -18,7 +18,6 @@ package org.apache.hadoop.ipc; -import com.google.common.base.Supplier; import com.google.protobuf.ServiceException; import org.apache.hadoop.HadoopIllegalArgumentException; import org.apache.hadoop.conf.Configuration; @@ -85,8 +84,11 @@ import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicReference; +import static org.assertj.core.api.Assertions.assertThat; import static org.apache.hadoop.test.MetricsAsserts.assertCounter; import static org.apache.hadoop.test.MetricsAsserts.assertCounterGt; +import static org.apache.hadoop.test.MetricsAsserts.assertGauge; +import static org.apache.hadoop.test.MetricsAsserts.getDoubleGauge; import static org.apache.hadoop.test.MetricsAsserts.getLongCounter; import static org.apache.hadoop.test.MetricsAsserts.getMetrics; import static org.junit.Assert.assertEquals; @@ -485,14 +487,14 @@ private void testCallsInternal(Configuration myConf) throws Exception { .setParam2(2).build(); TestProtos.AddResponseProto addResponse = proxy.add(null, addRequest); - assertEquals(addResponse.getResult(), 3); + assertThat(addResponse.getResult()).isEqualTo(3); Integer[] integers = new Integer[] {1, 2}; TestProtos.AddRequestProto2 addRequest2 = TestProtos.AddRequestProto2.newBuilder().addAllParams( Arrays.asList(integers)).build(); addResponse = proxy.add2(null, addRequest2); - assertEquals(addResponse.getResult(), 3); + assertThat(addResponse.getResult()).isEqualTo(3); boolean caught = false; try { @@ -1072,10 +1074,14 @@ public TestRpcService run() { } MetricsRecordBuilder rpcMetrics = getMetrics(server.getRpcMetrics().name()); - assertTrue("Expected non-zero rpc queue time", - getLongCounter("RpcQueueTimeNumOps", rpcMetrics) > 0); - assertTrue("Expected non-zero rpc processing time", - getLongCounter("RpcProcessingTimeNumOps", rpcMetrics) > 0); + assertEquals("Expected correct rpc queue count", + 3000, getLongCounter("RpcQueueTimeNumOps", rpcMetrics)); + assertEquals("Expected correct rpc processing count", + 3000, getLongCounter("RpcProcessingTimeNumOps", rpcMetrics)); + assertEquals("Expected correct rpc lock wait count", + 3000, getLongCounter("RpcLockWaitTimeNumOps", rpcMetrics)); + assertEquals("Expected zero rpc lock wait time", + 0, getDoubleGauge("RpcLockWaitTimeAvgTime", rpcMetrics), 0.001); MetricsAsserts.assertQuantileGauges("RpcQueueTime" + interval + "s", rpcMetrics); MetricsAsserts.assertQuantileGauges("RpcProcessingTime" + interval + "s", @@ -1086,6 +1092,10 @@ public TestRpcService run() { UserGroupInformation.getCurrentUser().getShortUserName(); assertTrue(actualUserVsCon.contains("\"" + proxyUser + "\":1")); assertTrue(actualUserVsCon.contains("\"" + testUser + "\":1")); + + proxy.lockAndSleep(null, newSleepRequest(5)); + rpcMetrics = getMetrics(server.getRpcMetrics().name()); + assertGauge("RpcLockWaitTimeAvgTime", 10000.0, rpcMetrics); } finally { if (proxy2 != null) { RPC.stopProxy(proxy2); @@ -1185,15 +1195,6 @@ public void testClientBackOffByResponseTime() throws Exception { Exception lastException = null; proxy = getClient(addr, conf); - MetricsRecordBuilder rb1 = - getMetrics("DecayRpcSchedulerMetrics2." + ns); - final long beginDecayedCallVolume = MetricsAsserts.getLongCounter( - "DecayedCallVolume", rb1); - final long beginRawCallVolume = MetricsAsserts.getLongCounter( - "CallVolume", rb1); - final int beginUniqueCaller = MetricsAsserts.getIntCounter("UniqueCallers", - rb1); - try { // start a sleep RPC call that sleeps 3s. for (int i = 0; i < numClients; i++) { @@ -1221,41 +1222,6 @@ public Void call() throws ServiceException, InterruptedException { } else { lastException = unwrapExeption; } - - // Lets Metric system update latest metrics - GenericTestUtils.waitFor(new Supplier() { - @Override - public Boolean get() { - MetricsRecordBuilder rb2 = - getMetrics("DecayRpcSchedulerMetrics2." + ns); - long decayedCallVolume1 = MetricsAsserts.getLongCounter( - "DecayedCallVolume", rb2); - long rawCallVolume1 = MetricsAsserts.getLongCounter( - "CallVolume", rb2); - int uniqueCaller1 = MetricsAsserts.getIntCounter( - "UniqueCallers", rb2); - long callVolumePriority0 = MetricsAsserts.getLongGauge( - "Priority.0.CompletedCallVolume", rb2); - long callVolumePriority1 = MetricsAsserts.getLongGauge( - "Priority.1.CompletedCallVolume", rb2); - double avgRespTimePriority0 = MetricsAsserts.getDoubleGauge( - "Priority.0.AvgResponseTime", rb2); - double avgRespTimePriority1 = MetricsAsserts.getDoubleGauge( - "Priority.1.AvgResponseTime", rb2); - - LOG.info("DecayedCallVolume: " + decayedCallVolume1); - LOG.info("CallVolume: " + rawCallVolume1); - LOG.info("UniqueCaller: " + uniqueCaller1); - LOG.info("Priority.0.CompletedCallVolume: " + callVolumePriority0); - LOG.info("Priority.1.CompletedCallVolume: " + callVolumePriority1); - LOG.info("Priority.0.AvgResponseTime: " + avgRespTimePriority0); - LOG.info("Priority.1.AvgResponseTime: " + avgRespTimePriority1); - - return decayedCallVolume1 > beginDecayedCallVolume && - rawCallVolume1 > beginRawCallVolume && - uniqueCaller1 > beginUniqueCaller; - } - }, 30, 60000); } } finally { executorService.shutdown(); @@ -1267,6 +1233,63 @@ public Boolean get() { assertTrue("RetriableException not received", succeeded); } + /** Test that the metrics for DecayRpcScheduler are updated. */ + @Test (timeout=30000) + public void testDecayRpcSchedulerMetrics() throws Exception { + final String ns = CommonConfigurationKeys.IPC_NAMESPACE + ".0"; + Server server = setupDecayRpcSchedulerandTestServer(ns + "."); + + MetricsRecordBuilder rb1 = + getMetrics("DecayRpcSchedulerMetrics2." + ns); + final long beginDecayedCallVolume = MetricsAsserts.getLongCounter( + "DecayedCallVolume", rb1); + final long beginRawCallVolume = MetricsAsserts.getLongCounter( + "CallVolume", rb1); + final int beginUniqueCaller = MetricsAsserts.getIntCounter("UniqueCallers", + rb1); + + TestRpcService proxy = getClient(addr, conf); + try { + for (int i = 0; i < 2; i++) { + proxy.sleep(null, newSleepRequest(100)); + } + + // Lets Metric system update latest metrics + GenericTestUtils.waitFor(() -> { + MetricsRecordBuilder rb2 = + getMetrics("DecayRpcSchedulerMetrics2." + ns); + long decayedCallVolume1 = MetricsAsserts.getLongCounter( + "DecayedCallVolume", rb2); + long rawCallVolume1 = MetricsAsserts.getLongCounter( + "CallVolume", rb2); + int uniqueCaller1 = MetricsAsserts.getIntCounter( + "UniqueCallers", rb2); + long callVolumePriority0 = MetricsAsserts.getLongGauge( + "Priority.0.CompletedCallVolume", rb2); + long callVolumePriority1 = MetricsAsserts.getLongGauge( + "Priority.1.CompletedCallVolume", rb2); + double avgRespTimePriority0 = MetricsAsserts.getDoubleGauge( + "Priority.0.AvgResponseTime", rb2); + double avgRespTimePriority1 = MetricsAsserts.getDoubleGauge( + "Priority.1.AvgResponseTime", rb2); + + LOG.info("DecayedCallVolume: {}", decayedCallVolume1); + LOG.info("CallVolume: {}", rawCallVolume1); + LOG.info("UniqueCaller: {}", uniqueCaller1); + LOG.info("Priority.0.CompletedCallVolume: {}", callVolumePriority0); + LOG.info("Priority.1.CompletedCallVolume: {}", callVolumePriority1); + LOG.info("Priority.0.AvgResponseTime: {}", avgRespTimePriority0); + LOG.info("Priority.1.AvgResponseTime: {}", avgRespTimePriority1); + + return decayedCallVolume1 > beginDecayedCallVolume && + rawCallVolume1 > beginRawCallVolume && + uniqueCaller1 > beginUniqueCaller; + }, 30, 60000); + } finally { + stop(server, proxy); + } + } + private Server setupDecayRpcSchedulerandTestServer(String ns) throws Exception { final int queueSizePerHandler = 3; diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRpcBase.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRpcBase.java index 0d2f975c1d676..2f2d36f7b45d7 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRpcBase.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRpcBase.java @@ -21,12 +21,16 @@ import com.google.protobuf.BlockingService; import com.google.protobuf.RpcController; import com.google.protobuf.ServiceException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.locks.Lock; +import java.util.concurrent.locks.ReentrantLock; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.ipc.protobuf.TestProtos; import org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos; import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.security.token.SecretManager; +import org.apache.hadoop.util.Time; import org.junit.Assert; import org.apache.hadoop.io.Text; @@ -278,6 +282,7 @@ public interface TestRpcService public static class PBServerImpl implements TestRpcService { CountDownLatch fastPingCounter = new CountDownLatch(2); private List postponedCalls = new ArrayList<>(); + private final Lock lock = new ReentrantLock(); @Override public TestProtos.EmptyResponseProto ping(RpcController unused, @@ -388,6 +393,29 @@ public TestProtos.EmptyResponseProto sleep( return TestProtos.EmptyResponseProto.newBuilder().build(); } + @Override + public TestProtos.EmptyResponseProto lockAndSleep( + RpcController controller, TestProtos.SleepRequestProto request) + throws ServiceException { + ProcessingDetails details = + Server.getCurCall().get().getProcessingDetails(); + lock.lock(); + long startNanos = Time.monotonicNowNanos(); + try { + Thread.sleep(request.getMilliSeconds()); + } catch (InterruptedException ignore) { + // ignore + } finally { + lock.unlock(); + } + // Add some arbitrary large lock wait time since in any test scenario + // the lock wait time will probably actually be too small to notice + details.add(ProcessingDetails.Timing.LOCKWAIT, 10, TimeUnit.SECONDS); + details.add(ProcessingDetails.Timing.LOCKEXCLUSIVE, + Time.monotonicNowNanos() - startNanos, TimeUnit.NANOSECONDS); + return TestProtos.EmptyResponseProto.newBuilder().build(); + } + @Override public TestProtos.AuthMethodResponseProto getAuthMethod( RpcController controller, TestProtos.EmptyRequestProto request) diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSocketFactory.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSocketFactory.java index ce481dc73dd32..1bad29e7750d1 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSocketFactory.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSocketFactory.java @@ -33,23 +33,20 @@ import javax.net.SocketFactory; -import org.junit.Assert; - import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.CommonConfigurationKeys; import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.net.SocksSocketFactory; import org.apache.hadoop.net.StandardSocketFactory; import org.junit.After; -import org.junit.Before; import org.junit.Test; -import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertSame; -import static org.junit.Assert.assertTrue; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertFalse; import static org.junit.Assert.fail; +import static org.assertj.core.api.Assertions.assertThat; + /** * test StandardSocketFactory and SocksSocketFactory NetUtils * @@ -111,10 +108,12 @@ public void testSocketFactoryAsKeyInMap() { .getDefaultSocketFactory(conf); dummyCache.put(defaultSocketFactory, toBeCached2); - Assert - .assertEquals("The cache contains two elements", 2, dummyCache.size()); - Assert.assertEquals("Equals of both socket factory shouldn't be same", - defaultSocketFactory.equals(dummySocketFactory), false); + assertThat(dummyCache.size()) + .withFailMessage("The cache contains two elements") + .isEqualTo(2); + assertThat(defaultSocketFactory) + .withFailMessage("Equals of both socket factory shouldn't be same") + .isNotEqualTo(dummySocketFactory); assertSame(toBeCached2, dummyCache.remove(defaultSocketFactory)); dummyCache.put(defaultSocketFactory, toBeCached2); @@ -184,14 +183,13 @@ public void testProxy() throws Exception { "localhost", 0)); SocksSocketFactory templateWithProxy = new SocksSocketFactory(proxy); - assertFalse(templateWithoutProxy.equals(templateWithProxy)); + assertThat(templateWithoutProxy).isNotEqualTo(templateWithProxy); Configuration configuration = new Configuration(); configuration.set("hadoop.socks.server", "localhost:0"); templateWithoutProxy.setConf(configuration); - assertTrue(templateWithoutProxy.equals(templateWithProxy)); - + assertThat(templateWithoutProxy).isEqualTo(templateWithProxy); } private void checkSocket(Socket socket) throws Exception { @@ -200,8 +198,7 @@ private void checkSocket(Socket socket) throws Exception { DataOutputStream out = new DataOutputStream(socket.getOutputStream()); out.writeBytes("test\n"); String answer = input.readLine(); - assertEquals("TEST", answer); - + assertThat(answer).isEqualTo("TEST"); } /** diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java index d4bc06ad3c3a1..11e2a9d917a19 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java @@ -18,7 +18,8 @@ package org.apache.hadoop.ipc; -import static org.junit.Assert.assertEquals; +import static org.assertj.core.api.Assertions.assertThat; + import org.junit.Test; import org.apache.hadoop.conf.Configuration; @@ -68,47 +69,47 @@ public void testDefaultPattern() { // Mux of size 1: 0 0 0 0 0, etc mux = new WeightedRoundRobinMultiplexer(1, "", new Configuration()); for(int i = 0; i < 10; i++) { - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); } // Mux of size 2: 0 0 1 0 0 1 0 0 1, etc mux = new WeightedRoundRobinMultiplexer(2, "", new Configuration()); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); // Size 3: 4x0 2x1 1x2, etc mux = new WeightedRoundRobinMultiplexer(3, "", new Configuration()); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); - assertEquals(mux.getAndAdvanceCurrentIndex(), 2); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); + assertThat(mux.getAndAdvanceCurrentIndex()).isEqualTo(2); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); // Size 4: 8x0 4x1 2x2 1x3 mux = new WeightedRoundRobinMultiplexer(4, "", new Configuration()); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); - assertEquals(mux.getAndAdvanceCurrentIndex(), 2); - assertEquals(mux.getAndAdvanceCurrentIndex(), 2); - assertEquals(mux.getAndAdvanceCurrentIndex(), 3); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); + assertThat(mux.getAndAdvanceCurrentIndex()).isEqualTo(2); + assertThat(mux.getAndAdvanceCurrentIndex()).isEqualTo(2); + assertThat(mux.getAndAdvanceCurrentIndex()).isEqualTo(3); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); } @Test @@ -119,10 +120,10 @@ public void testCustomPattern() { "1", "1"); mux = new WeightedRoundRobinMultiplexer(2, "test.custom", conf); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); // 1x0 3x1 2x2 conf.setStrings("test.custom." + IPC_CALLQUEUE_WRRMUX_WEIGHTS_KEY, @@ -131,12 +132,12 @@ public void testCustomPattern() { mux = new WeightedRoundRobinMultiplexer(3, "test.custom", conf); for(int i = 0; i < 5; i++) { - assertEquals(mux.getAndAdvanceCurrentIndex(), 0); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); - assertEquals(mux.getAndAdvanceCurrentIndex(), 1); - assertEquals(mux.getAndAdvanceCurrentIndex(), 2); - assertEquals(mux.getAndAdvanceCurrentIndex(), 2); + assertThat(mux.getAndAdvanceCurrentIndex()).isZero(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); + assertThat(mux.getAndAdvanceCurrentIndex()).isOne(); + assertThat(mux.getAndAdvanceCurrentIndex()).isEqualTo(2); + assertThat(mux.getAndAdvanceCurrentIndex()).isEqualTo(2); } // Ensure pattern repeats } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedTimeCostProvider.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedTimeCostProvider.java new file mode 100644 index 0000000000000..4f4a72b99ab4a --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedTimeCostProvider.java @@ -0,0 +1,86 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ipc; + +import java.util.concurrent.TimeUnit; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.ipc.ProcessingDetails.Timing; +import org.junit.Before; +import org.junit.Test; + +import static org.apache.hadoop.ipc.WeightedTimeCostProvider.DEFAULT_LOCKEXCLUSIVE_WEIGHT; +import static org.apache.hadoop.ipc.WeightedTimeCostProvider.DEFAULT_LOCKFREE_WEIGHT; +import static org.apache.hadoop.ipc.WeightedTimeCostProvider.DEFAULT_LOCKSHARED_WEIGHT; +import static org.junit.Assert.assertEquals; + +/** Tests for {@link WeightedTimeCostProvider}. */ +public class TestWeightedTimeCostProvider { + + private static final int QUEUE_TIME = 3; + private static final int LOCKFREE_TIME = 5; + private static final int LOCKSHARED_TIME = 7; + private static final int LOCKEXCLUSIVE_TIME = 11; + + private WeightedTimeCostProvider costProvider; + private ProcessingDetails processingDetails; + + @Before + public void setup() { + costProvider = new WeightedTimeCostProvider(); + processingDetails = new ProcessingDetails(TimeUnit.MILLISECONDS); + processingDetails.set(Timing.QUEUE, QUEUE_TIME); + processingDetails.set(Timing.LOCKFREE, LOCKFREE_TIME); + processingDetails.set(Timing.LOCKSHARED, LOCKSHARED_TIME); + processingDetails.set(Timing.LOCKEXCLUSIVE, LOCKEXCLUSIVE_TIME); + } + + @Test(expected = AssertionError.class) + public void testGetCostBeforeInit() { + costProvider.getCost(null); + } + + @Test + public void testGetCostDefaultWeights() { + costProvider.init("foo", new Configuration()); + long actualCost = costProvider.getCost(processingDetails); + long expectedCost = DEFAULT_LOCKFREE_WEIGHT * LOCKFREE_TIME + + DEFAULT_LOCKSHARED_WEIGHT * LOCKSHARED_TIME + + DEFAULT_LOCKEXCLUSIVE_WEIGHT * LOCKEXCLUSIVE_TIME; + assertEquals(expectedCost, actualCost); + } + + @Test + public void testGetCostConfiguredWeights() { + Configuration conf = new Configuration(); + int queueWeight = 1000; + int lockfreeWeight = 10000; + int locksharedWeight = 100000; + conf.setInt("foo.weighted-cost.queue", queueWeight); + conf.setInt("foo.weighted-cost.lockfree", lockfreeWeight); + conf.setInt("foo.weighted-cost.lockshared", locksharedWeight); + conf.setInt("bar.weighted-cost.lockexclusive", 0); // should not apply + costProvider.init("foo", conf); + long actualCost = costProvider.getCost(processingDetails); + long expectedCost = queueWeight * QUEUE_TIME + + lockfreeWeight * LOCKFREE_TIME + + locksharedWeight * LOCKSHARED_TIME + + DEFAULT_LOCKEXCLUSIVE_WEIGHT * LOCKEXCLUSIVE_TIME; + assertEquals(expectedCost, actualCost); + } +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/metrics/TestRpcMetrics.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/metrics/TestRpcMetrics.java new file mode 100644 index 0000000000000..1716433411181 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/metrics/TestRpcMetrics.java @@ -0,0 +1,60 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ipc.metrics; + +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.io.LongWritable; +import org.apache.hadoop.io.Writable; +import org.apache.hadoop.ipc.RPC; +import org.apache.hadoop.ipc.Server; +import org.apache.hadoop.metrics2.MetricsSystem; +import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; +import org.junit.Test; + +public class TestRpcMetrics { + + @Test + public void metricsAreUnregistered() throws Exception { + + Configuration conf = new Configuration(); + Server server = new Server("0.0.0.0", 0, LongWritable.class, 1, conf) { + @Override + public Writable call( + RPC.RpcKind rpcKind, String protocol, Writable param, + long receiveTime) throws Exception { + return null; + } + }; + MetricsSystem metricsSystem = DefaultMetricsSystem.instance(); + RpcMetrics rpcMetrics = server.getRpcMetrics(); + RpcDetailedMetrics rpcDetailedMetrics = server.getRpcDetailedMetrics(); + + assertNotNull(metricsSystem.getSource(rpcMetrics.name())); + assertNotNull(metricsSystem.getSource(rpcDetailedMetrics.name())); + + server.stop(); + + assertNull(metricsSystem.getSource(rpcMetrics.name())); + assertNull(metricsSystem.getSource(rpcDetailedMetrics.name())); + + } + +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogLevel.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogLevel.java index fd30b50141f28..3af70e95548ba 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogLevel.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogLevel.java @@ -34,6 +34,7 @@ import org.apache.hadoop.log.LogLevel.CLI; import org.apache.hadoop.minikdc.KerberosSecurityTestcase; import org.apache.hadoop.net.NetUtils; +import org.apache.hadoop.security.AuthenticationFilterInitializer; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.security.authentication.KerberosTestUtils; import org.apache.hadoop.security.authentication.client.KerberosAuthenticator; @@ -73,6 +74,7 @@ public class TestLogLevel extends KerberosSecurityTestcase { private final Logger log = ((Log4JLogger)testlog).getLogger(); private final static String PRINCIPAL = "loglevel.principal"; private final static String KEYTAB = "loglevel.keytab"; + private static final String PREFIX = "hadoop.http.authentication."; @BeforeClass public static void setUp() throws Exception { @@ -262,6 +264,13 @@ private void testDynamicLogLevel(final String bindProtocol, conf.set(KEYTAB, KerberosTestUtils.getKeytabFile()); conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, "kerberos"); + conf.set(PREFIX + "type", "kerberos"); + conf.set(PREFIX + "kerberos.keytab", KerberosTestUtils.getKeytabFile()); + conf.set(PREFIX + "kerberos.principal", + KerberosTestUtils.getServerPrincipal()); + conf.set(HttpServer2.FILTER_INITIALIZER_PROPERTY, + AuthenticationFilterInitializer.class.getName()); + conf.setBoolean(CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, true); UserGroupInformation.setConfiguration(conf); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java index 29d47cfab05ac..b5f62b189040e 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java @@ -274,6 +274,23 @@ private static void snapshotMutableRatesWithAggregation( } } + @Test + public void testDuplicateMetrics() { + MutableRatesWithAggregation rates = new MutableRatesWithAggregation(); + MutableRatesWithAggregation deferredRpcRates = + new MutableRatesWithAggregation(); + Class protocol = Long.class; + rates.init(protocol); + deferredRpcRates.init(protocol, "Deferred"); + MetricsRecordBuilder rb = mockMetricsRecordBuilder(); + rates.snapshot(rb, true); + deferredRpcRates.snapshot(rb, true); + verify(rb, times(1)) + .addCounter(info("GetLongNumOps", "Number of ops for getLong"), 0L); + verify(rb, times(1)).addCounter( + info("GetLongDeferredNumOps", "Number of ops for getLongDeferred"), 0L); + } + /** * Tests that when using {@link MutableStat#add(long, long)}, even with a high * sample count, the mean does not lose accuracy. diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestPrometheusMetricsSink.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestPrometheusMetricsSink.java new file mode 100644 index 0000000000000..3fc4aa4cc3430 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestPrometheusMetricsSink.java @@ -0,0 +1,133 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.metrics2.sink; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; + +import org.apache.hadoop.metrics2.MetricsSystem; +import org.apache.hadoop.metrics2.annotation.Metric; +import org.apache.hadoop.metrics2.annotation.Metrics; +import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; +import org.apache.hadoop.metrics2.lib.MutableCounterLong; + +import org.junit.Assert; +import org.junit.Test; + +import static java.nio.charset.StandardCharsets.UTF_8; + +/** + * Test prometheus Sink. + */ +public class TestPrometheusMetricsSink { + + @Test + public void testPublish() throws IOException { + //GIVEN + MetricsSystem metrics = DefaultMetricsSystem.instance(); + + metrics.init("test"); + PrometheusMetricsSink sink = new PrometheusMetricsSink(); + metrics.register("Prometheus", "Prometheus", sink); + TestMetrics testMetrics = metrics + .register("TestMetrics", "Testing metrics", new TestMetrics()); + + metrics.start(); + testMetrics.numBucketCreateFails.incr(); + metrics.publishMetricsNow(); + ByteArrayOutputStream stream = new ByteArrayOutputStream(); + OutputStreamWriter writer = new OutputStreamWriter(stream, UTF_8); + + //WHEN + sink.writeMetrics(writer); + writer.flush(); + + //THEN + String writtenMetrics = stream.toString(UTF_8.name()); + System.out.println(writtenMetrics); + Assert.assertTrue( + "The expected metric line is missing from prometheus metrics output", + writtenMetrics.contains( + "test_metrics_num_bucket_create_fails{context=\"dfs\"") + ); + + metrics.stop(); + metrics.shutdown(); + } + + @Test + public void testNamingCamelCase() { + PrometheusMetricsSink sink = new PrometheusMetricsSink(); + + Assert.assertEquals("rpc_time_some_metrics", + sink.prometheusName("RpcTime", "SomeMetrics")); + + Assert.assertEquals("om_rpc_time_om_info_keys", + sink.prometheusName("OMRpcTime", "OMInfoKeys")); + + Assert.assertEquals("rpc_time_small", + sink.prometheusName("RpcTime", "small")); + } + + @Test + public void testNamingPipeline() { + PrometheusMetricsSink sink = new PrometheusMetricsSink(); + + String recordName = "SCMPipelineMetrics"; + String metricName = "NumBlocksAllocated-" + + "RATIS-THREE-47659e3d-40c9-43b3-9792-4982fc279aba"; + Assert.assertEquals( + "scm_pipeline_metrics_" + + "num_blocks_allocated_" + + "ratis_three_47659e3d_40c9_43b3_9792_4982fc279aba", + sink.prometheusName(recordName, metricName)); + } + + @Test + public void testNamingPeriods() { + PrometheusMetricsSink sink = new PrometheusMetricsSink(); + + String recordName = "org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl"; + String metricName = "DfsUsed"; + Assert.assertEquals( + "org_apache_hadoop_hdfs_server_datanode_fsdataset_impl_fs_dataset_impl_dfs_used", + sink.prometheusName(recordName, metricName)); + } + + @Test + public void testNamingWhitespaces() { + PrometheusMetricsSink sink = new PrometheusMetricsSink(); + + String recordName = "JvmMetrics"; + String metricName = "GcCount" + "G1 Old Generation"; + Assert.assertEquals( + "jvm_metrics_gc_count_g1_old_generation", + sink.prometheusName(recordName, metricName)); + } + + /** + * Example metric pojo. + */ + @Metrics(about = "Test Metrics", context = "dfs") + private static class TestMetrics { + + @Metric + private MutableCounterLong numBucketCreateFails; + } +} \ No newline at end of file diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/util/TestSampleQuantiles.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/util/TestSampleQuantiles.java index 5c5f0368c04cf..c7d8f60b181d2 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/util/TestSampleQuantiles.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/util/TestSampleQuantiles.java @@ -18,8 +18,6 @@ package org.apache.hadoop.metrics2.util; -import static org.junit.Assert.*; - import java.io.IOException; import java.util.Arrays; import java.util.Collections; @@ -29,6 +27,8 @@ import org.junit.Before; import org.junit.Test; +import static org.assertj.core.api.Assertions.assertThat; + public class TestSampleQuantiles { static final Quantile[] quantiles = { new Quantile(0.50, 0.050), @@ -49,24 +49,24 @@ public void init() { @Test public void testCount() throws IOException { // Counts start off zero - assertEquals(estimator.getCount(), 0); - assertEquals(estimator.getSampleCount(), 0); + assertThat(estimator.getCount()).isZero(); + assertThat(estimator.getSampleCount()).isZero(); // Snapshot should be null if there are no entries. - assertNull(estimator.snapshot()); + assertThat(estimator.snapshot()).isNull(); // Count increment correctly by 1 estimator.insert(1337); - assertEquals(estimator.getCount(), 1); + assertThat(estimator.getCount()).isOne(); estimator.snapshot(); - assertEquals(estimator.getSampleCount(), 1); + assertThat(estimator.getSampleCount()).isOne(); - assertEquals( + assertThat(estimator.toString()).isEqualTo( "50.00 %ile +/- 5.00%: 1337\n" + "75.00 %ile +/- 2.50%: 1337\n" + "90.00 %ile +/- 1.00%: 1337\n" + "95.00 %ile +/- 0.50%: 1337\n" + - "99.00 %ile +/- 0.10%: 1337", estimator.toString()); + "99.00 %ile +/- 0.10%: 1337"); } /** @@ -79,9 +79,9 @@ public void testClear() throws IOException { estimator.insert(i); } estimator.clear(); - assertEquals(estimator.getCount(), 0); - assertEquals(estimator.getSampleCount(), 0); - assertNull(estimator.snapshot()); + assertThat(estimator.getCount()).isZero(); + assertThat(estimator.getSampleCount()).isZero(); + assertThat(estimator.snapshot()).isNull(); } /** @@ -113,8 +113,8 @@ public void testQuantileError() throws IOException { System.out .println(String.format("Expected %d with error %d, estimated %d", actual, error, estimate)); - assertTrue(estimate <= actual + error); - assertTrue(estimate >= actual - error); + assertThat(estimate <= actual + error).isTrue(); + assertThat(estimate >= actual - error).isTrue(); } } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetUtils.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetUtils.java index 62bd1b142e2da..b11b1e96ded59 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetUtils.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetUtils.java @@ -279,11 +279,9 @@ public void testWrapKerbAuthException() throws Throwable { @Test public void testWrapIOEWithNoStringConstructor() throws Throwable { IOException e = new CharacterCodingException(); - IOException wrapped = verifyExceptionClass(e, IOException.class); - assertInException(wrapped, "Failed on local exception"); - assertNotInException(wrapped, NetUtils.HADOOP_WIKI); - assertInException(wrapped, "Host Details "); - assertRemoteDetailsIncluded(wrapped); + IOException wrapped = + verifyExceptionClass(e, CharacterCodingException.class); + assertEquals(null, wrapped.getMessage()); } @Test @@ -295,11 +293,8 @@ private TestIOException(String cause){ } } IOException e = new TestIOException(); - IOException wrapped = verifyExceptionClass(e, IOException.class); - assertInException(wrapped, "Failed on local exception"); - assertNotInException(wrapped, NetUtils.HADOOP_WIKI); - assertInException(wrapped, "Host Details "); - assertRemoteDetailsIncluded(wrapped); + IOException wrapped = verifyExceptionClass(e, TestIOException.class); + assertEquals(null, wrapped.getMessage()); } @Test diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java index bba81522a4138..4c471da4e8c35 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java @@ -42,11 +42,10 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.CommonConfigurationKeys; -import org.apache.hadoop.security.Groups; -import org.apache.hadoop.security.ShellBasedUnixGroupsMapping; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.assertj.core.api.Assertions.assertThat; public class TestGroupsCaching { public static final Logger TESTLOG = @@ -494,7 +493,7 @@ public void testThreadNotBlockedWhenExpiredEntryExistsWithBackgroundRefresh() // Now get the cache entry - it should return immediately // with the old value and the cache will not have completed // a request to getGroups yet. - assertEquals(groups.getGroups("me").size(), 2); + assertThat(groups.getGroups("me").size()).isEqualTo(2); assertEquals(startingRequestCount, FakeGroupMapping.getRequestCount()); // Now sleep for over the delay time and the request count should @@ -502,7 +501,7 @@ public void testThreadNotBlockedWhenExpiredEntryExistsWithBackgroundRefresh() Thread.sleep(110); assertEquals(startingRequestCount + 1, FakeGroupMapping.getRequestCount()); // Another call to get groups should give 3 groups instead of 2 - assertEquals(groups.getGroups("me").size(), 3); + assertThat(groups.getGroups("me").size()).isEqualTo(3); } @Test @@ -532,7 +531,7 @@ public void testThreadBlockedWhenExpiredEntryExistsWithoutBackgroundRefresh() // Now get the cache entry - it should block and return the new // 3 group value - assertEquals(groups.getGroups("me").size(), 3); + assertThat(groups.getGroups("me").size()).isEqualTo(3); assertEquals(startingRequestCount + 1, FakeGroupMapping.getRequestCount()); } @@ -567,7 +566,7 @@ public void testExceptionOnBackgroundRefreshHandled() throws Exception { // Now get the cache entry - it should return immediately // with the old value and the cache will not have completed // a request to getGroups yet. - assertEquals(groups.getGroups("me").size(), 2); + assertThat(groups.getGroups("me").size()).isEqualTo(2); assertEquals(startingRequestCount, FakeGroupMapping.getRequestCount()); // Resume the getGroups operation and the cache can get refreshed FakeGroupMapping.resume(); @@ -577,14 +576,14 @@ public void testExceptionOnBackgroundRefreshHandled() throws Exception { waitForGroupCounters(groups, 0, 0, 0, 1); FakeGroupMapping.setThrowException(false); assertEquals(startingRequestCount + 1, FakeGroupMapping.getRequestCount()); - assertEquals(groups.getGroups("me").size(), 2); + assertThat(groups.getGroups("me").size()).isEqualTo(2); // Now the 3rd call to getGroups above will have kicked off // another refresh that updates the cache, since it no longer gives // exception, we now expect the counter for success is 1. waitForGroupCounters(groups, 0, 0, 1, 1); assertEquals(startingRequestCount + 2, FakeGroupMapping.getRequestCount()); - assertEquals(groups.getGroups("me").size(), 3); + assertThat(groups.getGroups("me").size()).isEqualTo(3); } @@ -613,7 +612,7 @@ public void testEntriesExpireIfBackgroundRefreshFails() throws Exception { // be triggered which will fail to update the key, but the keys old value // will be retrievable until it is evicted after about 10 seconds. for(int i=0; i<9; i++) { - assertEquals(groups.getGroups("me").size(), 2); + assertThat(groups.getGroups("me").size()).isEqualTo(2); timer.advance(1 * 1000); } // Wait until the 11th second. The call to getGroups should throw @@ -631,7 +630,7 @@ public void testEntriesExpireIfBackgroundRefreshFails() throws Exception { // Finally check groups are retrieve again after FakeGroupMapping // stops throw exceptions FakeGroupMapping.setThrowException(false); - assertEquals(groups.getGroups("me").size(), 2); + assertThat(groups.getGroups("me").size()).isEqualTo(2); } @Test @@ -725,14 +724,14 @@ public void testExceptionCallingLoadWithoutBackgroundRefreshReturnsOldValue() FakeGroupMapping.clearBlackList(); // First populate the cash - assertEquals(groups.getGroups("me").size(), 2); + assertThat(groups.getGroups("me").size()).isEqualTo(2); // Advance the timer so a refresh is required timer.advance(2 * 1000); // This call should throw an exception FakeGroupMapping.setThrowException(true); - assertEquals(groups.getGroups("me").size(), 2); + assertThat(groups.getGroups("me").size()).isEqualTo(2); } @Test diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestShellBasedIdMapping.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestShellBasedIdMapping.java index d589c3a34677e..e6fdc2bcdfbd4 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestShellBasedIdMapping.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestShellBasedIdMapping.java @@ -35,6 +35,8 @@ import com.google.common.collect.BiMap; import com.google.common.collect.HashBiMap; +import static org.assertj.core.api.Assertions.assertThat; + public class TestShellBasedIdMapping { private static final Map EMPTY_PASS_THROUGH_MAP = @@ -295,18 +297,19 @@ public void testIdOutOfIntegerRange() throws IOException { @Test public void testUserUpdateSetting() throws IOException { ShellBasedIdMapping iug = new ShellBasedIdMapping(new Configuration()); - assertEquals(iug.getTimeout(), + assertThat(iug.getTimeout()).isEqualTo( IdMappingConstant.USERGROUPID_UPDATE_MILLIS_DEFAULT); Configuration conf = new Configuration(); conf.setLong(IdMappingConstant.USERGROUPID_UPDATE_MILLIS_KEY, 0); iug = new ShellBasedIdMapping(conf); - assertEquals(iug.getTimeout(), IdMappingConstant.USERGROUPID_UPDATE_MILLIS_MIN); + assertThat(iug.getTimeout()).isEqualTo( + IdMappingConstant.USERGROUPID_UPDATE_MILLIS_MIN); conf.setLong(IdMappingConstant.USERGROUPID_UPDATE_MILLIS_KEY, IdMappingConstant.USERGROUPID_UPDATE_MILLIS_DEFAULT * 2); iug = new ShellBasedIdMapping(conf); - assertEquals(iug.getTimeout(), + assertThat(iug.getTimeout()).isEqualTo( IdMappingConstant.USERGROUPID_UPDATE_MILLIS_DEFAULT * 2); } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGIWithSecurityOn.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGIWithSecurityOn.java deleted file mode 100644 index 028cc38f1b358..0000000000000 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGIWithSecurityOn.java +++ /dev/null @@ -1,117 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with this - * work for additional information regarding copyright ownership. The ASF - * licenses this file to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - */ -package org.apache.hadoop.security; - -import java.io.IOException; -import java.security.PrivilegedAction; -import java.util.Set; - -import javax.security.auth.kerberos.KerberosPrincipal; - -import org.junit.Assert; -import static org.junit.Assert.*; - - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod; -import org.junit.Assume; -import org.junit.Before; -import org.junit.Test; - -public class TestUGIWithSecurityOn { - - public static boolean isKdcRunning() { - String startKdc = System.getProperty("startKdc"); - if(startKdc == null || !startKdc.equals("true")) { - return false; - } - return true; - } - - @Before - public void testKdcRunning() { - //Tests are skipped if KDC is not running - Assume.assumeTrue(isKdcRunning()); - } - @Test - public void testLogin() throws IOException { - String nn1keyTabFilepath = System.getProperty("kdc.resource.dir") - + "/keytabs/nn1.keytab"; - String user1keyTabFilepath = System.getProperty("kdc.resource.dir") - + "/keytabs/user1.keytab"; - Configuration conf = new Configuration(); - SecurityUtil.setAuthenticationMethod(AuthenticationMethod.KERBEROS, conf); - UserGroupInformation.setConfiguration(conf); - - UserGroupInformation ugiNn = UserGroupInformation - .loginUserFromKeytabAndReturnUGI("nn1/localhost@EXAMPLE.COM", - nn1keyTabFilepath); - UserGroupInformation ugiDn = UserGroupInformation - .loginUserFromKeytabAndReturnUGI("user1@EXAMPLE.COM", - user1keyTabFilepath); - - Assert.assertEquals(AuthenticationMethod.KERBEROS, - ugiNn.getAuthenticationMethod()); - Assert.assertEquals(AuthenticationMethod.KERBEROS, - ugiDn.getAuthenticationMethod()); - - try { - UserGroupInformation - .loginUserFromKeytabAndReturnUGI("bogus@EXAMPLE.COM", - nn1keyTabFilepath); - Assert.fail("Login should have failed"); - } catch (Exception ex) { - ex.printStackTrace(); - } - } - - @Test - public void testGetUGIFromKerberosSubject() throws IOException { - String user1keyTabFilepath = System.getProperty("kdc.resource.dir") - + "/keytabs/user1.keytab"; - - UserGroupInformation ugi = UserGroupInformation - .loginUserFromKeytabAndReturnUGI("user1@EXAMPLE.COM", - user1keyTabFilepath); - Set principals = ugi.getSubject().getPrincipals( - KerberosPrincipal.class); - if (principals.isEmpty()) { - Assert.fail("There should be a kerberos principal in the subject."); - } - else { - UserGroupInformation ugi2 = UserGroupInformation.getUGIFromSubject( - ugi.getSubject()); - if (ugi2 != null) { - ugi2.doAs(new PrivilegedAction() { - - @Override - public Object run() { - try { - UserGroupInformation ugi3 = UserGroupInformation.getCurrentUser(); - String doAsUserName = ugi3.getUserName(); - assertEquals(doAsUserName, "user1@EXAMPLE.COM"); - System.out.println("DO AS USERNAME: " + doAsUserName); - } catch (IOException e) { - e.printStackTrace(); - } - return null; - } - }); - } - } - } -} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredShell.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredShell.java index 569fe738a01e1..bf72b52b32066 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredShell.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredShell.java @@ -32,6 +32,7 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.security.ProviderUtils; import org.apache.hadoop.test.GenericTestUtils; +import org.assertj.core.api.Assertions; import org.junit.Before; import org.junit.Test; @@ -43,6 +44,11 @@ public class TestCredShell { /* The default JCEKS provider - for testing purposes */ private String jceksProvider; + private void assertOutputContains(String expected) { + Assertions.assertThat(outContent.toString()) + .contains(expected); + } + @Before public void setup() throws Exception { System.setOut(new PrintStream(outContent)); @@ -172,15 +178,28 @@ public void testPromptForCredential() throws Exception { shell.setPasswordReader(new MockPasswordReader(passwords)); rc = shell.run(args1); assertEquals(0, rc); - assertTrue(outContent.toString().contains("credential1 has been successfully " + - "created.")); - - String[] args2 = {"delete", "credential1", "-f", "-provider", + assertOutputContains("credential1 has been successfully created."); + + String[] args2 = {"check", "credential1", "-provider", jceksProvider}; + ArrayList password = new ArrayList(); + password.add("p@ssw0rd"); + shell.setPasswordReader(new MockPasswordReader(password)); rc = shell.run(args2); assertEquals(0, rc); - assertTrue(outContent.toString().contains("credential1 has been successfully " + - "deleted.")); + assertOutputContains("Password match success for credential1."); + ArrayList passwordError = new ArrayList(); + passwordError.add("p@ssw0rderr"); + shell.setPasswordReader(new MockPasswordReader(password)); + rc = shell.run(args2); + assertEquals(0, rc); + assertOutputContains("Password match failed for credential1."); + + String[] args3 = {"delete", "credential1", "-f", "-provider", + jceksProvider}; + rc = shell.run(args3); + assertEquals(0, rc); + assertOutputContains("credential1 has been successfully deleted."); } public class MockPasswordReader extends CredentialShell.PasswordReader { diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authentication/server/TestProxyUserAuthenticationFilter.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authentication/server/TestProxyUserAuthenticationFilter.java new file mode 100644 index 0000000000000..16c0e1eb112ac --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authentication/server/TestProxyUserAuthenticationFilter.java @@ -0,0 +1,125 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.security.authentication.server; + +import java.security.Principal; +import java.util.Collections; +import java.util.Enumeration; +import java.util.HashMap; +import java.util.Map; + +import javax.servlet.FilterConfig; +import javax.servlet.FilterChain; +import javax.servlet.ServletContext; +import javax.servlet.ServletResponse; +import javax.servlet.ServletRequest; +import javax.servlet.http.HttpServletRequest; + +import static org.assertj.core.api.Assertions.assertThat; +import org.glassfish.grizzly.servlet.HttpServletResponseImpl; +import org.junit.Test; +import org.mockito.Mockito; + + +/** + * Test ProxyUserAuthenticationFilter with doAs Request Parameter. + */ +public class TestProxyUserAuthenticationFilter { + + private String actualUser; + + private static class DummyFilterConfig implements FilterConfig { + private final Map map; + + DummyFilterConfig(Map map) { + this.map = map; + } + + @Override + public String getFilterName() { + return "dummy"; + } + + @Override + public String getInitParameter(String param) { + return map.get(param); + } + + @Override + public Enumeration getInitParameterNames() { + return Collections.enumeration(map.keySet()); + } + + @Override + public ServletContext getServletContext() { + ServletContext context = Mockito.mock(ServletContext.class); + Mockito.when(context.getAttribute( + AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) + .thenReturn(null); + return context; + } + } + + private class HttpServletResponseForTest extends HttpServletResponseImpl { + + } + + + @Test(timeout = 10000) + public void testFilter() throws Exception { + Map params = new HashMap(); + params.put("proxyuser.knox.users", "testuser"); + params.put("proxyuser.knox.hosts", "127.0.0.1"); + params.put("type", "simple"); + + FilterConfig config = new DummyFilterConfig(params); + + FilterChain chain = new FilterChain() { + @Override + public void doFilter(ServletRequest servletRequest, + ServletResponse servletResponse) { + HttpServletRequest request = (HttpServletRequest) servletRequest; + actualUser = request.getRemoteUser(); + } + }; + + ProxyUserAuthenticationFilter testFilter = + new ProxyUserAuthenticationFilter(); + testFilter.init(config); + + HttpServletRequest request = Mockito.mock(HttpServletRequest.class); + Mockito.when(request.getRemoteUser()).thenReturn("knox"); + Mockito.when(request.getParameter("doas")).thenReturn("testuser"); + Mockito.when(request.getRemoteAddr()).thenReturn("127.0.0.1"); + Mockito.when(request.getUserPrincipal()).thenReturn(new Principal() { + @Override + public String getName() { + return "knox@EXAMPLE.COM"; + } + }); + + HttpServletResponseForTest response = new HttpServletResponseForTest(); + + testFilter.doFilter(chain, request, response); + + assertThat(actualUser).isEqualTo("testuser"); + } + + +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java index 7039001b8fe45..8e1b82bea9605 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java @@ -17,9 +17,7 @@ */ package org.apache.hadoop.security.authorize; -import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; import java.util.Collection; @@ -37,6 +35,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.assertj.core.api.Assertions.assertThat; import static org.mockito.Mockito.never; import static org.mockito.Mockito.spy; import static org.mockito.Mockito.verify; @@ -187,29 +186,29 @@ public void testAclString() { AccessControlList acl; acl = new AccessControlList("*"); - assertEquals("All users are allowed", acl.toString()); + assertThat(acl.toString()).isEqualTo("All users are allowed"); validateGetAclString(acl); acl = new AccessControlList(" "); - assertEquals("No users are allowed", acl.toString()); + assertThat(acl.toString()).isEqualTo("No users are allowed"); acl = new AccessControlList("user1,user2"); - assertEquals("Users [user1, user2] are allowed", acl.toString()); + assertThat(acl.toString()).isEqualTo("Users [user1, user2] are allowed"); validateGetAclString(acl); acl = new AccessControlList("user1,user2 ");// with space - assertEquals("Users [user1, user2] are allowed", acl.toString()); + assertThat(acl.toString()).isEqualTo("Users [user1, user2] are allowed"); validateGetAclString(acl); acl = new AccessControlList(" group1,group2"); - assertTrue(acl.toString().equals( - "Members of the groups [group1, group2] are allowed")); + assertThat(acl.toString()).isEqualTo( + "Members of the groups [group1, group2] are allowed"); validateGetAclString(acl); acl = new AccessControlList("user1,user2 group1,group2"); - assertTrue(acl.toString().equals( + assertThat(acl.toString()).isEqualTo( "Users [user1, user2] and " + - "members of the groups [group1, group2] are allowed")); + "members of the groups [group1, group2] are allowed"); validateGetAclString(acl); } @@ -228,45 +227,45 @@ public void testAccessControlList() throws Exception { acl = new AccessControlList("drwho tardis"); users = acl.getUsers(); - assertEquals(users.size(), 1); - assertEquals(users.iterator().next(), "drwho"); + assertThat(users.size()).isOne(); + assertThat(users.iterator().next()).isEqualTo("drwho"); groups = acl.getGroups(); - assertEquals(groups.size(), 1); - assertEquals(groups.iterator().next(), "tardis"); + assertThat(groups.size()).isOne(); + assertThat(groups.iterator().next()).isEqualTo("tardis"); acl = new AccessControlList("drwho"); users = acl.getUsers(); - assertEquals(users.size(), 1); - assertEquals(users.iterator().next(), "drwho"); + assertThat(users.size()).isOne(); + assertThat(users.iterator().next()).isEqualTo("drwho"); groups = acl.getGroups(); - assertEquals(groups.size(), 0); + assertThat(groups.size()).isZero(); acl = new AccessControlList("drwho "); users = acl.getUsers(); - assertEquals(users.size(), 1); - assertEquals(users.iterator().next(), "drwho"); + assertThat(users.size()).isOne(); + assertThat(users.iterator().next()).isEqualTo("drwho"); groups = acl.getGroups(); - assertEquals(groups.size(), 0); + assertThat(groups.size()).isZero(); acl = new AccessControlList(" tardis"); users = acl.getUsers(); - assertEquals(users.size(), 0); + assertThat(users.size()).isZero(); groups = acl.getGroups(); - assertEquals(groups.size(), 1); - assertEquals(groups.iterator().next(), "tardis"); + assertThat(groups.size()).isOne(); + assertThat(groups.iterator().next()).isEqualTo("tardis"); Iterator iter; acl = new AccessControlList("drwho,joe tardis, users"); users = acl.getUsers(); - assertEquals(users.size(), 2); + assertThat(users.size()).isEqualTo(2); iter = users.iterator(); - assertEquals(iter.next(), "drwho"); - assertEquals(iter.next(), "joe"); + assertThat(iter.next()).isEqualTo("drwho"); + assertThat(iter.next()).isEqualTo("joe"); groups = acl.getGroups(); - assertEquals(groups.size(), 2); + assertThat(groups.size()).isEqualTo(2); iter = groups.iterator(); - assertEquals(iter.next(), "tardis"); - assertEquals(iter.next(), "users"); + assertThat(iter.next()).isEqualTo("tardis"); + assertThat(iter.next()).isEqualTo("users"); } /** @@ -278,58 +277,58 @@ public void testAddRemoveAPI() { Collection users; Collection groups; acl = new AccessControlList(" "); - assertEquals(0, acl.getUsers().size()); - assertEquals(0, acl.getGroups().size()); - assertEquals(" ", acl.getAclString()); + assertThat(acl.getUsers().size()).isZero(); + assertThat(acl.getGroups().size()).isZero(); + assertThat(acl.getAclString()).isEqualTo(" "); acl.addUser("drwho"); users = acl.getUsers(); - assertEquals(users.size(), 1); - assertEquals(users.iterator().next(), "drwho"); - assertEquals("drwho ", acl.getAclString()); + assertThat(users.size()).isOne(); + assertThat(users.iterator().next()).isEqualTo("drwho"); + assertThat(acl.getAclString()).isEqualTo("drwho "); acl.addGroup("tardis"); groups = acl.getGroups(); - assertEquals(groups.size(), 1); - assertEquals(groups.iterator().next(), "tardis"); - assertEquals("drwho tardis", acl.getAclString()); + assertThat(groups.size()).isOne(); + assertThat(groups.iterator().next()).isEqualTo("tardis"); + assertThat(acl.getAclString()).isEqualTo("drwho tardis"); acl.addUser("joe"); acl.addGroup("users"); users = acl.getUsers(); - assertEquals(users.size(), 2); + assertThat(users.size()).isEqualTo(2); Iterator iter = users.iterator(); - assertEquals(iter.next(), "drwho"); - assertEquals(iter.next(), "joe"); + assertThat(iter.next()).isEqualTo("drwho"); + assertThat(iter.next()).isEqualTo("joe"); groups = acl.getGroups(); - assertEquals(groups.size(), 2); + assertThat(groups.size()).isEqualTo(2); iter = groups.iterator(); - assertEquals(iter.next(), "tardis"); - assertEquals(iter.next(), "users"); - assertEquals("drwho,joe tardis,users", acl.getAclString()); + assertThat(iter.next()).isEqualTo("tardis"); + assertThat(iter.next()).isEqualTo("users"); + assertThat(acl.getAclString()).isEqualTo("drwho,joe tardis,users"); acl.removeUser("joe"); acl.removeGroup("users"); users = acl.getUsers(); - assertEquals(users.size(), 1); + assertThat(users.size()).isOne(); assertFalse(users.contains("joe")); groups = acl.getGroups(); - assertEquals(groups.size(), 1); + assertThat(groups.size()).isOne(); assertFalse(groups.contains("users")); - assertEquals("drwho tardis", acl.getAclString()); + assertThat(acl.getAclString()).isEqualTo("drwho tardis"); acl.removeGroup("tardis"); groups = acl.getGroups(); - assertEquals(0, groups.size()); + assertThat(groups.size()).isZero(); assertFalse(groups.contains("tardis")); - assertEquals("drwho ", acl.getAclString()); + assertThat(acl.getAclString()).isEqualTo("drwho "); acl.removeUser("drwho"); - assertEquals(0, users.size()); + assertThat(users.size()).isZero(); assertFalse(users.contains("drwho")); - assertEquals(0, acl.getGroups().size()); - assertEquals(0, acl.getUsers().size()); - assertEquals(" ", acl.getAclString()); + assertThat(acl.getGroups().size()).isZero(); + assertThat(acl.getUsers().size()).isZero(); + assertThat(acl.getAclString()).isEqualTo(" "); } /** @@ -345,8 +344,8 @@ public void testAddRemoveWildCard() { } catch (Throwable t) { th = t; } - assertNotNull(th); - assertTrue(th instanceof IllegalArgumentException); + assertThat(th).isNotNull(); + assertThat(th).isInstanceOf(IllegalArgumentException.class); th = null; try { @@ -354,24 +353,24 @@ public void testAddRemoveWildCard() { } catch (Throwable t) { th = t; } - assertNotNull(th); - assertTrue(th instanceof IllegalArgumentException); + assertThat(th).isNotNull(); + assertThat(th).isInstanceOf(IllegalArgumentException.class); th = null; try { acl.removeUser(" * "); } catch (Throwable t) { th = t; } - assertNotNull(th); - assertTrue(th instanceof IllegalArgumentException); + assertThat(th).isNotNull(); + assertThat(th).isInstanceOf(IllegalArgumentException.class); th = null; try { acl.removeGroup(" * "); } catch (Throwable t) { th = t; } - assertNotNull(th); - assertTrue(th instanceof IllegalArgumentException); + assertThat(th).isNotNull(); + assertThat(th).isInstanceOf(IllegalArgumentException.class); } /** diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestServiceAuthorization.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestServiceAuthorization.java index c473c502da824..d02fe604d79e3 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestServiceAuthorization.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestServiceAuthorization.java @@ -20,13 +20,18 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.fail; +import java.lang.annotation.Annotation; import java.net.InetAddress; import java.net.UnknownHostException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.CommonConfigurationKeys; import org.apache.hadoop.ipc.TestRPC.TestProtocol; +import org.apache.hadoop.security.KerberosInfo; +import org.apache.hadoop.security.SecurityInfo; +import org.apache.hadoop.security.SecurityUtil; import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.token.TokenInfo; import org.junit.Test; public class TestServiceAuthorization { @@ -52,6 +57,53 @@ public Service[] getServices() { } } + private static class CustomSecurityInfo extends SecurityInfo { + @Override + public KerberosInfo getKerberosInfo(Class protocol, + Configuration conf) { + return new KerberosInfo() { + @Override + public Class annotationType() { + return null; + } + @Override + public String serverPrincipal() { + return null; + } + @Override + public String clientPrincipal() { + return "dfs.datanode.kerberos.principal"; + } + }; + } + + @Override + public TokenInfo getTokenInfo(Class protocol, Configuration conf) { + return null; + } + } + + @Test + public void testWithClientPrincipalOnUnsecureMode() + throws UnknownHostException { + UserGroupInformation hdfsUser = UserGroupInformation.createUserForTesting( + "hdfs", new String[] {"hadoop"}); + ServiceAuthorizationManager serviceAuthorizationManager = + new ServiceAuthorizationManager(); + SecurityUtil.setSecurityInfoProviders(new CustomSecurityInfo()); + + Configuration conf = new Configuration(); + conf.set("dfs.datanode.kerberos.principal", "dn/_HOST@EXAMPLE.COM"); + conf.set(ACL_CONFIG, "user1 hadoop"); + serviceAuthorizationManager.refresh(conf, new TestPolicyProvider()); + try { + serviceAuthorizationManager.authorize(hdfsUser, TestProtocol.class, conf, + InetAddress.getByName(ADDRESS)); + } catch (AuthorizationException e) { + fail(); + } + } + @Test public void testDefaultAcl() { ServiceAuthorizationManager serviceAuthorizationManager = diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestXFrameOptionsFilter.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestXFrameOptionsFilter.java index 1c1d97d7cac7c..0f9f691322e70 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestXFrameOptionsFilter.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestXFrameOptionsFilter.java @@ -31,6 +31,7 @@ import org.mockito.invocation.InvocationOnMock; import org.mockito.stubbing.Answer; +import static org.assertj.core.api.Assertions.assertThat; import static org.mockito.ArgumentMatchers.any; /** @@ -84,8 +85,9 @@ public Object answer(InvocationOnMock invocation) throws Throwable { filter.doFilter(request, response, chain); - Assert.assertEquals("X-Frame-Options count not equal to 1.", - headers.size(), 1); + assertThat(headers.size()) + .withFailMessage("X-Frame-Options count not equal to 1.") + .isOne(); } @Test @@ -138,10 +140,12 @@ public Object answer(InvocationOnMock invocation) throws Throwable { filter.doFilter(request, response, chain); - Assert.assertEquals("X-Frame-Options count not equal to 1.", - headers.size(), 1); + assertThat(headers.size()) + .withFailMessage("X-Frame-Options count not equal to 1.") + .isOne(); - Assert.assertEquals("X-Frame-Options count not equal to 1.", - headers.toArray()[0], "SAMEORIGIN"); + assertThat(headers.toArray()[0]) + .withFailMessage("X-Frame-Options count not equal to 1.") + .isEqualTo("SAMEORIGIN"); } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/TestDelegatingSSLSocketFactory.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/TestDelegatingSSLSocketFactory.java new file mode 100644 index 0000000000000..f19f65b18cfe6 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/TestDelegatingSSLSocketFactory.java @@ -0,0 +1,57 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.security.ssl; + +import java.io.IOException; +import java.util.Arrays; + +import org.junit.Test; + +import org.apache.hadoop.util.NativeCodeLoader; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.Assume.assumeTrue; + +/** + * Tests for {@link DelegatingSSLSocketFactory}. + */ +public class TestDelegatingSSLSocketFactory { + + @Test + public void testOpenSSL() throws IOException { + assumeTrue("Unable to load native libraries", + NativeCodeLoader.isNativeCodeLoaded()); + assumeTrue("Build was not compiled with support for OpenSSL", + NativeCodeLoader.buildSupportsOpenssl()); + DelegatingSSLSocketFactory.initializeDefaultFactory( + DelegatingSSLSocketFactory.SSLChannelMode.OpenSSL); + assertThat(DelegatingSSLSocketFactory.getDefaultFactory() + .getProviderName()).contains("openssl"); + } + + @Test + public void testJSEENoGCMJava8() throws IOException { + assumeTrue("Not running on Java 8", + System.getProperty("java.version").startsWith("1.8")); + DelegatingSSLSocketFactory.initializeDefaultFactory( + DelegatingSSLSocketFactory.SSLChannelMode.Default_JSSE); + assertThat(Arrays.stream(DelegatingSSLSocketFactory.getDefaultFactory() + .getSupportedCipherSuites())).noneMatch("GCM"::contains); + } +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/TestOpenSSLSocketFactory.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/TestOpenSSLSocketFactory.java deleted file mode 100644 index ea881e990b934..0000000000000 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/TestOpenSSLSocketFactory.java +++ /dev/null @@ -1,53 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.security.ssl; - -import java.io.IOException; -import java.util.Arrays; - -import org.junit.Test; - -import org.apache.hadoop.util.NativeCodeLoader; - -import static org.assertj.core.api.Assertions.assertThat; -import static org.junit.Assume.assumeTrue; - -/** - * Tests for {@link OpenSSLSocketFactory}. - */ -public class TestOpenSSLSocketFactory { - - @Test - public void testOpenSSL() throws IOException { - assumeTrue(NativeCodeLoader.buildSupportsOpenssl()); - OpenSSLSocketFactory.initializeDefaultFactory( - OpenSSLSocketFactory.SSLChannelMode.OpenSSL); - assertThat(OpenSSLSocketFactory.getDefaultFactory() - .getProviderName()).contains("openssl"); - } - - @Test - public void testJSEEJava8() throws IOException { - assumeTrue(System.getProperty("java.version").startsWith("1.8")); - OpenSSLSocketFactory.initializeDefaultFactory( - OpenSSLSocketFactory.SSLChannelMode.Default_JSSE); - assertThat(Arrays.stream(OpenSSLSocketFactory.getDefaultFactory() - .getSupportedCipherSuites())).noneMatch("GCM"::contains); - } -} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestDelegationToken.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestDelegationToken.java index df685cf681a2d..8bc881ae5d1da 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestDelegationToken.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestDelegationToken.java @@ -50,6 +50,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.assertj.core.api.Assertions.assertThat; import static org.junit.Assert.*; public class TestDelegationToken { @@ -266,17 +267,17 @@ public void testDelegationTokenCount() throws Exception { 3*1000, 1*1000, 3600000); try { dtSecretManager.startThreads(); - Assert.assertEquals(dtSecretManager.getCurrentTokensSize(), 0); + assertThat(dtSecretManager.getCurrentTokensSize()).isZero(); final Token token1 = generateDelegationToken(dtSecretManager, "SomeUser", "JobTracker"); - Assert.assertEquals(dtSecretManager.getCurrentTokensSize(), 1); + assertThat(dtSecretManager.getCurrentTokensSize()).isOne(); final Token token2 = generateDelegationToken(dtSecretManager, "SomeUser", "JobTracker"); - Assert.assertEquals(dtSecretManager.getCurrentTokensSize(), 2); + assertThat(dtSecretManager.getCurrentTokensSize()).isEqualTo(2); dtSecretManager.cancelToken(token1, "JobTracker"); - Assert.assertEquals(dtSecretManager.getCurrentTokensSize(), 1); + assertThat(dtSecretManager.getCurrentTokensSize()).isOne(); dtSecretManager.cancelToken(token2, "JobTracker"); - Assert.assertEquals(dtSecretManager.getCurrentTokensSize(), 0); + assertThat(dtSecretManager.getCurrentTokensSize()).isZero(); } finally { dtSecretManager.stopThreads(); } @@ -386,7 +387,7 @@ public void testRollMasterKey() throws Exception { //after rolling, the length of the keys list must increase int currNumKeys = dtSecretManager.getAllKeys().length; - Assert.assertEquals((currNumKeys - prevNumKeys) >= 1, true); + assertThat(currNumKeys - prevNumKeys).isGreaterThanOrEqualTo(1); //after rolling, the token that was generated earlier must //still be valid (retrievePassword will fail if the token diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java index 1fcc6faac646a..9b5bd22dbe6ac 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java @@ -22,6 +22,7 @@ import org.apache.hadoop.io.Text; import org.apache.hadoop.minikdc.MiniKdc; import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.authentication.KerberosTestUtils; import org.apache.hadoop.security.authentication.client.AuthenticationException; import org.apache.hadoop.security.authentication.client.KerberosAuthenticator; import org.apache.hadoop.security.authentication.server.AuthenticationFilter; @@ -743,12 +744,6 @@ private void testKerberosDelegationTokenAuthenticator( final boolean doAs) throws Exception { final String doAsUser = doAs ? OK_USER : null; - // setting hadoop security to kerberos - org.apache.hadoop.conf.Configuration conf = - new org.apache.hadoop.conf.Configuration(); - conf.set("hadoop.security.authentication", "kerberos"); - UserGroupInformation.setConfiguration(conf); - File testDir = new File("target/" + UUID.randomUUID().toString()); Assert.assertTrue(testDir.mkdirs()); MiniKdc kdc = new MiniKdc(MiniKdc.createConf(), testDir); @@ -759,6 +754,10 @@ private void testKerberosDelegationTokenAuthenticator( context.addFilter(new FilterHolder(KDTAFilter.class), "/*", EnumSet.of(DispatcherType.REQUEST)); context.addServlet(new ServletHolder(UserServlet.class), "/bar"); + org.apache.hadoop.conf.Configuration conf = + new org.apache.hadoop.conf.Configuration(); + conf.set("hadoop.security.authentication", "kerberos"); + conf.set("java.security.krb5.realm", KerberosTestUtils.getRealm()); try { kdc.start(); File keytabFile = new File(testDir, "test.keytab"); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java index c1b6cc4081e5c..db36154c158ac 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java @@ -575,6 +575,9 @@ private static String robustToString(Object o) { if (o == null) { return NULL_RESULT; } else { + if (o instanceof String) { + return '"' + (String)o + '"'; + } try { return o.toString(); } catch (Exception e) { diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java index 19f213d31a12b..c043d1e590e5d 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java @@ -27,7 +27,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import static org.junit.Assert.*; +import static org.assertj.core.api.Assertions.assertThat; /** Testing {@link LightWeightResizableGSet} */ public class TestLightWeightResizableGSet { @@ -132,23 +132,23 @@ public void testBasicOperations() { final LightWeightResizableGSet set = new LightWeightResizableGSet(); - assertEquals(set.size(), 0); + assertThat(set.size()).isZero(); // put all elements for (int i = 0; i < elements.length; i++) { TestElement element = set.put(elements[i]); - assertTrue(element == null); + assertThat(element).isNull(); } // check the set size - assertEquals(set.size(), elements.length); + assertThat(set.size()).isEqualTo(elements.length); // check all elements exist in the set and the data is correct for (int i = 0; i < elements.length; i++) { - assertTrue(set.contains(elements[i])); + assertThat(set.contains(elements[i])).isTrue(); TestElement element = set.get(elements[i]); - assertEquals(elements[i].getData(), element.getData()); + assertThat(elements[i].getData()).isEqualTo(element.getData()); } TestKey[] keys = getKeys(elements); @@ -157,39 +157,38 @@ public void testBasicOperations() { // update the set for (int i = 0; i < newElements.length; i++) { TestElement element = set.put(newElements[i]); - assertTrue(element != null); + assertThat(element).isNotNull(); } // check the set size - assertEquals(set.size(), elements.length); + assertThat(set.size()).isEqualTo(elements.length); // check all elements exist in the set and the data is updated to new value for (int i = 0; i < keys.length; i++) { - assertTrue(set.contains(keys[i])); + assertThat(set.contains(keys[i])).isTrue(); TestElement element = set.get(keys[i]); - assertEquals(newElements[i].getData(), element.getData()); + assertThat(newElements[i].getData()).isEqualTo(element.getData()); } // test LightWeightHashGSet#values Collection cElements = set.values(); - assertEquals(cElements.size(), elements.length); + assertThat(cElements.size()).isEqualTo(elements.length); for (TestElement element : cElements) { - assertTrue(set.contains(element)); + assertThat(set.contains(element)).isTrue(); } // remove elements for (int i = 0; i < keys.length; i++) { TestElement element = set.remove(keys[i]); - - assertTrue(element != null); + assertThat(element).isNotNull(); // the element should not exist after remove - assertFalse(set.contains(keys[i])); + assertThat(set.contains(keys[i])).isFalse(); } // check the set size - assertEquals(set.size(), 0); + assertThat(set.size()).isZero(); } @Test(timeout = 60000) @@ -198,33 +197,33 @@ public void testRemoveAll() { final LightWeightResizableGSet set = new LightWeightResizableGSet(); - assertEquals(set.size(), 0); + assertThat(set.size()).isZero(); // put all elements for (int i = 0; i < elements.length; i++) { TestElement element = set.put(elements[i]); - assertTrue(element == null); + assertThat(element).isNull(); } // check the set size - assertEquals(set.size(), elements.length); + assertThat(set.size()).isEqualTo(elements.length); // remove all through clear { set.clear(); - assertEquals(set.size(), 0); + assertThat(set.size()).isZero(); // check all elements removed for (int i = 0; i < elements.length; i++) { - assertFalse(set.contains(elements[i])); + assertThat(set.contains(elements[i])).isFalse(); } - assertFalse(set.iterator().hasNext()); + assertThat(set.iterator().hasNext()).isFalse(); } // put all elements back for (int i = 0; i < elements.length; i++) { TestElement element = set.put(elements[i]); - assertTrue(element == null); + assertThat(element).isNull(); } // remove all through iterator @@ -232,22 +231,22 @@ public void testRemoveAll() { for (Iterator iter = set.iterator(); iter.hasNext(); ) { TestElement element = iter.next(); // element should be there before removing - assertTrue(set.contains(element)); + assertThat(set.contains(element)).isTrue(); iter.remove(); // element should not be there now - assertFalse(set.contains(element)); + assertThat(set.contains(element)).isFalse(); } // the deleted elements should not be there for (int i = 0; i < elements.length; i++) { - assertFalse(set.contains(elements[i])); + assertThat(set.contains(elements[i])).isFalse(); } // iterator should not have next - assertFalse(set.iterator().hasNext()); + assertThat(set.iterator().hasNext()).isFalse(); // check the set size - assertEquals(set.size(), 0); + assertThat(set.size()).isZero(); } } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java index 8fc64d10a2952..2748c0b581a88 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java @@ -94,6 +94,8 @@ public void testNodeHealthScript() throws Exception { String timeOutScript = Shell.WINDOWS ? "@echo off\nping -n 4 127.0.0.1 >nul\necho \"I am fine\"" : "sleep 4\necho \"I am fine\""; + String exitCodeScript = "exit 127"; + Configuration conf = new Configuration(); writeNodeHealthScriptFile(normalScript, true); NodeHealthScriptRunner nodeHealthScriptRunner = new NodeHealthScriptRunner( @@ -132,5 +134,12 @@ public void testNodeHealthScript() throws Exception { Assert.assertEquals( NodeHealthScriptRunner.NODE_HEALTH_SCRIPT_TIMED_OUT_MSG, nodeHealthScriptRunner.getHealthReport()); + + // Exit code 127 + writeNodeHealthScriptFile(exitCodeScript, true); + timerTask.run(); + Assert.assertTrue("Node health status reported unhealthy", + nodeHealthScriptRunner.isHealthy()); + Assert.assertEquals("", nodeHealthScriptRunner.getHealthReport()); } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestProtoUtil.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestProtoUtil.java index ab891b8f200d6..6b72089faab84 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestProtoUtil.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestProtoUtil.java @@ -69,7 +69,7 @@ public void testVarInt() throws IOException { private void doVarIntTest(int value) throws IOException { ByteArrayOutputStream baos = new ByteArrayOutputStream(); CodedOutputStream cout = CodedOutputStream.newInstance(baos); - cout.writeRawVarint32(value); + cout.writeUInt32NoTag(value); cout.flush(); DataInputStream dis = new DataInputStream( diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShutdownHookManager.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShutdownHookManager.java index 03fa903170f2f..59430e8bd9453 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShutdownHookManager.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShutdownHookManager.java @@ -21,7 +21,6 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; -import org.junit.After; import org.junit.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -43,13 +42,10 @@ public class TestShutdownHookManager { LoggerFactory.getLogger(TestShutdownHookManager.class.getName()); /** - * remove all the shutdown hooks so that they never get invoked later - * on in this test process. + * A new instance of ShutdownHookManager to ensure parallel tests + * don't have shared context. */ - @After - public void clearShutdownHooks() { - ShutdownHookManager.get().clearShutdownHooks(); - } + private final ShutdownHookManager mgr = new ShutdownHookManager(); /** * Verify hook registration, then execute the hook callback stage @@ -58,7 +54,6 @@ public void clearShutdownHooks() { */ @Test public void shutdownHookManager() { - ShutdownHookManager mgr = ShutdownHookManager.get(); assertNotNull("No ShutdownHookManager", mgr); assertEquals(0, mgr.getShutdownHooksInOrder().size()); Hook hook1 = new Hook("hook1", 0, false); @@ -119,7 +114,7 @@ public void shutdownHookManager() { // now execute the hook shutdown sequence INVOCATION_COUNT.set(0); LOG.info("invoking executeShutdown()"); - int timeouts = ShutdownHookManager.executeShutdown(); + int timeouts = mgr.executeShutdown(); LOG.info("Shutdown completed"); assertEquals("Number of timed out hooks", 1, timeouts); @@ -193,7 +188,6 @@ public void testShutdownTimeoutBadConfiguration() throws Throwable { */ @Test public void testDuplicateRegistration() throws Throwable { - ShutdownHookManager mgr = ShutdownHookManager.get(); Hook hook = new Hook("hook1", 0, false); // add the hook @@ -222,6 +216,21 @@ public void testDuplicateRegistration() throws Throwable { } + @Test + public void testShutdownRemove() throws Throwable { + assertNotNull("No ShutdownHookManager", mgr); + assertEquals(0, mgr.getShutdownHooksInOrder().size()); + Hook hook1 = new Hook("hook1", 0, false); + Hook hook2 = new Hook("hook2", 0, false); + mgr.addShutdownHook(hook1, 9); // create Hook1 with priority 9 + assertTrue("No hook1", mgr.hasShutdownHook(hook1)); // hook1 lookup works + assertEquals(1, mgr.getShutdownHooksInOrder().size()); // 1 hook + assertFalse("Delete hook2 should not be allowed", + mgr.removeShutdownHook(hook2)); + assertTrue("Can't delete hook1", mgr.removeShutdownHook(hook1)); + assertEquals(0, mgr.getShutdownHooksInOrder().size()); + } + private static final AtomicInteger INVOCATION_COUNT = new AtomicInteger(); /** diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestChildReaper.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestChildReaper.java index 11b254fc697eb..960471841948c 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestChildReaper.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestChildReaper.java @@ -26,13 +26,14 @@ import org.apache.curator.test.Timing; import org.apache.zookeeper.data.Stat; import org.junit.After; -import org.junit.Assert; import org.junit.Before; import org.junit.Test; import java.net.BindException; import java.util.Random; +import static org.assertj.core.api.Assertions.assertThat; + /** * This is a copy of Curator 2.7.1's TestChildReaper class, with minor * modifications to make it work with JUnit (some setup code taken from @@ -90,7 +91,7 @@ public void testSomeNodes() throws Exception timing.forWaiting().sleepABit(); Stat stat = client.checkExists().forPath("/test"); - Assert.assertEquals(stat.getNumChildren(), nonEmptyNodes); + assertThat(stat.getNumChildren()).isEqualTo(nonEmptyNodes); } finally { @@ -120,7 +121,7 @@ public void testSimple() throws Exception timing.forWaiting().sleepABit(); Stat stat = client.checkExists().forPath("/test"); - Assert.assertEquals(stat.getNumChildren(), 0); + assertThat(stat.getNumChildren()).isZero(); } finally { @@ -153,11 +154,11 @@ public void testMultiPath() throws Exception timing.forWaiting().sleepABit(); Stat stat = client.checkExists().forPath("/test1"); - Assert.assertEquals(stat.getNumChildren(), 0); + assertThat(stat.getNumChildren()).isZero(); stat = client.checkExists().forPath("/test2"); - Assert.assertEquals(stat.getNumChildren(), 0); + assertThat(stat.getNumChildren()).isZero(); stat = client.checkExists().forPath("/test3"); - Assert.assertEquals(stat.getNumChildren(), 10); + assertThat(stat.getNumChildren()).isEqualTo(10); } finally { @@ -193,11 +194,11 @@ public void testNamespace() throws Exception timing.forWaiting().sleepABit(); Stat stat = client.checkExists().forPath("/test"); - Assert.assertEquals(stat.getNumChildren(), 0); + assertThat(stat.getNumChildren()).isZero(); stat = client.usingNamespace(null).checkExists().forPath("/foo/test"); - Assert.assertNotNull(stat); - Assert.assertEquals(stat.getNumChildren(), 0); + assertThat(stat).isNotNull(); + assertThat(stat.getNumChildren()).isZero(); } finally { diff --git a/hadoop-common-project/hadoop-common/src/test/proto/test.proto b/hadoop-common-project/hadoop-common/src/test/proto/test.proto index 37e9a0bf7aacd..2c41aa2bc7c67 100644 --- a/hadoop-common-project/hadoop-common/src/test/proto/test.proto +++ b/hadoop-common-project/hadoop-common/src/test/proto/test.proto @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ - +syntax = "proto2"; option java_package = "org.apache.hadoop.ipc.protobuf"; option java_outer_classname = "TestProtos"; option java_generate_equals_and_hash = true; diff --git a/hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto b/hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto index 3746411c90b3f..f699027914061 100644 --- a/hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto +++ b/hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto @@ -15,6 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ +syntax = "proto2"; option java_package = "org.apache.hadoop.ipc.protobuf"; option java_outer_classname = "TestRpcServiceProtos"; option java_generic_services = true; @@ -39,6 +40,7 @@ service TestProtobufRpcProto { rpc testServerGet(EmptyRequestProto) returns (EmptyResponseProto); rpc exchange(ExchangeRequestProto) returns (ExchangeResponseProto); rpc sleep(SleepRequestProto) returns (EmptyResponseProto); + rpc lockAndSleep(SleepRequestProto) returns (EmptyResponseProto); rpc getAuthMethod(EmptyRequestProto) returns (AuthMethodResponseProto); rpc getAuthUser(EmptyRequestProto) returns (UserResponseProto); rpc echoPostponed(EchoRequestProto) returns (EchoResponseProto); diff --git a/hadoop-common-project/hadoop-common/src/test/resources/kdc/killKdc.sh b/hadoop-common-project/hadoop-common/src/test/resources/kdc/killKdc.sh deleted file mode 100644 index a6a3d77a3e570..0000000000000 --- a/hadoop-common-project/hadoop-common/src/test/resources/kdc/killKdc.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/sh -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -ps -ef | grep apacheds | grep -v grep | awk '{printf $2"\n"}' | xargs -t --no-run-if-empty kill -9 - diff --git a/hadoop-common-project/hadoop-common/src/test/resources/kdc/ldif/users.ldif b/hadoop-common-project/hadoop-common/src/test/resources/kdc/ldif/users.ldif deleted file mode 100644 index a3d2704949c44..0000000000000 --- a/hadoop-common-project/hadoop-common/src/test/resources/kdc/ldif/users.ldif +++ /dev/null @@ -1,78 +0,0 @@ -dn: dc=example,dc=com -objectClass: dcObject -objectClass: organization -objectClass: top -dc: example -o: example.com - -dn: ou=Users,dc=example,dc=com -objectClass: organizationalUnit -objectClass: top -ou: Users - -dn: uid=user1,ou=Users,dc=example,dc=com -objectClass: top -objectClass: person -objectClass: inetOrgPerson -objectClass: krb5principal -objectClass: krb5kdcentry -cn: user1 Service -sn: Service -uid: user1 -userPassword: secret -krb5PrincipalName: user1@EXAMPLE.COM -krb5KeyVersionNumber: 0 - -dn: uid=krbtgt,ou=Users,dc=example,dc=com -objectClass: top -objectClass: person -objectClass: inetOrgPerson -objectClass: krb5principal -objectClass: krb5kdcentry -cn: KDC Service -sn: Service -uid: krbtgt -userPassword: secret -krb5PrincipalName: krbtgt/EXAMPLE.COM@EXAMPLE.COM -krb5KeyVersionNumber: 0 - -dn: uid=ldap,ou=Users,dc=example,dc=com -objectClass: top -objectClass: person -objectClass: inetOrgPerson -objectClass: krb5principal -objectClass: krb5kdcentry -cn: LDAP -sn: Service -uid: ldap -userPassword: randall -krb5PrincipalName: ldap/localhost@EXAMPLE.COM -krb5KeyVersionNumber: 0 - -dn: uid=nn1,ou=Users,dc=example,dc=com -objectClass: top -objectClass: person -objectClass: inetOrgPerson -objectClass: krb5principal -objectClass: krb5kdcentry -cn: NameNode Service -sn: Service -uid: nn1 -userPassword: secret -krb5PrincipalName: nn1/localhost@EXAMPLE.COM -krb5KeyVersionNumber: 0 - -dn: uid=dn1,ou=Users,dc=example,dc=com -objectClass: top -objectClass: person -objectClass: inetOrgPerson -objectClass: krb5principal -objectClass: krb5kdcentry -cn: DataNode Service -sn: Service -uid: dn1 -userPassword: secret -krb5PrincipalName: dn1/localhost@EXAMPLE.COM -krb5KeyVersionNumber: 0 - - diff --git a/hadoop-common-project/hadoop-common/src/test/resources/kdc/server.xml b/hadoop-common-project/hadoop-common/src/test/resources/kdc/server.xml deleted file mode 100644 index bb8c52a9976ad..0000000000000 --- a/hadoop-common-project/hadoop-common/src/test/resources/kdc/server.xml +++ /dev/null @@ -1,258 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - #directoryService - - - - - - - - - - - - - - - - - - #directoryService - - - - - - - - - - - - - - example.com - apache.org - - - - - - - - - - - - - - #ldapServer - - - - - diff --git a/hadoop-common-project/hadoop-common/src/test/resources/testConf.xml b/hadoop-common-project/hadoop-common/src/test/resources/testConf.xml index e38c2592f8573..392d39170d5fe 100644 --- a/hadoop-common-project/hadoop-common/src/test/resources/testConf.xml +++ b/hadoop-common-project/hadoop-common/src/test/resources/testConf.xml @@ -923,7 +923,7 @@ RegexpComparator - ^-test -\[defsz\] <path> :\s* + ^-test -\[defswrz\] <path> :\s* RegexpComparator @@ -931,7 +931,7 @@ RegexpComparator - ^\s*-[defsz]\s+return 0 if .* + ^\s*-[defswrz]\s+return 0 if .* diff --git a/hadoop-common-project/hadoop-kms/pom.xml b/hadoop-common-project/hadoop-kms/pom.xml index 0eed09a256fa4..6f4ff09952ac4 100644 --- a/hadoop-common-project/hadoop-kms/pom.xml +++ b/hadoop-common-project/hadoop-kms/pom.xml @@ -17,7 +17,7 @@ + https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KeyAuthorizationKeyProvider.java b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KeyAuthorizationKeyProvider.java index 868db765c2edd..101591b0310d2 100644 --- a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KeyAuthorizationKeyProvider.java +++ b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KeyAuthorizationKeyProvider.java @@ -402,7 +402,7 @@ protected KeyProvider getKeyProvider() { @Override public String toString() { - return provider.toString(); + return this.getClass().getName()+": "+provider.toString(); } } diff --git a/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSMDCFilter.java b/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSMDCFilter.java new file mode 100644 index 0000000000000..42d1dc0672ea0 --- /dev/null +++ b/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSMDCFilter.java @@ -0,0 +1,88 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.crypto.key.kms.server; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; +import static org.mockito.Mockito.when; + +import java.io.IOException; + +import javax.servlet.FilterChain; +import javax.servlet.ServletException; +import javax.servlet.ServletRequest; +import javax.servlet.ServletResponse; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; + +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mockito; + +/** + * Test for {@link KMSMDCFilter}. + * + */ +public class TestKMSMDCFilter { + + private static final String REMOTE_ADDRESS = "192.168.100.100"; + private static final String URL = "/admin"; + private static final String METHOD = "GET"; + + private KMSMDCFilter filter; + private HttpServletRequest httpRequest; + private HttpServletResponse httpResponse; + + @Before + public void setUp() throws IOException { + filter = new KMSMDCFilter(); + httpRequest = Mockito.mock(HttpServletRequest.class); + httpResponse = Mockito.mock(HttpServletResponse.class); + KMSMDCFilter.setContext(null, null, null, null); + } + + @Test + public void testFilter() throws IOException, ServletException { + when(httpRequest.getMethod()).thenReturn(METHOD); + when(httpRequest.getRequestURL()).thenReturn(new StringBuffer(URL)); + when(httpRequest.getRemoteAddr()).thenReturn(REMOTE_ADDRESS); + + FilterChain filterChain = new FilterChain() { + @Override + public void doFilter(ServletRequest request, ServletResponse response) + throws IOException, ServletException { + assertEquals("filter.remoteClientAddress", REMOTE_ADDRESS, + KMSMDCFilter.getRemoteClientAddress()); + assertEquals("filter.method", METHOD, KMSMDCFilter.getMethod()); + assertEquals("filter.url", URL, KMSMDCFilter.getURL()); + } + }; + + checkMDCValuesAreEmpty(); + filter.doFilter(httpRequest, httpResponse, filterChain); + checkMDCValuesAreEmpty(); + } + + private void checkMDCValuesAreEmpty() { + assertNull("getRemoteClientAddress", KMSMDCFilter.getRemoteClientAddress()); + assertNull("getMethod", KMSMDCFilter.getMethod()); + assertNull("getURL", KMSMDCFilter.getURL()); + assertNull("getUgi", KMSMDCFilter.getUgi()); + } + +} diff --git a/hadoop-common-project/hadoop-minikdc/pom.xml b/hadoop-common-project/hadoop-minikdc/pom.xml index f1566f1f1c69b..adbd6e32bee58 100644 --- a/hadoop-common-project/hadoop-minikdc/pom.xml +++ b/hadoop-common-project/hadoop-minikdc/pom.xml @@ -14,7 +14,7 @@ --> + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> org.apache.hadoop hadoop-project diff --git a/hadoop-common-project/hadoop-nfs/pom.xml b/hadoop-common-project/hadoop-nfs/pom.xml index fb3fe68ba9d42..e0fedaf1434e6 100644 --- a/hadoop-common-project/hadoop-nfs/pom.xml +++ b/hadoop-common-project/hadoop-nfs/pom.xml @@ -15,7 +15,7 @@ + https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -97,6 +97,11 @@ com.google.guava guava + + org.assertj + assertj-core + test + diff --git a/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestFileHandle.java b/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestFileHandle.java index 53916641e7ee5..70875c2913dd3 100644 --- a/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestFileHandle.java +++ b/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestFileHandle.java @@ -17,24 +17,23 @@ */ package org.apache.hadoop.nfs.nfs3; -import org.junit.Assert; - -import org.apache.hadoop.nfs.nfs3.FileHandle; import org.apache.hadoop.oncrpc.XDR; import org.junit.Test; +import static org.assertj.core.api.Assertions.assertThat; + public class TestFileHandle { @Test public void testConstructor() { FileHandle handle = new FileHandle(1024); XDR xdr = new XDR(); handle.serialize(xdr); - Assert.assertEquals(handle.getFileId(), 1024); + assertThat(handle.getFileId()).isEqualTo(1024); // Deserialize it back FileHandle handle2 = new FileHandle(); handle2.deserialize(xdr.asReadOnlyWrap()); - Assert.assertEquals("Failed: Assert 1024 is id ", 1024, - handle.getFileId()); + assertThat(handle.getFileId()) + .withFailMessage("Failed: Assert 1024 is id ").isEqualTo(1024); } } diff --git a/hadoop-common-project/hadoop-registry/pom.xml b/hadoop-common-project/hadoop-registry/pom.xml index 7ca1c9e7a8a8c..dc45309dca296 100644 --- a/hadoop-common-project/hadoop-registry/pom.xml +++ b/hadoop-common-project/hadoop-registry/pom.xml @@ -15,7 +15,7 @@ + https://maven.apache.org/xsd/maven-4.0.0.xsd"> hadoop-project org.apache.hadoop diff --git a/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java b/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java index dac11353a4aee..c3cb021fb53f3 100644 --- a/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java +++ b/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java @@ -32,6 +32,7 @@ import org.apache.hadoop.util.ZKUtil; import org.apache.zookeeper.Environment; import org.apache.zookeeper.ZooDefs; +import org.apache.zookeeper.client.ZKClientConfig; import org.apache.zookeeper.client.ZooKeeperSaslClient; import org.apache.zookeeper.data.ACL; import org.apache.zookeeper.data.Id; @@ -769,19 +770,19 @@ public void applySecurityEnvironment(CuratorFrameworkFactory.Builder JaasConfiguration jconf = new JaasConfiguration(jaasClientEntry, principal, keytab); javax.security.auth.login.Configuration.setConfiguration(jconf); - setSystemPropertyIfUnset(ZooKeeperSaslClient.ENABLE_CLIENT_SASL_KEY, - "true"); - setSystemPropertyIfUnset(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY, - jaasClientEntry); + setSystemPropertyIfUnset(ZKClientConfig.ENABLE_CLIENT_SASL_KEY, + "true"); + setSystemPropertyIfUnset(ZKClientConfig.LOGIN_CONTEXT_NAME_KEY, + jaasClientEntry); } else { // in this case, jaas config is specified so we will not change it LOG.info("Using existing ZK sasl configuration: " + - "jaasClientEntry = " + System.getProperty( - ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY, "Client") + - ", sasl client = " + System.getProperty( - ZooKeeperSaslClient.ENABLE_CLIENT_SASL_KEY, - ZooKeeperSaslClient.ENABLE_CLIENT_SASL_DEFAULT) + - ", jaas = " + existingJaasConf); + "jaasClientEntry = " + System.getProperty( + ZKClientConfig.LOGIN_CONTEXT_NAME_KEY, "Client") + + ", sasl client = " + System.getProperty( + ZKClientConfig.ENABLE_CLIENT_SASL_KEY, + ZKClientConfig.ENABLE_CLIENT_SASL_DEFAULT) + + ", jaas = " + existingJaasConf); } break; @@ -926,7 +927,7 @@ public void logCurrentHadoopUser() { UserGroupInformation realUser = currentUser.getRealUser(); LOG.info("Real User = {}" , realUser); } catch (IOException e) { - LOG.warn("Failed to get current user {}, {}", e); + LOG.warn("Failed to get current user, {}", e); } } diff --git a/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/ZookeeperConfigOptions.java b/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/ZookeeperConfigOptions.java index edcf0859fc345..115af3237abc0 100644 --- a/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/ZookeeperConfigOptions.java +++ b/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/ZookeeperConfigOptions.java @@ -18,7 +18,7 @@ package org.apache.hadoop.registry.client.impl.zk; -import org.apache.zookeeper.client.ZooKeeperSaslClient; +import org.apache.zookeeper.client.ZKClientConfig; import org.apache.zookeeper.server.ZooKeeperSaslServer; /** @@ -62,10 +62,10 @@ public interface ZookeeperConfigOptions { * *

    * Default value is derived from - * {@link ZooKeeperSaslClient#LOGIN_CONTEXT_NAME_KEY} + * {@link ZKClientConfig#LOGIN_CONTEXT_NAME_KEY} */ String PROP_ZK_SASL_CLIENT_CONTEXT = - ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY; + ZKClientConfig.LOGIN_CONTEXT_NAME_KEY; /** * The SASL client username: {@value}. diff --git a/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java b/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java index b6cf9fc519121..a7e2611b3df9d 100644 --- a/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java +++ b/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java @@ -42,6 +42,7 @@ import java.io.PrintWriter; import java.io.StringWriter; import java.net.InetSocketAddress; +import java.net.ServerSocket; import java.net.UnknownHostException; /** @@ -121,7 +122,7 @@ public InetSocketAddress getConnectionAddress() { * @throws UnknownHostException if the server cannot resolve the host */ private InetSocketAddress getAddress(int port) throws UnknownHostException { - return new InetSocketAddress(host, port < 0 ? 0 : port); + return new InetSocketAddress(host, port <= 0 ? getRandomAvailablePort() : port); } /** @@ -227,10 +228,8 @@ protected void serviceStart() throws Exception { setupSecurity(); - ZooKeeperServer zkServer = new ZooKeeperServer(); FileTxnSnapLog ftxn = new FileTxnSnapLog(dataDir, dataDir); - zkServer.setTxnLogFactory(ftxn); - zkServer.setTickTime(tickTime); + ZooKeeperServer zkServer = new ZooKeeperServer(ftxn, tickTime); LOG.info("Starting Local Zookeeper service"); factory = ServerCnxnFactory.createFactory(); @@ -245,7 +244,7 @@ protected void serviceStart() throws Exception { PrintWriter pw = new PrintWriter(sw); zkServer.dumpConf(pw); pw.flush(); - LOG.debug(sw.toString()); + LOG.debug("ZooKeeper config:\n" + sw.toString()); } binding = new BindingInformation(); binding.ensembleProvider = new FixedEnsembleProvider(connectString); @@ -279,4 +278,20 @@ public BindingInformation supplyBindingInformation() { "Service is not started: binding information undefined"); return binding; } + + /** + * Returns with a random open port can be used to set as server port for ZooKeeper. + * @return a random open port or 0 (in case of error) + */ + private int getRandomAvailablePort() { + port = 0; + try { + final ServerSocket s = new ServerSocket(0); + port = s.getLocalPort(); + s.close(); + } catch (IOException e) { + LOG.warn("ERROR during selecting random port for ZooKeeper server to bind." , e); + } + return port; + } } diff --git a/hadoop-common-project/hadoop-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureRegistry.java b/hadoop-common-project/hadoop-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureRegistry.java index 9d5848ea03497..27d32ea9d2672 100644 --- a/hadoop-common-project/hadoop-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureRegistry.java +++ b/hadoop-common-project/hadoop-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureRegistry.java @@ -24,16 +24,12 @@ import org.apache.hadoop.registry.client.impl.zk.CuratorService; import org.apache.hadoop.registry.client.impl.zk.RegistrySecurity; import org.apache.zookeeper.CreateMode; -import org.apache.zookeeper.Login; -import org.apache.zookeeper.server.ZooKeeperSaslServer; -import org.apache.zookeeper.server.auth.SaslServerCallbackHandler; import org.junit.After; import org.junit.Before; import org.junit.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import javax.security.auth.login.AppConfigurationEntry; import javax.security.auth.login.LoginContext; import static org.apache.hadoop.registry.client.api.RegistryConstants.*; @@ -58,36 +54,6 @@ public void afterTestSecureZKService() throws Throwable { RegistrySecurity.clearZKSaslClientProperties(); } - /** - * this is a cut and paste of some of the ZK internal code that was - * failing on windows and swallowing its exceptions - */ - @Test - public void testLowlevelZKSaslLogin() throws Throwable { - RegistrySecurity.bindZKToServerJAASContext(ZOOKEEPER_SERVER_CONTEXT); - String serverSection = - System.getProperty(ZooKeeperSaslServer.LOGIN_CONTEXT_NAME_KEY, - ZooKeeperSaslServer.DEFAULT_LOGIN_CONTEXT_NAME); - assertEquals(ZOOKEEPER_SERVER_CONTEXT, serverSection); - - AppConfigurationEntry entries[]; - entries = javax.security.auth.login.Configuration.getConfiguration() - .getAppConfigurationEntry( - serverSection); - - assertNotNull("null entries", entries); - - SaslServerCallbackHandler saslServerCallbackHandler = - new SaslServerCallbackHandler( - javax.security.auth.login.Configuration.getConfiguration()); - Login login = new Login(serverSection, saslServerCallbackHandler); - try { - login.startThreadIfNeeded(); - } finally { - login.shutdown(); - } - } - @Test public void testCreateSecureZK() throws Throwable { startSecureZK(); diff --git a/hadoop-common-project/pom.xml b/hadoop-common-project/pom.xml index e96d1ba9e1608..8be2593c21ffd 100644 --- a/hadoop-common-project/pom.xml +++ b/hadoop-common-project/pom.xml @@ -15,7 +15,7 @@ + https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-dist/pom.xml b/hadoop-dist/pom.xml index 1d3db65930845..07aa7b10a8320 100644 --- a/hadoop-dist/pom.xml +++ b/hadoop-dist/pom.xml @@ -15,7 +15,7 @@ + https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop diff --git a/hadoop-hdds/client/pom.xml b/hadoop-hdds/client/pom.xml index 040332e504ddf..673af41aeef0d 100644 --- a/hadoop-hdds/client/pom.xml +++ b/hadoop-hdds/client/pom.xml @@ -15,7 +15,7 @@ +https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -41,4 +41,4 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> - \ No newline at end of file + diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java index a535c9f3f2aa4..04a8a1aaa1db3 100644 --- a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java @@ -31,6 +31,7 @@ import org.apache.hadoop.hdds.protocol.proto.HddsProtos; import org.apache.hadoop.hdds.scm.client.HddsClientUtils; import org.apache.hadoop.hdds.scm.pipeline.Pipeline; +import org.apache.hadoop.hdds.scm.storage.CheckedBiFunction; import org.apache.hadoop.hdds.security.exception.SCMSecurityException; import org.apache.hadoop.hdds.security.x509.SecurityConfig; import org.apache.hadoop.hdds.tracing.GrpcClientInterceptor; @@ -51,8 +52,9 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import java.io.File; import java.io.IOException; +import java.security.cert.X509Certificate; +import java.util.Collections; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -62,13 +64,13 @@ import java.util.concurrent.Semaphore; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; -import java.util.stream.Collectors; /** - * A Client for the storageContainer protocol. + * A Client for the storageContainer protocol for read object data. */ public class XceiverClientGrpc extends XceiverClientSpi { static final Logger LOG = LoggerFactory.getLogger(XceiverClientGrpc.class); + private static final String COMPONENT = "dn"; private final Pipeline pipeline; private final Configuration config; private Map asyncStubs; @@ -77,47 +79,70 @@ public class XceiverClientGrpc extends XceiverClientSpi { private final Semaphore semaphore; private boolean closed = false; private SecurityConfig secConfig; + private final boolean topologyAwareRead; + private X509Certificate caCert; /** * Constructs a client that can communicate with the Container framework on * data nodes. * * @param pipeline - Pipeline that defines the machines. - * @param config -- Ozone Config + * @param config -- Ozone Config + * @param caCert - SCM ca certificate. */ - public XceiverClientGrpc(Pipeline pipeline, Configuration config) { + public XceiverClientGrpc(Pipeline pipeline, Configuration config, + X509Certificate caCert) { super(); Preconditions.checkNotNull(pipeline); Preconditions.checkNotNull(config); this.pipeline = pipeline; this.config = config; - this.secConfig = new SecurityConfig(config); + this.secConfig = new SecurityConfig(config); this.semaphore = new Semaphore(HddsClientUtils.getMaxOutstandingRequests(config)); this.metrics = XceiverClientManager.getXceiverClientMetrics(); this.channels = new HashMap<>(); this.asyncStubs = new HashMap<>(); + this.topologyAwareRead = config.getBoolean( + OzoneConfigKeys.OZONE_NETWORK_TOPOLOGY_AWARE_READ_KEY, + OzoneConfigKeys.OZONE_NETWORK_TOPOLOGY_AWARE_READ_DEFAULT); + this.caCert = caCert; + } + + /** + * Constructs a client that can communicate with the Container framework on + * data nodes. + * + * @param pipeline - Pipeline that defines the machines. + * @param config -- Ozone Config + */ + public XceiverClientGrpc(Pipeline pipeline, Configuration config) { + this(pipeline, config, null); } /** * To be used when grpc token is not enabled. - * */ + */ @Override public void connect() throws Exception { - // leader by default is the 1st datanode in the datanode list of pipleline - DatanodeDetails dn = this.pipeline.getFirstNode(); - // just make a connection to the 1st datanode at the beginning + // connect to the closest node, if closest node doesn't exist, delegate to + // first node, which is usually the leader in the pipeline. + DatanodeDetails dn = topologyAwareRead ? this.pipeline.getClosestNode() : + this.pipeline.getFirstNode(); + // just make a connection to the picked datanode at the beginning connectToDatanode(dn, null); } /** * Passed encoded token to GRPC header when security is enabled. - * */ + */ @Override public void connect(String encodedToken) throws Exception { - // leader by default is the 1st datanode in the datanode list of pipleline - DatanodeDetails dn = this.pipeline.getFirstNode(); - // just make a connection to the 1st datanode at the beginning + // connect to the closest node, if closest node doesn't exist, delegate to + // first node, which is usually the leader in the pipeline. + DatanodeDetails dn = topologyAwareRead ? this.pipeline.getClosestNode() : + this.pipeline.getFirstNode(); + // just make a connection to the picked datanode at the beginning connectToDatanode(dn, encodedToken); } @@ -132,28 +157,21 @@ private void connectToDatanode(DatanodeDetails dn, String encodedToken) } // Add credential context to the client call - String userName = UserGroupInformation.getCurrentUser() - .getShortUserName(); - LOG.debug("Connecting to server Port : " + dn.getIpAddress()); - NettyChannelBuilder channelBuilder = NettyChannelBuilder.forAddress(dn - .getIpAddress(), port).usePlaintext() + String userName = UserGroupInformation.getCurrentUser().getShortUserName(); + if (LOG.isDebugEnabled()) { + LOG.debug("Nodes in pipeline : {}", pipeline.getNodes().toString()); + LOG.debug("Connecting to server : {}", dn.getIpAddress()); + } + NettyChannelBuilder channelBuilder = + NettyChannelBuilder.forAddress(dn.getIpAddress(), port).usePlaintext() .maxInboundMessageSize(OzoneConsts.OZONE_SCM_CHUNK_MAX_SIZE) .intercept(new ClientCredentialInterceptor(userName, encodedToken), new GrpcClientInterceptor()); if (secConfig.isGrpcTlsEnabled()) { - File trustCertCollectionFile = secConfig.getTrustStoreFile(); - File privateKeyFile = secConfig.getClientPrivateKeyFile(); - File clientCertChainFile = secConfig.getClientCertChainFile(); - SslContextBuilder sslContextBuilder = GrpcSslContexts.forClient(); - if (trustCertCollectionFile != null) { - sslContextBuilder.trustManager(trustCertCollectionFile); - } - if (secConfig.isGrpcMutualTlsRequired() && clientCertChainFile != null && - privateKeyFile != null) { - sslContextBuilder.keyManager(clientCertChainFile, privateKeyFile); + if (caCert != null) { + sslContextBuilder.trustManager(caCert); } - if (secConfig.useTestCert()) { channelBuilder.overrideAuthority("localhost"); } @@ -216,67 +234,85 @@ public ContainerCommandResponseProto sendCommand( } @Override - public XceiverClientReply sendCommand( - ContainerCommandRequestProto request, List excludeDns) + public ContainerCommandResponseProto sendCommand( + ContainerCommandRequestProto request, List validators) throws IOException { - Preconditions.checkState(HddsUtils.isReadOnly(request)); - return sendCommandWithTraceIDAndRetry(request, excludeDns); + try { + XceiverClientReply reply; + reply = sendCommandWithTraceIDAndRetry(request, validators); + ContainerCommandResponseProto responseProto = reply.getResponse().get(); + return responseProto; + } catch (ExecutionException | InterruptedException e) { + throw new IOException("Failed to execute command " + request, e); + } } private XceiverClientReply sendCommandWithTraceIDAndRetry( - ContainerCommandRequestProto request, List excludeDns) + ContainerCommandRequestProto request, List validators) throws IOException { try (Scope scope = GlobalTracer.get() .buildSpan("XceiverClientGrpc." + request.getCmdType().name()) .startActive(true)) { ContainerCommandRequestProto finalPayload = ContainerCommandRequestProto.newBuilder(request) - .setTraceID(TracingUtil.exportCurrentSpan()) - .build(); - return sendCommandWithRetry(finalPayload, excludeDns); + .setTraceID(TracingUtil.exportCurrentSpan()).build(); + return sendCommandWithRetry(finalPayload, validators); } } private XceiverClientReply sendCommandWithRetry( - ContainerCommandRequestProto request, List excludeDns) + ContainerCommandRequestProto request, List validators) throws IOException { ContainerCommandResponseProto responseProto = null; + IOException ioException = null; // In case of an exception or an error, we will try to read from the // datanodes in the pipeline in a round robin fashion. // TODO: cache the correct leader info in here, so that any subsequent calls // should first go to leader - List dns = pipeline.getNodes(); - List healthyDns = - excludeDns != null ? dns.stream().filter(dnId -> { - for (DatanodeDetails excludeId : excludeDns) { - if (dnId.equals(excludeId)) { - return false; - } - } - return true; - }).collect(Collectors.toList()) : dns; XceiverClientReply reply = new XceiverClientReply(null); - for (DatanodeDetails dn : healthyDns) { + List datanodeList; + if ((request.getCmdType() == ContainerProtos.Type.ReadChunk || + request.getCmdType() == ContainerProtos.Type.GetSmallFile) && + topologyAwareRead) { + datanodeList = pipeline.getNodesInOrder(); + } else { + datanodeList = pipeline.getNodes(); + // Shuffle datanode list so that clients do not read in the same order + // every time. + Collections.shuffle(datanodeList); + } + for (DatanodeDetails dn : datanodeList) { try { - LOG.debug("Executing command " + request + " on datanode " + dn); + if (LOG.isDebugEnabled()) { + LOG.debug("Executing command " + request + " on datanode " + dn); + } // In case the command gets retried on a 2nd datanode, // sendCommandAsyncCall will create a new channel and async stub // in case these don't exist for the specific datanode. reply.addDatanode(dn); responseProto = sendCommandAsync(request, dn).getResponse().get(); - if (responseProto.getResult() == ContainerProtos.Result.SUCCESS) { - break; + if (validators != null && !validators.isEmpty()) { + for (CheckedBiFunction validator : validators) { + validator.apply(request, responseProto); + } } - } catch (ExecutionException | InterruptedException e) { - LOG.debug("Failed to execute command " + request + " on datanode " + dn + break; + } catch (ExecutionException | InterruptedException | IOException e) { + LOG.error("Failed to execute command " + request + " on datanode " + dn .getUuidString(), e); - if (Status.fromThrowable(e.getCause()).getCode() - == Status.UNAUTHENTICATED.getCode()) { - throw new SCMSecurityException("Failed to authenticate with " - + "GRPC XceiverServer with Ozone block token."); + if (!(e instanceof IOException)) { + if (Status.fromThrowable(e.getCause()).getCode() + == Status.UNAUTHENTICATED.getCode()) { + throw new SCMSecurityException("Failed to authenticate with " + + "GRPC XceiverServer with Ozone block token."); + } + ioException = new IOException(e); + } else { + ioException = (IOException) e; } + responseProto = null; } } @@ -284,9 +320,10 @@ private XceiverClientReply sendCommandWithRetry( reply.setResponse(CompletableFuture.completedFuture(responseProto)); return reply; } else { - throw new IOException( - "Failed to execute command " + request + " on the pipeline " - + pipeline.getId()); + Preconditions.checkNotNull(ioException); + LOG.error("Failed to execute command {} on the pipeline {}.", request, + pipeline); + throw ioException; } } @@ -311,8 +348,13 @@ public XceiverClientReply sendCommandAsync( try (Scope scope = GlobalTracer.get() .buildSpan("XceiverClientGrpc." + request.getCmdType().name()) .startActive(true)) { + + ContainerCommandRequestProto finalPayload = + ContainerCommandRequestProto.newBuilder(request) + .setTraceID(TracingUtil.exportCurrentSpan()) + .build(); XceiverClientReply asyncReply = - sendCommandAsync(request, pipeline.getFirstNode()); + sendCommandAsync(finalPayload, pipeline.getFirstNode()); // TODO : for now make this API sync in nature as async requests are // served out of order over XceiverClientGrpc. This needs to be fixed // if this API is to be used for I/O path. Currently, this is not @@ -339,7 +381,10 @@ private XceiverClientReply sendCommandAsync( if (!isConnected(channel)) { reconnect(dn, token); } - + if (LOG.isDebugEnabled()) { + LOG.debug("Send command {} to datanode {}", + request.getCmdType().toString(), dn.getNetworkFullPath()); + } final CompletableFuture replyFuture = new CompletableFuture<>(); semaphore.acquire(); diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientManager.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientManager.java index f9b5e6d05cb8f..b15828a153098 100644 --- a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientManager.java +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientManager.java @@ -25,24 +25,33 @@ import com.google.common.cache.RemovalListener; import com.google.common.cache.RemovalNotification; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hdds.conf.Config; +import org.apache.hadoop.hdds.conf.ConfigGroup; +import org.apache.hadoop.hdds.conf.ConfigType; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdds.scm.pipeline.Pipeline; import org.apache.hadoop.hdds.protocol.proto.HddsProtos; +import org.apache.hadoop.hdds.security.exception.SCMSecurityException; +import org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec; +import org.apache.hadoop.ozone.OzoneConfigKeys; import org.apache.hadoop.ozone.OzoneSecurityUtil; import org.apache.hadoop.security.UserGroupInformation; +import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import java.io.Closeable; import java.io.IOException; +import java.nio.ByteBuffer; +import java.security.cert.CertificateException; +import java.security.cert.X509Certificate; import java.util.concurrent.Callable; import java.util.concurrent.TimeUnit; +import java.util.function.Function; -import static org.apache.hadoop.hdds.scm.ScmConfigKeys - .SCM_CONTAINER_CLIENT_MAX_SIZE_DEFAULT; -import static org.apache.hadoop.hdds.scm.ScmConfigKeys - .SCM_CONTAINER_CLIENT_MAX_SIZE_KEY; -import static org.apache.hadoop.hdds.scm.ScmConfigKeys - .SCM_CONTAINER_CLIENT_STALE_THRESHOLD_DEFAULT; -import static org.apache.hadoop.hdds.scm.ScmConfigKeys - .SCM_CONTAINER_CLIENT_STALE_THRESHOLD_KEY; +import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static org.apache.hadoop.hdds.conf.ConfigTag.OZONE; +import static org.apache.hadoop.hdds.conf.ConfigTag.PERFORMANCE; /** * XceiverClientManager is responsible for the lifecycle of XceiverClient @@ -57,34 +66,52 @@ * not being used for a period of time. */ public class XceiverClientManager implements Closeable { - + private static final Logger LOG = + LoggerFactory.getLogger(XceiverClientManager.class); //TODO : change this to SCM configuration class private final Configuration conf; private final Cache clientCache; private final boolean useRatis; + private X509Certificate caCert; private static XceiverClientMetrics metrics; private boolean isSecurityEnabled; + private final boolean topologyAwareRead; /** - * Creates a new XceiverClientManager. + * Creates a new XceiverClientManager for non secured ozone cluster. + * For security enabled ozone cluster, client should use the other constructor + * with a valid ca certificate in pem string format. * * @param conf configuration */ - public XceiverClientManager(Configuration conf) { + public XceiverClientManager(Configuration conf) throws IOException { + this(conf, OzoneConfiguration.of(conf).getObject(ScmClientConfig.class), + null); + } + + public XceiverClientManager(Configuration conf, ScmClientConfig clientConf, + String caCertPem) throws IOException { + Preconditions.checkNotNull(clientConf); Preconditions.checkNotNull(conf); - int maxSize = conf.getInt(SCM_CONTAINER_CLIENT_MAX_SIZE_KEY, - SCM_CONTAINER_CLIENT_MAX_SIZE_DEFAULT); - long staleThresholdMs = conf.getTimeDuration( - SCM_CONTAINER_CLIENT_STALE_THRESHOLD_KEY, - SCM_CONTAINER_CLIENT_STALE_THRESHOLD_DEFAULT, TimeUnit.MILLISECONDS); + long staleThresholdMs = clientConf.getStaleThreshold(MILLISECONDS); this.useRatis = conf.getBoolean( ScmConfigKeys.DFS_CONTAINER_RATIS_ENABLED_KEY, ScmConfigKeys.DFS_CONTAINER_RATIS_ENABLED_DEFAULT); this.conf = conf; this.isSecurityEnabled = OzoneSecurityUtil.isSecurityEnabled(conf); + if (isSecurityEnabled) { + Preconditions.checkNotNull(caCertPem); + try { + this.caCert = CertificateCodec.getX509Cert(caCertPem); + } catch (CertificateException ex) { + throw new SCMSecurityException("Error: Fail to get SCM CA certificate", + ex); + } + } + this.clientCache = CacheBuilder.newBuilder() - .expireAfterAccess(staleThresholdMs, TimeUnit.MILLISECONDS) - .maximumSize(maxSize) + .expireAfterAccess(staleThresholdMs, MILLISECONDS) + .maximumSize(clientConf.getMaxSize()) .removalListener( new RemovalListener() { @Override @@ -98,6 +125,9 @@ public void onRemoval( } } }).build(); + topologyAwareRead = conf.getBoolean( + OzoneConfigKeys.OZONE_NETWORK_TOPOLOGY_AWARE_READ_KEY, + OzoneConfigKeys.OZONE_NETWORK_TOPOLOGY_AWARE_READ_DEFAULT); } @VisibleForTesting @@ -118,12 +148,32 @@ public Cache getClientCache() { */ public XceiverClientSpi acquireClient(Pipeline pipeline) throws IOException { + return acquireClient(pipeline, false); + } + + /** + * Acquires a XceiverClientSpi connected to a container for read. + * + * If there is already a cached XceiverClientSpi, simply return + * the cached otherwise create a new one. + * + * @param pipeline the container pipeline for the client connection + * @return XceiverClientSpi connected to a container + * @throws IOException if a XceiverClientSpi cannot be acquired + */ + public XceiverClientSpi acquireClientForReadData(Pipeline pipeline) + throws IOException { + return acquireClient(pipeline, true); + } + + private XceiverClientSpi acquireClient(Pipeline pipeline, boolean read) + throws IOException { Preconditions.checkNotNull(pipeline); Preconditions.checkArgument(pipeline.getNodes() != null); Preconditions.checkArgument(!pipeline.getNodes().isEmpty()); synchronized (clientCache) { - XceiverClientSpi info = getClient(pipeline); + XceiverClientSpi info = getClient(pipeline, read); info.incrementReference(); return info; } @@ -136,12 +186,28 @@ public XceiverClientSpi acquireClient(Pipeline pipeline) * @param invalidateClient if true, invalidates the client in cache */ public void releaseClient(XceiverClientSpi client, boolean invalidateClient) { + releaseClient(client, invalidateClient, false); + } + + /** + * Releases a read XceiverClientSpi after use. + * + * @param client client to release + * @param invalidateClient if true, invalidates the client in cache + */ + public void releaseClientForReadData(XceiverClientSpi client, + boolean invalidateClient) { + releaseClient(client, invalidateClient, true); + } + + private void releaseClient(XceiverClientSpi client, boolean invalidateClient, + boolean read) { Preconditions.checkNotNull(client); synchronized (clientCache) { client.decrementReference(); if (invalidateClient) { Pipeline pipeline = client.getPipeline(); - String key = pipeline.getId().getId().toString() + pipeline.getType(); + String key = getPipelineCacheKey(pipeline, read); XceiverClientSpi cachedClient = clientCache.getIfPresent(key); if (cachedClient == client) { clientCache.invalidate(key); @@ -150,11 +216,13 @@ public void releaseClient(XceiverClientSpi client, boolean invalidateClient) { } } - private XceiverClientSpi getClient(Pipeline pipeline) + private XceiverClientSpi getClient(Pipeline pipeline, boolean forRead) throws IOException { HddsProtos.ReplicationType type = pipeline.getType(); try { - String key = pipeline.getId().getId().toString() + type; + // create different client for read different pipeline node based on + // network topology + String key = getPipelineCacheKey(pipeline, forRead); // Append user short name to key to prevent a different user // from using same instance of xceiverClient. key = isSecurityEnabled ? @@ -165,11 +233,12 @@ public XceiverClientSpi call() throws Exception { XceiverClientSpi client = null; switch (type) { case RATIS: - client = XceiverClientRatis.newXceiverClientRatis(pipeline, conf); + client = XceiverClientRatis.newXceiverClientRatis(pipeline, conf, + caCert); client.connect(); break; case STAND_ALONE: - client = new XceiverClientGrpc(pipeline, conf); + client = new XceiverClientGrpc(pipeline, conf, caCert); break; case CHAINED: default: @@ -184,6 +253,19 @@ public XceiverClientSpi call() throws Exception { } } + private String getPipelineCacheKey(Pipeline pipeline, boolean forRead) { + String key = pipeline.getId().getId().toString() + pipeline.getType(); + if (topologyAwareRead && forRead) { + try { + key += pipeline.getClosestNode().getHostName(); + } catch (IOException e) { + LOG.error("Failed to get closest node to create pipeline cache key:" + + e.getMessage()); + } + } + return key; + } + /** * Close and remove all the cached clients. */ @@ -230,6 +312,10 @@ public HddsProtos.ReplicationType getType() { return HddsProtos.ReplicationType.STAND_ALONE; } + public Function byteBufferToByteStringConversion(){ + return ByteStringConversion.createByteBufferConversion(conf); + } + /** * Get xceiver client metric. */ @@ -240,4 +326,65 @@ public synchronized static XceiverClientMetrics getXceiverClientMetrics() { return metrics; } + + /** + * Configuration for HDDS client. + */ + @ConfigGroup(prefix = "scm.container.client") + public static class ScmClientConfig { + + private int maxSize; + private long staleThreshold; + private int maxOutstandingRequests; + + public long getStaleThreshold(TimeUnit unit) { + return unit.convert(staleThreshold, MILLISECONDS); + } + + @Config(key = "idle.threshold", + type = ConfigType.TIME, timeUnit = MILLISECONDS, + defaultValue = "10s", + tags = { OZONE, PERFORMANCE }, + description = + "In the standalone pipelines, the SCM clients use netty to " + + " communicate with the container. It also uses connection pooling" + + " to reduce client side overheads. This allows a connection to" + + " stay idle for a while before the connection is closed." + ) + public void setStaleThreshold(long staleThreshold) { + this.staleThreshold = staleThreshold; + } + + public int getMaxSize() { + return maxSize; + } + + @Config(key = "max.size", + defaultValue = "256", + tags = { OZONE, PERFORMANCE }, + description = + "Controls the maximum number of connections that are cached via" + + " client connection pooling. If the number of connections" + + " exceed this count, then the oldest idle connection is evicted." + ) + public void setMaxSize(int maxSize) { + this.maxSize = maxSize; + } + + public int getMaxOutstandingRequests() { + return maxOutstandingRequests; + } + + @Config(key = "max.outstanding.requests", + defaultValue = "100", + tags = { OZONE, PERFORMANCE }, + description = + "Controls the maximum number of outstanding async requests that can" + + " be handled by the Standalone as well as Ratis client." + ) + public void setMaxOutstandingRequests(int maxOutstandingRequests) { + this.maxOutstandingRequests = maxOutstandingRequests; + } + } + } diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientMetrics.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientMetrics.java index 6c40921c174a0..5d43c5ef22585 100644 --- a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientMetrics.java +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientMetrics.java @@ -69,6 +69,7 @@ public XceiverClientMetrics() { } public static XceiverClientMetrics create() { + DefaultMetricsSystem.initialize(SOURCE_NAME); MetricsSystem ms = DefaultMetricsSystem.instance(); return ms.register(SOURCE_NAME, "Storage Container Client Metrics", new XceiverClientMetrics()); diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientRatis.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientRatis.java index efd82bce7bbd5..04fababf50447 100644 --- a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientRatis.java +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientRatis.java @@ -18,52 +18,55 @@ package org.apache.hadoop.hdds.scm; -import com.google.common.annotations.VisibleForTesting; +import java.io.IOException; +import java.security.cert.X509Certificate; +import java.util.Collection; +import java.util.List; +import java.util.Objects; +import java.util.OptionalLong; +import java.util.UUID; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CompletionException; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicReference; +import java.util.stream.Collectors; + +import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdds.HddsUtils; import org.apache.hadoop.hdds.protocol.DatanodeDetails; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandRequestProto; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandResponseProto; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; +import org.apache.hadoop.hdds.ratis.ContainerCommandRequestMessage; +import org.apache.hadoop.hdds.scm.client.HddsClientUtils; +import org.apache.hadoop.hdds.scm.pipeline.Pipeline; import org.apache.hadoop.hdds.security.x509.SecurityConfig; - -import io.opentracing.Scope; -import io.opentracing.util.GlobalTracer; +import org.apache.hadoop.hdds.tracing.TracingUtil; import org.apache.hadoop.util.Time; +import org.apache.hadoop.hdds.ratis.RatisHelper; +import org.apache.ratis.client.RaftClient; import org.apache.ratis.grpc.GrpcTlsConfig; import org.apache.ratis.proto.RaftProtos; import org.apache.ratis.protocol.GroupMismatchException; -import org.apache.ratis.protocol.RaftRetryFailureException; -import org.apache.ratis.retry.RetryPolicy; -import org.apache.ratis.thirdparty.com.google.protobuf - .InvalidProtocolBufferException; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hdds.scm.client.HddsClientUtils; -import org.apache.hadoop.hdds.scm.pipeline.Pipeline; -import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos - .ContainerCommandRequestProto; -import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos - .ContainerCommandResponseProto; -import org.apache.hadoop.hdds.protocol.proto.HddsProtos; -import org.apache.hadoop.hdds.tracing.TracingUtil; - -import org.apache.ratis.RatisHelper; -import org.apache.ratis.client.RaftClient; import org.apache.ratis.protocol.RaftClientReply; +import org.apache.ratis.protocol.RaftException; +import org.apache.ratis.retry.RetryPolicy; import org.apache.ratis.rpc.RpcType; import org.apache.ratis.rpc.SupportedRpcType; -import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; +import org.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException; import org.apache.ratis.util.TimeDuration; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import java.io.IOException; -import java.util.*; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.CompletionException; -import java.util.concurrent.TimeUnit; -import java.util.concurrent.ExecutionException; -import java.util.concurrent.TimeoutException; -import java.util.concurrent.atomic.AtomicReference; -import java.util.concurrent.ConcurrentHashMap; -import java.util.stream.Collectors; +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Preconditions; + +import io.opentracing.Scope; +import io.opentracing.util.GlobalTracer; /** * An abstract implementation of {@link XceiverClientSpi} using Ratis. @@ -76,6 +79,12 @@ public final class XceiverClientRatis extends XceiverClientSpi { public static XceiverClientRatis newXceiverClientRatis( org.apache.hadoop.hdds.scm.pipeline.Pipeline pipeline, Configuration ozoneConf) { + return newXceiverClientRatis(pipeline, ozoneConf, null); + } + + public static XceiverClientRatis newXceiverClientRatis( + org.apache.hadoop.hdds.scm.pipeline.Pipeline pipeline, + Configuration ozoneConf, X509Certificate caCert) { final String rpcType = ozoneConf .get(ScmConfigKeys.DFS_CONTAINER_RATIS_RPC_TYPE_KEY, ScmConfigKeys.DFS_CONTAINER_RATIS_RPC_TYPE_DEFAULT); @@ -85,7 +94,7 @@ public static XceiverClientRatis newXceiverClientRatis( HddsClientUtils.getMaxOutstandingRequests(ozoneConf); final RetryPolicy retryPolicy = RatisHelper.createRetryPolicy(ozoneConf); final GrpcTlsConfig tlsConfig = RatisHelper.createTlsClientConfig(new - SecurityConfig(ozoneConf)); + SecurityConfig(ozoneConf), caCert); return new XceiverClientRatis(pipeline, SupportedRpcType.valueOfIgnoreCase(rpcType), maxOutstandingRequests, retryPolicy, tlsConfig, clientRequestTimeout); @@ -161,8 +170,10 @@ public Pipeline getPipeline() { @Override public void connect() throws Exception { - LOG.debug("Connecting to pipeline:{} datanode:{}", getPipeline().getId(), - RatisHelper.toRaftPeerId(pipeline.getFirstNode())); + if (LOG.isDebugEnabled()) { + LOG.debug("Connecting to pipeline:{} datanode:{}", getPipeline().getId(), + RatisHelper.toRaftPeerId(pipeline.getFirstNode())); + } // TODO : XceiverClient ratis should pass the config value of // maxOutstandingRequests so as to set the upper bound on max no of async // requests to be handled by raft client @@ -210,16 +221,20 @@ private CompletableFuture sendRequestAsync( try (Scope scope = GlobalTracer.get() .buildSpan("XceiverClientRatis." + request.getCmdType().name()) .startActive(true)) { - ContainerCommandRequestProto finalPayload = - ContainerCommandRequestProto.newBuilder(request) - .setTraceID(TracingUtil.exportCurrentSpan()) - .build(); - boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload); - ByteString byteString = finalPayload.toByteString(); - LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload); - return isReadOnlyRequest ? - getClient().sendReadOnlyAsync(() -> byteString) : - getClient().sendAsync(() -> byteString); + final ContainerCommandRequestMessage message + = ContainerCommandRequestMessage.toMessage( + request, TracingUtil.exportCurrentSpan()); + if (HddsUtils.isReadOnly(request)) { + if (LOG.isDebugEnabled()) { + LOG.debug("sendCommandAsync ReadOnly {}", message); + } + return getClient().sendReadOnlyAsync(message); + } else { + if (LOG.isDebugEnabled()) { + LOG.debug("sendCommandAsync {}", message); + } + return getClient().sendAsync(message); + } } } @@ -249,7 +264,9 @@ public XceiverClientReply watchForCommit(long index, long timeout) clientReply.setLogIndex(commitIndex); return clientReply; } - LOG.debug("commit index : {} watch timeout : {}", index, timeout); + if (LOG.isDebugEnabled()) { + LOG.debug("commit index : {} watch timeout : {}", index, timeout); + } RaftClientReply reply; try { CompletableFuture replyFuture = getClient() @@ -257,7 +274,7 @@ public XceiverClientReply watchForCommit(long index, long timeout) replyFuture.get(timeout, TimeUnit.MILLISECONDS); } catch (Exception e) { Throwable t = HddsClientUtils.checkForException(e); - LOG.warn("3 way commit failed ", e); + LOG.warn("3 way commit failed on pipeline {}", pipeline, e); if (t instanceof GroupMismatchException) { throw e; } @@ -276,8 +293,9 @@ public XceiverClientReply watchForCommit(long index, long timeout) // replication. commitInfoMap.remove(address); LOG.info( - "Could not commit " + index + " to all the nodes. Server " + address - + " has failed." + " Committed by majority."); + "Could not commit index {} on pipeline {} to all the nodes. " + + "Server {} has failed. Committed by majority.", + index, pipeline, address); }); } clientReply.setLogIndex(index); @@ -300,19 +318,18 @@ public XceiverClientReply sendCommandAsync( metrics.incrPendingContainerOpsMetrics(request.getCmdType()); CompletableFuture containerCommandResponse = raftClientReply.whenComplete((reply, e) -> { - LOG.debug("received reply {} for request: cmdType={} containerID={}" - + " pipelineID={} traceID={} exception: {}", reply, - request.getCmdType(), request.getContainerID(), - request.getPipelineID(), request.getTraceID(), e); + if (LOG.isDebugEnabled()) { + LOG.debug("received reply {} for request: cmdType={} containerID={}" + + " pipelineID={} traceID={} exception: {}", reply, + request.getCmdType(), request.getContainerID(), + request.getPipelineID(), request.getTraceID(), e); + } metrics.decrPendingContainerOpsMetrics(request.getCmdType()); metrics.addContainerOpsLatency(request.getCmdType(), Time.monotonicNowNanos() - requestTime); }).thenApply(reply -> { try { - // we need to handle RaftRetryFailure Exception - RaftRetryFailureException raftRetryFailureException = - reply.getRetryFailureException(); - if (raftRetryFailureException != null) { + if (!reply.isSuccess()) { // in case of raft retry failure, the raft client is // not able to connect to the leader hence the pipeline // can not be used but this instance of RaftClient will close @@ -324,7 +341,10 @@ public XceiverClientReply sendCommandAsync( // to SCM as in this case, it is the raft client which is not // able to connect to leader in the pipeline, though the // pipeline can still be functional. - throw new CompletionException(raftRetryFailureException); + RaftException exception = reply.getException(); + Preconditions.checkNotNull(exception, "Raft reply failure but " + + "no exception propagated."); + throw new CompletionException(exception); } ContainerCommandResponseProto response = ContainerCommandResponseProto diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/client/ContainerOperationClient.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/client/ContainerOperationClient.java index b0be34dd9bfff..982fb8ea1eec6 100644 --- a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/client/ContainerOperationClient.java +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/client/ContainerOperationClient.java @@ -37,7 +37,6 @@ import java.io.IOException; import java.util.List; -import java.util.UUID; /** * This class provides the client-facing APIs of container operations. @@ -113,8 +112,7 @@ public ContainerWithPipeline createContainer(String owner) */ public void createContainer(XceiverClientSpi client, long containerId) throws IOException { - String traceID = UUID.randomUUID().toString(); - ContainerProtocolCalls.createContainer(client, containerId, traceID, null); + ContainerProtocolCalls.createContainer(client, containerId, null); // Let us log this info after we let SCM know that we have completed the // creation state. @@ -167,8 +165,10 @@ private void createPipeline(XceiverClientSpi client, Pipeline pipeline) // TODO : Should we change the state on the client side ?? // That makes sense, but it is not needed for the client to work. - LOG.debug("Pipeline creation successful. Pipeline: {}", - pipeline.toString()); + if (LOG.isDebugEnabled()) { + LOG.debug("Pipeline creation successful. Pipeline: {}", + pipeline.toString()); + } } @Override @@ -228,6 +228,18 @@ public List listPipelines() throws IOException { return storageContainerLocationClient.listPipelines(); } + @Override + public void activatePipeline(HddsProtos.PipelineID pipelineID) + throws IOException { + storageContainerLocationClient.activatePipeline(pipelineID); + } + + @Override + public void deactivatePipeline(HddsProtos.PipelineID pipelineID) + throws IOException { + storageContainerLocationClient.deactivatePipeline(pipelineID); + } + @Override public void closePipeline(HddsProtos.PipelineID pipelineID) throws IOException { @@ -257,9 +269,8 @@ public void deleteContainer(long containerId, Pipeline pipeline, XceiverClientSpi client = null; try { client = xceiverClientManager.acquireClient(pipeline); - String traceID = UUID.randomUUID().toString(); ContainerProtocolCalls - .deleteContainer(client, containerId, force, traceID, null); + .deleteContainer(client, containerId, force, null); storageContainerLocationClient .deleteContainer(containerId); if (LOG.isDebugEnabled()) { @@ -307,10 +318,8 @@ public ContainerDataProto readContainer(long containerID, XceiverClientSpi client = null; try { client = xceiverClientManager.acquireClient(pipeline); - String traceID = UUID.randomUUID().toString(); ReadContainerResponseProto response = - ContainerProtocolCalls.readContainer(client, containerID, traceID, - null); + ContainerProtocolCalls.readContainer(client, containerID, null); if (LOG.isDebugEnabled()) { LOG.debug("Read container {}, machines: {} ", containerID, pipeline.getNodes()); @@ -372,7 +381,9 @@ public void closeContainer(long containerId, Pipeline pipeline) throws IOException { XceiverClientSpi client = null; try { - LOG.debug("Close container {}", pipeline); + if (LOG.isDebugEnabled()) { + LOG.debug("Close container {}", pipeline); + } /* TODO: two orders here, revisit this later: 1. close on SCM first, then on data node @@ -393,7 +404,6 @@ public void closeContainer(long containerId, Pipeline pipeline) */ // Actually close the container on Datanode client = xceiverClientManager.acquireClient(pipeline); - String traceID = UUID.randomUUID().toString(); storageContainerLocationClient.notifyObjectStageChange( ObjectStageChangeRequestProto.Type.container, @@ -401,7 +411,7 @@ public void closeContainer(long containerId, Pipeline pipeline) ObjectStageChangeRequestProto.Op.close, ObjectStageChangeRequestProto.Stage.begin); - ContainerProtocolCalls.closeContainer(client, containerId, traceID, + ContainerProtocolCalls.closeContainer(client, containerId, null); // Notify SCM to close the container storageContainerLocationClient.notifyObjectStageChange( @@ -465,4 +475,21 @@ public boolean inSafeMode() throws IOException { public boolean forceExitSafeMode() throws IOException { return storageContainerLocationClient.forceExitSafeMode(); } + + @Override + public void startReplicationManager() throws IOException { + storageContainerLocationClient.startReplicationManager(); + } + + @Override + public void stopReplicationManager() throws IOException { + storageContainerLocationClient.stopReplicationManager(); + } + + @Override + public boolean getReplicationManagerStatus() throws IOException { + return storageContainerLocationClient.getReplicationManagerStatus(); + } + + } diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/client/HddsClientUtils.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/client/HddsClientUtils.java index 97b8f95b87e51..d3bb31aa69878 100644 --- a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/client/HddsClientUtils.java +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/client/HddsClientUtils.java @@ -27,8 +27,8 @@ import org.apache.hadoop.hdds.protocol.SCMSecurityProtocol; import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB; import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolPB; -import org.apache.hadoop.hdds.scm.ScmConfigKeys; -import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException; +import org.apache.hadoop.hdds.scm.XceiverClientManager.ScmClientConfig; +import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException; import org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol; import org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolPB; import org.apache.hadoop.io.retry.RetryPolicies; @@ -86,7 +86,7 @@ private HddsClientUtils() { private static final List> EXCEPTION_LIST = new ArrayList>() {{ add(TimeoutException.class); - add(ContainerNotOpenException.class); + add(StorageContainerException.class); add(RaftRetryFailureException.class); add(AlreadyClosedException.class); add(GroupMismatchException.class); @@ -126,8 +126,6 @@ public static long formatDateTime(String date) throws ParseException { .toInstant().toEpochMilli(); } - - /** * verifies that bucket name / volume name is a valid DNS name. * @@ -137,27 +135,26 @@ public static long formatDateTime(String date) throws ParseException { */ public static void verifyResourceName(String resName) throws IllegalArgumentException { - if (resName == null) { throw new IllegalArgumentException("Bucket or Volume name is null"); } - if ((resName.length() < OzoneConsts.OZONE_MIN_BUCKET_NAME_LENGTH) || - (resName.length() > OzoneConsts.OZONE_MAX_BUCKET_NAME_LENGTH)) { + if (resName.length() < OzoneConsts.OZONE_MIN_BUCKET_NAME_LENGTH || + resName.length() > OzoneConsts.OZONE_MAX_BUCKET_NAME_LENGTH) { throw new IllegalArgumentException( - "Bucket or Volume length is illegal, " + - "valid length is 3-63 characters"); + "Bucket or Volume length is illegal, " + + "valid length is 3-63 characters"); } - if ((resName.charAt(0) == '.') || (resName.charAt(0) == '-')) { + if (resName.charAt(0) == '.' || resName.charAt(0) == '-') { throw new IllegalArgumentException( "Bucket or Volume name cannot start with a period or dash"); } - if ((resName.charAt(resName.length() - 1) == '.') || - (resName.charAt(resName.length() - 1) == '-')) { - throw new IllegalArgumentException( - "Bucket or Volume name cannot end with a period or dash"); + if (resName.charAt(resName.length() - 1) == '.' || + resName.charAt(resName.length() - 1) == '-') { + throw new IllegalArgumentException("Bucket or Volume name " + + "cannot end with a period or dash"); } boolean isIPv4 = true; @@ -165,36 +162,30 @@ public static void verifyResourceName(String resName) for (int index = 0; index < resName.length(); index++) { char currChar = resName.charAt(index); - if (currChar != '.') { isIPv4 = ((currChar >= '0') && (currChar <= '9')) && isIPv4; } - if (currChar > 'A' && currChar < 'Z') { throw new IllegalArgumentException( "Bucket or Volume name does not support uppercase characters"); } - - if ((currChar != '.') && (currChar != '-')) { - if ((currChar < '0') || (currChar > '9' && currChar < 'a') || - (currChar > 'z')) { + if (currChar != '.' && currChar != '-') { + if (currChar < '0' || (currChar > '9' && currChar < 'a') || + currChar > 'z') { throw new IllegalArgumentException("Bucket or Volume name has an " + "unsupported character : " + currChar); } } - - if ((prev == '.') && (currChar == '.')) { + if (prev == '.' && currChar == '.') { throw new IllegalArgumentException("Bucket or Volume name should not " + "have two contiguous periods"); } - - if ((prev == '-') && (currChar == '.')) { + if (prev == '-' && currChar == '.') { throw new IllegalArgumentException( "Bucket or Volume name should not have period after dash"); } - - if ((prev == '.') && (currChar == '-')) { + if (prev == '.' && currChar == '-') { throw new IllegalArgumentException( "Bucket or Volume name should not have dash after period"); } @@ -285,10 +276,9 @@ public static CloseableHttpClient newHttpClient(Configuration conf) { * Standalone and Ratis client. */ public static int getMaxOutstandingRequests(Configuration config) { - return config - .getInt(ScmConfigKeys.SCM_CONTAINER_CLIENT_MAX_OUTSTANDING_REQUESTS, - ScmConfigKeys - .SCM_CONTAINER_CLIENT_MAX_OUTSTANDING_REQUESTS_DEFAULT); + return OzoneConfiguration.of(config) + .getObject(ScmClientConfig.class) + .getMaxOutstandingRequests(); } /** @@ -314,7 +304,7 @@ public static SCMSecurityProtocol getScmSecurityClient( return scmSecurityClient; } - public static Throwable checkForException(Exception e) throws IOException { + public static Throwable checkForException(Exception e) { Throwable t = e; while (t != null) { for (Class cls : getExceptionList()) { @@ -324,8 +314,7 @@ public static Throwable checkForException(Exception e) throws IOException { } t = t.getCause(); } - - throw e instanceof IOException ? (IOException)e : new IOException(e); + return t; } public static RetryPolicy createRetryPolicy(int maxRetryCount, diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java index bb4a5b05633d9..40bbd93b16f1e 100644 --- a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java @@ -19,491 +19,370 @@ package org.apache.hadoop.hdds.scm.storage; import com.google.common.annotations.VisibleForTesting; -import com.google.common.base.Preconditions; -import org.apache.hadoop.hdds.protocol.DatanodeDetails; -import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; -import org.apache.hadoop.hdds.scm.XceiverClientReply; -import org.apache.hadoop.hdds.scm.container.common.helpers - .StorageContainerException; -import org.apache.hadoop.ozone.common.Checksum; -import org.apache.hadoop.ozone.common.ChecksumData; -import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; + +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; +import org.apache.hadoop.hdds.scm.pipeline.Pipeline; +import org.apache.hadoop.hdds.security.token.OzoneBlockTokenIdentifier; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.token.Token; import org.apache.hadoop.fs.Seekable; import org.apache.hadoop.hdds.scm.XceiverClientManager; import org.apache.hadoop.hdds.scm.XceiverClientSpi; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ChunkInfo; -import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos - .ReadChunkResponseProto; import org.apache.hadoop.hdds.client.BlockID; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.DatanodeBlockID; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.GetBlockResponseProto; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import java.io.EOFException; import java.io.IOException; import java.io.InputStream; -import java.nio.ByteBuffer; import java.util.ArrayList; import java.util.Arrays; import java.util.List; -import java.util.concurrent.ExecutionException; /** - * An {@link InputStream} used by the REST service in combination with the - * SCMClient to read the value of a key from a sequence - * of container chunks. All bytes of the key value are stored in container - * chunks. Each chunk may contain multiple underlying {@link ByteBuffer} - * instances. This class encapsulates all state management for iterating - * through the sequence of chunks and the sequence of buffers within each chunk. + * An {@link InputStream} called from KeyInputStream to read a block from the + * container. + * This class encapsulates all state management for iterating + * through the sequence of chunks through {@link ChunkInputStream}. */ public class BlockInputStream extends InputStream implements Seekable { + private static final Logger LOG = + LoggerFactory.getLogger(BlockInputStream.class); + private static final int EOF = -1; private final BlockID blockID; - private final String traceID; + private final long length; + private Pipeline pipeline; + private final Token token; + private final boolean verifyChecksum; private XceiverClientManager xceiverClientManager; private XceiverClientSpi xceiverClient; - private List chunks; - // ChunkIndex points to the index current chunk in the buffers or the the - // index of chunk which will be read next into the buffers in - // readChunkFromContainer(). + private boolean initialized = false; + + // List of ChunkInputStreams, one for each chunk in the block + private List chunkStreams; + + // chunkOffsets[i] stores the index of the first data byte in + // chunkStream i w.r.t the block data. + // Let’s say we have chunk size as 40 bytes. And let's say the parent + // block stores data from index 200 and has length 400. + // The first 40 bytes of this block will be stored in chunk[0], next 40 in + // chunk[1] and so on. But since the chunkOffsets are w.r.t the block only + // and not the key, the values in chunkOffsets will be [0, 40, 80,....]. + private long[] chunkOffsets = null; + + // Index of the chunkStream corresponding to the current position of the + // BlockInputStream i.e offset of the data to be read next from this block private int chunkIndex; - // ChunkIndexOfCurrentBuffer points to the index of chunk read into the - // buffers or index of the last chunk in the buffers. It is updated only - // when a new chunk is read from container into the buffers. - private int chunkIndexOfCurrentBuffer; - private long[] chunkOffset; - private List buffers; - private int bufferIndex; - private long bufferPosition; - private final boolean verifyChecksum; - /** - * Creates a new BlockInputStream. - * - * @param blockID block ID of the chunk - * @param xceiverClientManager client manager that controls client - * @param xceiverClient client to perform container calls - * @param chunks list of chunks to read - * @param traceID container protocol call traceID - * @param verifyChecksum verify checksum - * @param initialPosition the initial position of the stream pointer. This - * position is seeked now if the up-stream was seeked - * before this was created. - */ - public BlockInputStream( - BlockID blockID, XceiverClientManager xceiverClientManager, - XceiverClientSpi xceiverClient, List chunks, String traceID, - boolean verifyChecksum, long initialPosition) throws IOException { - this.blockID = blockID; - this.traceID = traceID; - this.xceiverClientManager = xceiverClientManager; - this.xceiverClient = xceiverClient; - this.chunks = chunks; - this.chunkIndex = 0; - this.chunkIndexOfCurrentBuffer = -1; - // chunkOffset[i] stores offset at which chunk i stores data in - // BlockInputStream - this.chunkOffset = new long[this.chunks.size()]; - initializeChunkOffset(); - this.buffers = null; - this.bufferIndex = 0; - this.bufferPosition = -1; + // Position of the BlockInputStream is maintainted by this variable till + // the stream is initialized. This position is w.r.t to the block only and + // not the key. + // For the above example, if we seek to position 240 before the stream is + // initialized, then value of blockPosition will be set to 40. + // Once, the stream is initialized, the position of the stream + // will be determined by the current chunkStream and its position. + private long blockPosition = 0; + + // Tracks the chunkIndex corresponding to the last blockPosition so that it + // can be reset if a new position is seeked. + private int chunkIndexOfPrevPosition; + + public BlockInputStream(BlockID blockId, long blockLen, Pipeline pipeline, + Token token, boolean verifyChecksum, + XceiverClientManager xceiverClientManager) { + this.blockID = blockId; + this.length = blockLen; + this.pipeline = pipeline; + this.token = token; this.verifyChecksum = verifyChecksum; - if (initialPosition > 0) { - // The stream was seeked to a position before the stream was - // initialized. So seeking to the position now. - seek(initialPosition); - } + this.xceiverClientManager = xceiverClientManager; } - private void initializeChunkOffset() { - long tempOffset = 0; - for (int i = 0; i < chunks.size(); i++) { - chunkOffset[i] = tempOffset; - tempOffset += chunks.get(i).getLen(); + /** + * Initialize the BlockInputStream. Get the BlockData (list of chunks) from + * the Container and create the ChunkInputStreams for each Chunk in the Block. + */ + public synchronized void initialize() throws IOException { + + // Pre-check that the stream has not been intialized already + if (initialized) { + return; } - } - @Override - public synchronized int read() - throws IOException { - checkOpen(); - int available = prepareRead(1); - int dataout = EOF; + List chunks = getChunkInfos(); + if (chunks != null && !chunks.isEmpty()) { + // For each chunk in the block, create a ChunkInputStream and compute + // its chunkOffset + this.chunkOffsets = new long[chunks.size()]; + long tempOffset = 0; + + this.chunkStreams = new ArrayList<>(chunks.size()); + for (int i = 0; i < chunks.size(); i++) { + addStream(chunks.get(i)); + chunkOffsets[i] = tempOffset; + tempOffset += chunks.get(i).getLen(); + } - if (available == EOF) { - Preconditions - .checkState(buffers == null); //should have released by now, see below - } else { - dataout = Byte.toUnsignedInt(buffers.get(bufferIndex).get()); - } + initialized = true; + this.chunkIndex = 0; - if (blockStreamEOF()) { - // consumer might use getPos to determine EOF, - // so release buffers when serving the last byte of data - releaseBuffers(); + if (blockPosition > 0) { + // Stream was seeked to blockPosition before initialization. Seek to the + // blockPosition now. + seek(blockPosition); + } } - - return dataout; } - @Override - public synchronized int read(byte[] b, int off, int len) throws IOException { - // According to the JavaDocs for InputStream, it is recommended that - // subclasses provide an override of bulk read if possible for performance - // reasons. In addition to performance, we need to do it for correctness - // reasons. The Ozone REST service uses PipedInputStream and - // PipedOutputStream to relay HTTP response data between a Jersey thread and - // a Netty thread. It turns out that PipedInputStream/PipedOutputStream - // have a subtle dependency (bug?) on the wrapped stream providing separate - // implementations of single-byte read and bulk read. Without this, get key - // responses might close the connection before writing all of the bytes - // advertised in the Content-Length. - if (b == null) { - throw new NullPointerException(); - } - if (off < 0 || len < 0 || len > b.length - off) { - throw new IndexOutOfBoundsException(); - } - if (len == 0) { - return 0; + /** + * Send RPC call to get the block info from the container. + * @return List of chunks in this block. + */ + protected List getChunkInfos() throws IOException { + // irrespective of the container state, we will always read via Standalone + // protocol. + if (pipeline.getType() != HddsProtos.ReplicationType.STAND_ALONE) { + pipeline = Pipeline.newBuilder(pipeline) + .setType(HddsProtos.ReplicationType.STAND_ALONE).build(); } - checkOpen(); - int total = 0; - while (len > 0) { - int available = prepareRead(len); - if (available == EOF) { - Preconditions - .checkState(buffers == null); //should have been released by now - return total != 0 ? total : EOF; + xceiverClient = xceiverClientManager.acquireClientForReadData(pipeline); + boolean success = false; + List chunks; + try { + if (LOG.isDebugEnabled()) { + LOG.debug("Initializing BlockInputStream for get key to access {}", + blockID.getContainerID()); } - buffers.get(bufferIndex).get(b, off + total, available); - len -= available; - total += available; - } - if (blockStreamEOF()) { - // smart consumers determine EOF by calling getPos() - // so we release buffers when serving the final bytes of data - releaseBuffers(); + if (token != null) { + UserGroupInformation.getCurrentUser().addToken(token); + } + DatanodeBlockID datanodeBlockID = blockID + .getDatanodeBlockIDProtobuf(); + GetBlockResponseProto response = ContainerProtocolCalls + .getBlock(xceiverClient, datanodeBlockID); + + chunks = response.getBlockData().getChunksList(); + success = true; + } finally { + if (!success) { + xceiverClientManager.releaseClientForReadData(xceiverClient, false); + } } - return total; + return chunks; } /** - * Determines if all data in the stream has been consumed. - * - * @return true if EOF, false if more data is available + * Append another ChunkInputStream to the end of the list. Note that the + * ChunkInputStream is only created here. The chunk will be read from the + * Datanode only when a read operation is performed on for that chunk. */ - protected boolean blockStreamEOF() { - if (buffersHaveData() || chunksRemaining()) { - return false; - } else { - // if there are any chunks, we better be at the last chunk for EOF - Preconditions.checkState(((chunks == null) || chunks.isEmpty() || - chunkIndex == (chunks.size() - 1)), - "EOF detected, but not at the last chunk"); - return true; - } - } - - private void releaseBuffers() { - //ashes to ashes, dust to dust - buffers = null; - bufferIndex = 0; + protected synchronized void addStream(ChunkInfo chunkInfo) { + chunkStreams.add(new ChunkInputStream(chunkInfo, blockID, + xceiverClient, verifyChecksum)); } - @Override - public synchronized void close() { - if (xceiverClientManager != null && xceiverClient != null) { - xceiverClientManager.releaseClient(xceiverClient, false); - xceiverClientManager = null; - xceiverClient = null; - } + public synchronized long getRemaining() throws IOException { + return length - getPos(); } /** - * Checks if the stream is open. If not, throws an exception. - * - * @throws IOException if stream is closed + * {@inheritDoc} */ - private synchronized void checkOpen() throws IOException { - if (xceiverClient == null) { - throw new IOException("BlockInputStream has been closed."); + @Override + public synchronized int read() throws IOException { + byte[] buf = new byte[1]; + if (read(buf, 0, 1) == EOF) { + return EOF; } + return Byte.toUnsignedInt(buf[0]); } /** - * Prepares to read by advancing through chunks and buffers as needed until it - * finds data to return or encounters EOF. - * - * @param len desired length of data to read - * @return length of data available to read, possibly less than desired length + * {@inheritDoc} */ - private synchronized int prepareRead(int len) throws IOException { - for (;;) { - if (!buffersAllocated()) { - // The current chunk at chunkIndex has not been read from the - // container. Read the chunk and put the data into buffers. - readChunkFromContainer(); - } - if (buffersHaveData()) { - // Data is available from buffers - ByteBuffer bb = buffers.get(bufferIndex); - return len > bb.remaining() ? bb.remaining() : len; - } else if (chunksRemaining()) { - // There are additional chunks available. - // Read the next chunk in the block. - chunkIndex += 1; - readChunkFromContainer(); - } else { - // All available input has been consumed. - return EOF; - } + @Override + public synchronized int read(byte[] b, int off, int len) throws IOException { + if (b == null) { + throw new NullPointerException(); } - } - - private boolean buffersAllocated() { - if (buffers == null || buffers.isEmpty()) { - return false; + if (off < 0 || len < 0 || len > b.length - off) { + throw new IndexOutOfBoundsException(); } - return true; - } - - private boolean buffersHaveData() { - boolean hasData = false; - - if (buffersAllocated()) { - while (bufferIndex < (buffers.size())) { - if (buffers.get(bufferIndex).hasRemaining()) { - // current buffer has data - hasData = true; - break; - } else { - if (buffersRemaining()) { - // move to next available buffer - ++bufferIndex; - Preconditions.checkState(bufferIndex < buffers.size()); - } else { - // no more buffers remaining - break; - } - } - } + if (len == 0) { + return 0; } - return hasData; - } + if (!initialized) { + initialize(); + } - private boolean buffersRemaining() { - return (bufferIndex < (buffers.size() - 1)); - } + checkOpen(); + int totalReadLen = 0; + while (len > 0) { + // if we are at the last chunk and have read the entire chunk, return + if (chunkStreams.size() == 0 || + (chunkStreams.size() - 1 <= chunkIndex && + chunkStreams.get(chunkIndex) + .getRemaining() == 0)) { + return totalReadLen == 0 ? EOF : totalReadLen; + } - private boolean chunksRemaining() { - if ((chunks == null) || chunks.isEmpty()) { - return false; - } - // Check if more chunks are remaining in the stream after chunkIndex - if (chunkIndex < (chunks.size() - 1)) { - return true; + // Get the current chunkStream and read data from it + ChunkInputStream current = chunkStreams.get(chunkIndex); + int numBytesToRead = Math.min(len, (int)current.getRemaining()); + int numBytesRead = current.read(b, off, numBytesToRead); + if (numBytesRead != numBytesToRead) { + // This implies that there is either data loss or corruption in the + // chunk entries. Even EOF in the current stream would be covered in + // this case. + throw new IOException(String.format( + "Inconsistent read for chunkName=%s length=%d numBytesRead=%d", + current.getChunkName(), current.getLength(), numBytesRead)); + } + totalReadLen += numBytesRead; + off += numBytesRead; + len -= numBytesRead; + if (current.getRemaining() <= 0 && + ((chunkIndex + 1) < chunkStreams.size())) { + chunkIndex += 1; + } } - // ChunkIndex is the last chunk in the stream. Check if this chunk has - // been read from container or not. Return true if chunkIndex has not - // been read yet and false otherwise. - return chunkIndexOfCurrentBuffer != chunkIndex; + return totalReadLen; } /** - * Attempts to read the chunk at the specified offset in the chunk list. If - * successful, then the data of the read chunk is saved so that its bytes can - * be returned from subsequent read calls. + * Seeks the BlockInputStream to the specified position. If the stream is + * not initialized, save the seeked position via blockPosition. Otherwise, + * update the position in 2 steps: + * 1. Updating the chunkIndex to the chunkStream corresponding to the + * seeked position. + * 2. Seek the corresponding chunkStream to the adjusted position. * - * @throws IOException if there is an I/O error while performing the call + * Let’s say we have chunk size as 40 bytes. And let's say the parent block + * stores data from index 200 and has length 400. If the key was seeked to + * position 90, then this block will be seeked to position 90. + * When seek(90) is called on this blockStream, then + * 1. chunkIndex will be set to 2 (as indices 80 - 120 reside in chunk[2]). + * 2. chunkStream[2] will be seeked to position 10 + * (= 90 - chunkOffset[2] (= 80)). */ - private synchronized void readChunkFromContainer() throws IOException { - // Read the chunk at chunkIndex - final ChunkInfo chunkInfo = chunks.get(chunkIndex); - List excludeDns = null; - ByteString byteString; - List dnList = getDatanodeList(); - while (true) { - List dnListFromReadChunkCall = new ArrayList<>(); - byteString = readChunk(chunkInfo, excludeDns, dnListFromReadChunkCall); - try { - if (byteString.size() != chunkInfo.getLen()) { - // Bytes read from chunk should be equal to chunk size. - throw new IOException(String - .format("Inconsistent read for chunk=%s len=%d bytesRead=%d", - chunkInfo.getChunkName(), chunkInfo.getLen(), - byteString.size())); - } - ChecksumData checksumData = - ChecksumData.getFromProtoBuf(chunkInfo.getChecksumData()); - if (verifyChecksum) { - Checksum.verifyChecksum(byteString, checksumData); - } - break; - } catch (IOException ioe) { - // we will end up in this situation only if the checksum mismatch - // happens or the length of the chunk mismatches. - // In this case, read should be retried on a different replica. - // TODO: Inform SCM of a possible corrupt container replica here - if (excludeDns == null) { - excludeDns = new ArrayList<>(); - } - excludeDns.addAll(dnListFromReadChunkCall); - if (excludeDns.size() == dnList.size()) { - throw ioe; - } - } + @Override + public synchronized void seek(long pos) throws IOException { + if (!initialized) { + // Stream has not been initialized yet. Save the position so that it + // can be seeked when the stream is initialized. + blockPosition = pos; + return; } - buffers = byteString.asReadOnlyByteBufferList(); - bufferIndex = 0; - chunkIndexOfCurrentBuffer = chunkIndex; - - // The bufferIndex and position might need to be adjusted if seek() was - // called on the stream before. This needs to be done so that the buffer - // position can be advanced to the 'seeked' position. - adjustBufferIndex(); - } - - /** - * Send RPC call to get the chunk from the container. - */ - @VisibleForTesting - protected ByteString readChunk(final ChunkInfo chunkInfo, - List excludeDns, List dnListFromReply) - throws IOException { - XceiverClientReply reply; - ReadChunkResponseProto readChunkResponse = null; - try { - reply = ContainerProtocolCalls - .readChunk(xceiverClient, chunkInfo, blockID, traceID, excludeDns); - ContainerProtos.ContainerCommandResponseProto response; - response = reply.getResponse().get(); - ContainerProtocolCalls.validateContainerResponse(response); - readChunkResponse = response.getReadChunk(); - dnListFromReply.addAll(reply.getDatanodes()); - } catch (IOException e) { - if (e instanceof StorageContainerException) { - throw e; + checkOpen(); + if (pos < 0 || pos >= length) { + if (pos == 0) { + // It is possible for length and pos to be zero in which case + // seek should return instead of throwing exception + return; } - throw new IOException("Unexpected OzoneException: " + e.toString(), e); - } catch (ExecutionException | InterruptedException e) { - throw new IOException( - "Failed to execute ReadChunk command for chunk " + chunkInfo - .getChunkName(), e); - } - return readChunkResponse.getData(); - } - - @VisibleForTesting - protected List getDatanodeList() { - return xceiverClient.getPipeline().getNodes(); - } - - @Override - public synchronized void seek(long pos) throws IOException { - if (pos < 0 || (chunks.size() == 0 && pos > 0) - || pos >= chunkOffset[chunks.size() - 1] + chunks.get(chunks.size() - 1) - .getLen()) { - throw new EOFException("EOF encountered pos: " + pos + " container key: " - + blockID.getLocalID()); + throw new EOFException( + "EOF encountered at pos: " + pos + " for block: " + blockID); } - if (pos < chunkOffset[chunkIndex]) { - chunkIndex = Arrays.binarySearch(chunkOffset, 0, chunkIndex, pos); - } else if (pos >= chunkOffset[chunkIndex] + chunks.get(chunkIndex) - .getLen()) { + if (chunkIndex >= chunkStreams.size()) { + chunkIndex = Arrays.binarySearch(chunkOffsets, pos); + } else if (pos < chunkOffsets[chunkIndex]) { chunkIndex = - Arrays.binarySearch(chunkOffset, chunkIndex + 1, chunks.size(), pos); + Arrays.binarySearch(chunkOffsets, 0, chunkIndex, pos); + } else if (pos >= chunkOffsets[chunkIndex] + chunkStreams + .get(chunkIndex).getLength()) { + chunkIndex = Arrays.binarySearch(chunkOffsets, + chunkIndex + 1, chunkStreams.size(), pos); } if (chunkIndex < 0) { // Binary search returns -insertionPoint - 1 if element is not present // in the array. insertionPoint is the point at which element would be // inserted in the sorted array. We need to adjust the chunkIndex // accordingly so that chunkIndex = insertionPoint - 1 - chunkIndex = -chunkIndex -2; + chunkIndex = -chunkIndex - 2; } - // The bufferPosition should be adjusted to account for the chunk offset - // of the chunk the the pos actually points to. - bufferPosition = pos - chunkOffset[chunkIndex]; + // Reset the previous chunkStream's position + chunkStreams.get(chunkIndexOfPrevPosition).resetPosition(); - // Check if current buffers correspond to the chunk index being seeked - // and if the buffers have any data. - if (chunkIndex == chunkIndexOfCurrentBuffer && buffersAllocated()) { - // Position the buffer to the seeked position. - adjustBufferIndex(); - } else { - // Release the current buffers. The next readChunkFromContainer will - // read the required chunk and position the buffer to the seeked - // position. - releaseBuffers(); - } + // seek to the proper offset in the ChunkInputStream + chunkStreams.get(chunkIndex).seek(pos - chunkOffsets[chunkIndex]); + chunkIndexOfPrevPosition = chunkIndex; } - private void adjustBufferIndex() { - if (bufferPosition == -1) { - // The stream has not been seeked to a position. No need to adjust the - // buffer Index and position. - return; + @Override + public synchronized long getPos() throws IOException { + if (length == 0) { + return 0; } - // The bufferPosition is w.r.t the buffers for current chunk. - // Adjust the bufferIndex and position to the seeked position. - long tempOffest = 0; - for (int i = 0; i < buffers.size(); i++) { - if (bufferPosition - tempOffest >= buffers.get(i).capacity()) { - tempOffest += buffers.get(i).capacity(); - } else { - bufferIndex = i; - break; - } + + if (!initialized) { + // The stream is not initialized yet. Return the blockPosition + return blockPosition; + } else { + return chunkOffsets[chunkIndex] + chunkStreams.get(chunkIndex).getPos(); } - buffers.get(bufferIndex).position((int) (bufferPosition - tempOffest)); - // Reset the bufferPosition as the seek() operation has been completed. - bufferPosition = -1; } @Override - public synchronized long getPos() throws IOException { - // position = chunkOffset of current chunk (at chunkIndex) + position of - // the buffer corresponding to the chunk. - long bufferPos = 0; - - if (bufferPosition >= 0) { - // seek has been called but the buffers were empty. Hence, the buffer - // position will be advanced after the buffers are filled. - // We return the chunkOffset + bufferPosition here as that will be the - // position of the buffer pointer after reading the chunk file. - bufferPos = bufferPosition; - - } else if (blockStreamEOF()) { - // all data consumed, buffers have been released. - // get position from the chunk offset and chunk length of last chunk - bufferPos = chunks.get(chunkIndex).getLen(); - - } else if (buffersAllocated()) { - // get position from available buffers of current chunk - bufferPos = buffers.get(bufferIndex).position(); + public boolean seekToNewSource(long targetPos) throws IOException { + return false; + } + @Override + public synchronized void close() { + if (xceiverClientManager != null && xceiverClient != null) { + xceiverClientManager.releaseClient(xceiverClient, false); + xceiverClientManager = null; + xceiverClient = null; } + } - return chunkOffset[chunkIndex] + bufferPos; + public synchronized void resetPosition() { + this.blockPosition = 0; } - @Override - public boolean seekToNewSource(long targetPos) throws IOException { - return false; + /** + * Checks if the stream is open. If not, throw an exception. + * + * @throws IOException if stream is closed + */ + protected synchronized void checkOpen() throws IOException { + if (xceiverClient == null) { + throw new IOException("BlockInputStream has been closed."); + } } public BlockID getBlockID() { return blockID; } + public long getLength() { + return length; + } + @VisibleForTesting - protected int getChunkIndex() { + synchronized int getChunkIndex() { return chunkIndex; } + + @VisibleForTesting + synchronized long getBlockPosition() { + return blockPosition; + } + + @VisibleForTesting + synchronized List getChunkStreams() { + return chunkStreams; + } } diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java index 5ca32630c87c4..b15ca3f6c85fc 100644 --- a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java @@ -21,7 +21,6 @@ import com.google.common.base.Preconditions; import org.apache.hadoop.hdds.protocol.DatanodeDetails; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; -import org.apache.hadoop.hdds.scm.ByteStringHelper; import org.apache.hadoop.hdds.scm.XceiverClientReply; import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException; import org.apache.hadoop.hdds.scm.pipeline.Pipeline; @@ -29,7 +28,6 @@ import org.apache.hadoop.ozone.common.ChecksumData; import org.apache.hadoop.ozone.common.OzoneChecksumException; import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; -import org.apache.commons.codec.digest.DigestUtils; import org.apache.hadoop.hdds.scm.XceiverClientManager; import org.apache.hadoop.hdds.scm.XceiverClientSpi; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ChecksumType; @@ -43,8 +41,6 @@ import java.io.IOException; import java.io.OutputStream; import java.nio.ByteBuffer; -import java.util.Collections; -import java.util.UUID; import java.util.List; import java.util.ArrayList; import java.util.Map; @@ -81,14 +77,12 @@ public class BlockOutputStream extends OutputStream { LoggerFactory.getLogger(BlockOutputStream.class); private volatile BlockID blockID; - private final String key; - private final String traceID; + private final BlockData.Builder containerBlockData; private XceiverClientManager xceiverClientManager; private XceiverClientSpi xceiverClient; private final ContainerProtos.ChecksumType checksumType; private final int bytesPerChecksum; - private final String streamId; private int chunkIndex; private int chunkSize; private final long streamBufferFlushSize; @@ -125,10 +119,8 @@ public class BlockOutputStream extends OutputStream { * Creates a new BlockOutputStream. * * @param blockID block ID - * @param key chunk key * @param xceiverClientManager client manager that controls client * @param pipeline pipeline where block will be written - * @param traceID container protocol call args * @param chunkSize chunk size * @param bufferPool pool of buffers * @param streamBufferFlushSize flush size @@ -138,15 +130,13 @@ public class BlockOutputStream extends OutputStream { * @param bytesPerChecksum Bytes per checksum */ @SuppressWarnings("parameternumber") - public BlockOutputStream(BlockID blockID, String key, + public BlockOutputStream(BlockID blockID, XceiverClientManager xceiverClientManager, Pipeline pipeline, - String traceID, int chunkSize, long streamBufferFlushSize, - long streamBufferMaxSize, long watchTimeout, BufferPool bufferPool, - ChecksumType checksumType, int bytesPerChecksum) + int chunkSize, long streamBufferFlushSize, long streamBufferMaxSize, + long watchTimeout, BufferPool bufferPool, ChecksumType checksumType, + int bytesPerChecksum) throws IOException { this.blockID = blockID; - this.key = key; - this.traceID = traceID; this.chunkSize = chunkSize; KeyValue keyValue = KeyValue.newBuilder().setKey("TYPE").setValue("KEY").build(); @@ -155,7 +145,6 @@ public BlockOutputStream(BlockID blockID, String key, .addMetadata(keyValue); this.xceiverClientManager = xceiverClientManager; this.xceiverClient = xceiverClientManager.acquireClient(pipeline); - this.streamId = UUID.randomUUID().toString(); this.chunkIndex = 0; this.streamBufferFlushSize = streamBufferFlushSize; this.streamBufferMaxSize = streamBufferMaxSize; @@ -169,7 +158,7 @@ public BlockOutputStream(BlockID blockID, String key, bufferList = null; totalDataFlushedLength = 0; writtenDataLength = 0; - failedServers = Collections.emptyList(); + failedServers = new ArrayList<>(0); ioException = new AtomicReference<>(null); } @@ -358,9 +347,10 @@ private void watchForCommit(boolean bufferFull) throws IOException { if (reply != null) { List dnList = reply.getDatanodes(); if (!dnList.isEmpty()) { - if (failedServers.isEmpty()) { - failedServers = new ArrayList<>(); - } + Pipeline pipe = xceiverClient.getPipeline(); + + LOG.warn("Failed to commit BlockId {} on {}. Failed nodes: {}", + blockID, pipe, dnList); failedServers.addAll(dnList); } } @@ -379,13 +369,12 @@ ContainerCommandResponseProto> executePutBlock() List byteBufferList = bufferList; bufferList = null; Preconditions.checkNotNull(byteBufferList); - String requestId = - traceID + ContainerProtos.Type.PutBlock + chunkIndex + blockID; + CompletableFuture flushFuture; try { XceiverClientReply asyncReply = - putBlockAsync(xceiverClient, containerBlockData.build(), requestId); + putBlockAsync(xceiverClient, containerBlockData.build()); CompletableFuture future = asyncReply.getResponse(); flushFuture = future.thenApplyAsync(e -> { @@ -402,22 +391,26 @@ ContainerCommandResponseProto> executePutBlock() .equals(responseBlockID.getContainerBlockID())); // updates the bcsId of the block blockID = responseBlockID; - LOG.debug( - "Adding index " + asyncReply.getLogIndex() + " commitMap size " - + commitWatcher.getCommitInfoMapSize() + " flushLength " - + flushPos + " numBuffers " + byteBufferList.size() - + " blockID " + blockID + " bufferPool size" + bufferPool - .getSize() + " currentBufferIndex " + bufferPool - .getCurrentBufferIndex()); + if (LOG.isDebugEnabled()) { + LOG.debug( + "Adding index " + asyncReply.getLogIndex() + " commitMap size " + + commitWatcher.getCommitInfoMapSize() + " flushLength " + + flushPos + " numBuffers " + byteBufferList.size() + + " blockID " + blockID + " bufferPool size" + bufferPool + .getSize() + " currentBufferIndex " + bufferPool + .getCurrentBufferIndex()); + } // for standalone protocol, logIndex will always be 0. commitWatcher .updateCommitInfoMap(asyncReply.getLogIndex(), byteBufferList); } return e; }, responseExecutor).exceptionally(e -> { - LOG.debug( - "putBlock failed for blockID " + blockID + " with exception " + e - .getLocalizedMessage()); + if (LOG.isDebugEnabled()) { + LOG.debug( + "putBlock failed for blockID " + blockID + " with exception " + e + .getLocalizedMessage()); + } CompletionException ce = new CompletionException(e); setIoException(ce); throw ce; @@ -596,23 +589,19 @@ public boolean isClosed() { */ private void writeChunkToContainer(ByteBuffer chunk) throws IOException { int effectiveChunkSize = chunk.remaining(); - ByteString data = ByteStringHelper.getByteString(chunk); + ByteString data = bufferPool.byteStringConversion().apply(chunk); Checksum checksum = new Checksum(checksumType, bytesPerChecksum); ChecksumData checksumData = checksum.computeChecksum(chunk); ChunkInfo chunkInfo = ChunkInfo.newBuilder() - .setChunkName(DigestUtils.md5Hex(key) + "_stream_" + streamId + - "_chunk_" + ++chunkIndex) + .setChunkName(blockID.getLocalID() + "_chunk_" + ++chunkIndex) .setOffset(0) .setLen(effectiveChunkSize) .setChecksumData(checksumData.getProtoBufMessage()) .build(); - // generate a unique requestId - String requestId = - traceID + ContainerProtos.Type.WriteChunk + chunkIndex + chunkInfo - .getChunkName(); + try { XceiverClientReply asyncReply = - writeChunkAsync(xceiverClient, chunkInfo, blockID, data, requestId); + writeChunkAsync(xceiverClient, chunkInfo, blockID, data); CompletableFuture future = asyncReply.getResponse(); future.thenApplyAsync(e -> { @@ -623,9 +612,11 @@ private void writeChunkToContainer(ByteBuffer chunk) throws IOException { } return e; }, responseExecutor).exceptionally(e -> { - LOG.debug( - "writing chunk failed " + chunkInfo.getChunkName() + " blockID " - + blockID + " with exception " + e.getLocalizedMessage()); + if (LOG.isDebugEnabled()) { + LOG.debug( + "writing chunk failed " + chunkInfo.getChunkName() + " blockID " + + blockID + " with exception " + e.getLocalizedMessage()); + } CompletionException ce = new CompletionException(e); setIoException(ce); throw ce; @@ -634,9 +625,11 @@ private void writeChunkToContainer(ByteBuffer chunk) throws IOException { throw new IOException( "Unexpected Storage Container Exception: " + e.toString(), e); } - LOG.debug( - "writing chunk " + chunkInfo.getChunkName() + " blockID " + blockID - + " length " + effectiveChunkSize); + if (LOG.isDebugEnabled()) { + LOG.debug( + "writing chunk " + chunkInfo.getChunkName() + " blockID " + blockID + + " length " + effectiveChunkSize); + } containerBlockData.addChunks(chunkInfo); } diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BufferPool.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BufferPool.java index 17788c70a6c67..6d534579c8605 100644 --- a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BufferPool.java +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BufferPool.java @@ -19,10 +19,13 @@ package org.apache.hadoop.hdds.scm.storage; import com.google.common.base.Preconditions; +import org.apache.hadoop.hdds.scm.ByteStringConversion; +import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; import java.nio.ByteBuffer; import java.util.ArrayList; import java.util.List; +import java.util.function.Function; /** * This class creates and manages pool of n buffers. @@ -33,12 +36,24 @@ public class BufferPool { private int currentBufferIndex; private final int bufferSize; private final int capacity; + private final Function byteStringConversion; public BufferPool(int bufferSize, int capacity) { + this(bufferSize, capacity, + ByteStringConversion.createByteBufferConversion(null)); + } + + public BufferPool(int bufferSize, int capacity, + Function byteStringConversion){ this.capacity = capacity; this.bufferSize = bufferSize; bufferList = new ArrayList<>(capacity); currentBufferIndex = -1; + this.byteStringConversion = byteStringConversion; + } + + public Function byteStringConversion(){ + return byteStringConversion; } public ByteBuffer getCurrentBuffer() { diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ChunkInputStream.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ChunkInputStream.java new file mode 100644 index 0000000000000..f94d2d87340be --- /dev/null +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ChunkInputStream.java @@ -0,0 +1,544 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.scm.storage; + +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Preconditions; +import org.apache.hadoop.fs.Seekable; +import org.apache.hadoop.hdds.client.BlockID; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandRequestProto; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandResponseProto; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ChunkInfo; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ReadChunkResponseProto; +import org.apache.hadoop.hdds.scm.XceiverClientSpi; +import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException; +import org.apache.hadoop.ozone.common.Checksum; +import org.apache.hadoop.ozone.common.ChecksumData; +import org.apache.hadoop.ozone.common.OzoneChecksumException; +import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; + +import java.io.EOFException; +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.util.List; + +/** + * An {@link InputStream} called from BlockInputStream to read a chunk from the + * container. Each chunk may contain multiple underlying {@link ByteBuffer} + * instances. + */ +public class ChunkInputStream extends InputStream implements Seekable { + + private ChunkInfo chunkInfo; + private final long length; + private final BlockID blockID; + private XceiverClientSpi xceiverClient; + private boolean verifyChecksum; + private boolean allocated = false; + + // Buffer to store the chunk data read from the DN container + private List buffers; + + // Index of the buffers corresponding to the current position of the buffers + private int bufferIndex; + + // The offset of the current data residing in the buffers w.r.t the start + // of chunk data + private long bufferOffset; + + // The number of bytes of chunk data residing in the buffers currently + private long bufferLength; + + // Position of the ChunkInputStream is maintained by this variable (if a + // seek is performed. This position is w.r.t to the chunk only and not the + // block or key. This variable is set only if either the buffers are not + // yet allocated or the if the allocated buffers do not cover the seeked + // position. Once the chunk is read, this variable is reset. + private long chunkPosition = -1; + + private static final int EOF = -1; + + ChunkInputStream(ChunkInfo chunkInfo, BlockID blockId, + XceiverClientSpi xceiverClient, boolean verifyChecksum) { + this.chunkInfo = chunkInfo; + this.length = chunkInfo.getLen(); + this.blockID = blockId; + this.xceiverClient = xceiverClient; + this.verifyChecksum = verifyChecksum; + } + + public synchronized long getRemaining() throws IOException { + return length - getPos(); + } + + /** + * {@inheritDoc} + */ + @Override + public synchronized int read() throws IOException { + checkOpen(); + int available = prepareRead(1); + int dataout = EOF; + + if (available == EOF) { + // There is no more data in the chunk stream. The buffers should have + // been released by now + Preconditions.checkState(buffers == null); + } else { + dataout = Byte.toUnsignedInt(buffers.get(bufferIndex).get()); + } + + if (chunkStreamEOF()) { + // consumer might use getPos to determine EOF, + // so release buffers when serving the last byte of data + releaseBuffers(); + } + + return dataout; + } + + /** + * {@inheritDoc} + */ + @Override + public synchronized int read(byte[] b, int off, int len) throws IOException { + // According to the JavaDocs for InputStream, it is recommended that + // subclasses provide an override of bulk read if possible for performance + // reasons. In addition to performance, we need to do it for correctness + // reasons. The Ozone REST service uses PipedInputStream and + // PipedOutputStream to relay HTTP response data between a Jersey thread and + // a Netty thread. It turns out that PipedInputStream/PipedOutputStream + // have a subtle dependency (bug?) on the wrapped stream providing separate + // implementations of single-byte read and bulk read. Without this, get key + // responses might close the connection before writing all of the bytes + // advertised in the Content-Length. + if (b == null) { + throw new NullPointerException(); + } + if (off < 0 || len < 0 || len > b.length - off) { + throw new IndexOutOfBoundsException(); + } + if (len == 0) { + return 0; + } + checkOpen(); + int total = 0; + while (len > 0) { + int available = prepareRead(len); + if (available == EOF) { + // There is no more data in the chunk stream. The buffers should have + // been released by now + Preconditions.checkState(buffers == null); + return total != 0 ? total : EOF; + } + buffers.get(bufferIndex).get(b, off + total, available); + len -= available; + total += available; + } + + if (chunkStreamEOF()) { + // smart consumers determine EOF by calling getPos() + // so we release buffers when serving the final bytes of data + releaseBuffers(); + } + + return total; + } + + /** + * Seeks the ChunkInputStream to the specified position. This is done by + * updating the chunkPosition to the seeked position in case the buffers + * are not allocated or buffers do not contain the data corresponding to + * the seeked position (determined by buffersHavePosition()). Otherwise, + * the buffers position is updated to the seeked position. + */ + @Override + public synchronized void seek(long pos) throws IOException { + if (pos < 0 || pos >= length) { + if (pos == 0) { + // It is possible for length and pos to be zero in which case + // seek should return instead of throwing exception + return; + } + throw new EOFException("EOF encountered at pos: " + pos + " for chunk: " + + chunkInfo.getChunkName()); + } + + if (buffersHavePosition(pos)) { + // The bufferPosition is w.r.t the current chunk. + // Adjust the bufferIndex and position to the seeked position. + adjustBufferPosition(pos - bufferOffset); + } else { + chunkPosition = pos; + } + } + + @Override + public synchronized long getPos() throws IOException { + if (chunkPosition >= 0) { + return chunkPosition; + } + if (chunkStreamEOF()) { + return length; + } + if (buffersHaveData()) { + return bufferOffset + buffers.get(bufferIndex).position(); + } + if (buffersAllocated()) { + return bufferOffset + bufferLength; + } + return 0; + } + + @Override + public boolean seekToNewSource(long targetPos) throws IOException { + return false; + } + + @Override + public synchronized void close() { + if (xceiverClient != null) { + xceiverClient = null; + } + } + + /** + * Checks if the stream is open. If not, throw an exception. + * + * @throws IOException if stream is closed + */ + protected synchronized void checkOpen() throws IOException { + if (xceiverClient == null) { + throw new IOException("BlockInputStream has been closed."); + } + } + + /** + * Prepares to read by advancing through buffers or allocating new buffers, + * as needed until it finds data to return, or encounters EOF. + * @param len desired lenght of data to read + * @return length of data available to read, possibly less than desired length + */ + private synchronized int prepareRead(int len) throws IOException { + for (;;) { + if (chunkPosition >= 0) { + if (buffersHavePosition(chunkPosition)) { + // The current buffers have the seeked position. Adjust the buffer + // index and position to point to the chunkPosition. + adjustBufferPosition(chunkPosition - bufferOffset); + } else { + // Read a required chunk data to fill the buffers with seeked + // position data + readChunkFromContainer(len); + } + } + if (buffersHaveData()) { + // Data is available from buffers + ByteBuffer bb = buffers.get(bufferIndex); + return len > bb.remaining() ? bb.remaining() : len; + } else if (dataRemainingInChunk()) { + // There is more data in the chunk stream which has not + // been read into the buffers yet. + readChunkFromContainer(len); + } else { + // All available input from this chunk stream has been consumed. + return EOF; + } + } + } + + /** + * Reads full or partial Chunk from DN Container based on the current + * position of the ChunkInputStream, the number of bytes of data to read + * and the checksum boundaries. + * If successful, then the read data in saved in the buffers so that + * subsequent read calls can utilize it. + * @param len number of bytes of data to be read + * @throws IOException if there is an I/O error while performing the call + * to Datanode + */ + private synchronized void readChunkFromContainer(int len) throws IOException { + + // index of first byte to be read from the chunk + long startByteIndex; + if (chunkPosition >= 0) { + // If seek operation was called to advance the buffer position, the + // chunk should be read from that position onwards. + startByteIndex = chunkPosition; + } else { + // Start reading the chunk from the last chunkPosition onwards. + startByteIndex = bufferOffset + bufferLength; + } + + if (verifyChecksum) { + // Update the bufferOffset and bufferLength as per the checksum + // boundary requirement. + computeChecksumBoundaries(startByteIndex, len); + } else { + // Read from the startByteIndex + bufferOffset = startByteIndex; + bufferLength = len; + } + + // Adjust the chunkInfo so that only the required bytes are read from + // the chunk. + final ChunkInfo adjustedChunkInfo = ChunkInfo.newBuilder(chunkInfo) + .setOffset(bufferOffset) + .setLen(bufferLength) + .build(); + + ByteString byteString = readChunk(adjustedChunkInfo); + + buffers = byteString.asReadOnlyByteBufferList(); + bufferIndex = 0; + allocated = true; + + // If the stream was seeked to position before, then the buffer + // position should be adjusted as the reads happen at checksum boundaries. + // The buffers position might need to be adjusted for the following + // scenarios: + // 1. Stream was seeked to a position before the chunk was read + // 2. Chunk was read from index < the current position to account for + // checksum boundaries. + adjustBufferPosition(startByteIndex - bufferOffset); + } + + /** + * Send RPC call to get the chunk from the container. + */ + @VisibleForTesting + protected ByteString readChunk(ChunkInfo readChunkInfo) throws IOException { + ReadChunkResponseProto readChunkResponse; + + try { + List validators = + ContainerProtocolCalls.getValidatorList(); + validators.add(validator); + + readChunkResponse = ContainerProtocolCalls.readChunk(xceiverClient, + readChunkInfo, blockID, validators); + + } catch (IOException e) { + if (e instanceof StorageContainerException) { + throw e; + } + throw new IOException("Unexpected OzoneException: " + e.toString(), e); + } + + return readChunkResponse.getData(); + } + + private CheckedBiFunction validator = + (request, response) -> { + final ChunkInfo reqChunkInfo = + request.getReadChunk().getChunkData(); + + ReadChunkResponseProto readChunkResponse = response.getReadChunk(); + ByteString byteString = readChunkResponse.getData(); + + if (byteString.size() != reqChunkInfo.getLen()) { + // Bytes read from chunk should be equal to chunk size. + throw new OzoneChecksumException(String + .format("Inconsistent read for chunk=%s len=%d bytesRead=%d", + reqChunkInfo.getChunkName(), reqChunkInfo.getLen(), + byteString.size())); + } + + if (verifyChecksum) { + ChecksumData checksumData = ChecksumData.getFromProtoBuf( + chunkInfo.getChecksumData()); + + // ChecksumData stores checksum for each 'numBytesPerChecksum' + // number of bytes in a list. Compute the index of the first + // checksum to match with the read data + + int checkumStartIndex = (int) (reqChunkInfo.getOffset() / + checksumData.getBytesPerChecksum()); + Checksum.verifyChecksum( + byteString, checksumData, checkumStartIndex); + } + }; + + /** + * Return the offset and length of bytes that need to be read from the + * chunk file to cover the checksum boundaries covering the actual start and + * end of the chunk index to be read. + * For example, lets say the client is reading from index 120 to 450 in the + * chunk. And let's say checksum is stored for every 100 bytes in the chunk + * i.e. the first checksum is for bytes from index 0 to 99, the next for + * bytes from index 100 to 199 and so on. To verify bytes from 120 to 450, + * we would need to read from bytes 100 to 499 so that checksum + * verification can be done. + * + * @param startByteIndex the first byte index to be read by client + * @param dataLen number of bytes to be read from the chunk + */ + private void computeChecksumBoundaries(long startByteIndex, int dataLen) { + + int bytesPerChecksum = chunkInfo.getChecksumData().getBytesPerChecksum(); + // index of the last byte to be read from chunk, inclusively. + final long endByteIndex = startByteIndex + dataLen - 1; + + bufferOffset = (startByteIndex / bytesPerChecksum) + * bytesPerChecksum; // inclusive + final long endIndex = ((endByteIndex / bytesPerChecksum) + 1) + * bytesPerChecksum; // exclusive + bufferLength = Math.min(endIndex, length) - bufferOffset; + } + + /** + * Adjust the buffers position to account for seeked position and/ or checksum + * boundary reads. + * @param bufferPosition the position to which the buffers must be advanced + */ + private void adjustBufferPosition(long bufferPosition) { + // The bufferPosition is w.r.t the current chunk. + // Adjust the bufferIndex and position to the seeked chunkPosition. + long tempOffest = 0; + for (int i = 0; i < buffers.size(); i++) { + if (bufferPosition - tempOffest >= buffers.get(i).capacity()) { + tempOffest += buffers.get(i).capacity(); + } else { + bufferIndex = i; + break; + } + } + buffers.get(bufferIndex).position((int) (bufferPosition - tempOffest)); + + // Reset the chunkPosition as chunk stream has been initialized i.e. the + // buffers have been allocated. + resetPosition(); + } + + /** + * Check if the buffers have been allocated data and false otherwise. + */ + private boolean buffersAllocated() { + return buffers != null && !buffers.isEmpty(); + } + + /** + * Check if the buffers have any data remaining between the current + * position and the limit. + */ + private boolean buffersHaveData() { + boolean hasData = false; + + if (buffersAllocated()) { + while (bufferIndex < (buffers.size())) { + if (buffers.get(bufferIndex).hasRemaining()) { + // current buffer has data + hasData = true; + break; + } else { + if (buffersRemaining()) { + // move to next available buffer + ++bufferIndex; + Preconditions.checkState(bufferIndex < buffers.size()); + } else { + // no more buffers remaining + break; + } + } + } + } + + return hasData; + } + + private boolean buffersRemaining() { + return (bufferIndex < (buffers.size() - 1)); + } + + /** + * Check if curernt buffers have the data corresponding to the input position. + */ + private boolean buffersHavePosition(long pos) { + // Check if buffers have been allocated + if (buffersAllocated()) { + // Check if the current buffers cover the input position + return pos >= bufferOffset && + pos < bufferOffset + bufferLength; + } + return false; + } + + /** + * Check if there is more data in the chunk which has not yet been read + * into the buffers. + */ + private boolean dataRemainingInChunk() { + long bufferPos; + if (chunkPosition >= 0) { + bufferPos = chunkPosition; + } else { + bufferPos = bufferOffset + bufferLength; + } + + return bufferPos < length; + } + + /** + * Check if end of chunkStream has been reached. + */ + private boolean chunkStreamEOF() { + if (!allocated) { + // Chunk data has not been read yet + return false; + } + + if (buffersHaveData() || dataRemainingInChunk()) { + return false; + } else { + Preconditions.checkState(bufferOffset + bufferLength == length, + "EOF detected, but not at the last byte of the chunk"); + return true; + } + } + + /** + * If EOF is reached, release the buffers. + */ + private void releaseBuffers() { + buffers = null; + bufferIndex = 0; + } + + /** + * Reset the chunkPosition once the buffers are allocated. + */ + void resetPosition() { + this.chunkPosition = -1; + } + + String getChunkName() { + return chunkInfo.getChunkName(); + } + + protected long getLength() { + return length; + } + + @VisibleForTesting + protected long getChunkPosition() { + return chunkPosition; + } +} diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/CommitWatcher.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/CommitWatcher.java index d4606b514c461..1d9d55bfbfbb6 100644 --- a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/CommitWatcher.java +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/CommitWatcher.java @@ -131,7 +131,9 @@ public XceiverClientReply watchOnFirstIndex() throws IOException { long index = commitIndex2flushedDataMap.keySet().stream().mapToLong(v -> v).min() .getAsLong(); - LOG.debug("waiting for first index " + index + " to catch up"); + if (LOG.isDebugEnabled()) { + LOG.debug("waiting for first index " + index + " to catch up"); + } return watchForCommit(index); } else { return null; @@ -153,7 +155,9 @@ public XceiverClientReply watchOnLastIndex() long index = commitIndex2flushedDataMap.keySet().stream().mapToLong(v -> v).max() .getAsLong(); - LOG.debug("waiting for last flush Index " + index + " to catch up"); + if (LOG.isDebugEnabled()) { + LOG.debug("waiting for last flush Index " + index + " to catch up"); + } return watchForCommit(index); } else { return null; diff --git a/hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBlockInputStream.java b/hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBlockInputStream.java index 35c102257f4b4..042bfd941743e 100644 --- a/hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBlockInputStream.java +++ b/hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBlockInputStream.java @@ -1,32 +1,33 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with this - * work for additional information regarding copyright ownership. The ASF - * licenses this file to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - *

    - * http://www.apache.org/licenses/LICENSE-2.0 - *

    - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. */ + package org.apache.hadoop.hdds.scm.storage; +import com.google.common.primitives.Bytes; import org.apache.hadoop.hdds.client.BlockID; import org.apache.hadoop.hdds.client.ContainerBlockID; -import org.apache.hadoop.hdds.protocol.DatanodeDetails; -import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos - .ChecksumData; -import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos - .ChecksumType; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ChecksumType; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ChunkInfo; import org.apache.hadoop.hdds.scm.XceiverClientManager; -import org.apache.hadoop.hdds.scm.XceiverClientSpi; -import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; +import org.apache.hadoop.hdds.scm.pipeline.Pipeline; +import org.apache.hadoop.hdds.security.token.OzoneBlockTokenIdentifier; +import org.apache.hadoop.ozone.common.Checksum; +import org.apache.hadoop.security.token.Token; import org.junit.Assert; import org.junit.Before; import org.junit.Test; @@ -34,107 +35,126 @@ import java.io.EOFException; import java.io.IOException; import java.util.ArrayList; +import java.util.HashMap; import java.util.List; +import java.util.Map; import java.util.Random; -import java.util.UUID; + +import static org.apache.hadoop.hdds.scm.storage.TestChunkInputStream.generateRandomData; /** - * Tests {@link BlockInputStream}. + * Tests for {@link BlockInputStream}'s functionality. */ public class TestBlockInputStream { - private static BlockInputStream blockInputStream; - private static List chunks; - private static int blockSize; + private static final int CHUNK_SIZE = 100; + private static Checksum checksum; - private static final int CHUNK_SIZE = 20; + private BlockInputStream blockStream; + private byte[] blockData; + private int blockSize; + private List chunks; + private Map chunkDataMap; @Before public void setup() throws Exception { BlockID blockID = new BlockID(new ContainerBlockID(1, 1)); - chunks = createChunkList(10); - String traceID = UUID.randomUUID().toString(); - blockInputStream = new DummyBlockInputStream(blockID, null, null, chunks, - traceID, false, 0); - - blockSize = 0; - for (ChunkInfo chunk : chunks) { - blockSize += chunk.getLen(); - } + checksum = new Checksum(ChecksumType.NONE, CHUNK_SIZE); + createChunkList(5); + + blockStream = new DummyBlockInputStream(blockID, blockSize, null, null, + false, null); } /** * Create a mock list of chunks. The first n-1 chunks of length CHUNK_SIZE * and the last chunk with length CHUNK_SIZE/2. - * @param numChunks - * @return */ - private static List createChunkList(int numChunks) { - ChecksumData dummyChecksumData = ChecksumData.newBuilder() - .setType(ChecksumType.NONE) - .setBytesPerChecksum(100) - .build(); - List chunkList = new ArrayList<>(numChunks); - int i; - for (i = 0; i < numChunks - 1; i++) { - String chunkName = "chunk-" + i; + private void createChunkList(int numChunks) + throws Exception { + + chunks = new ArrayList<>(numChunks); + chunkDataMap = new HashMap<>(); + blockData = new byte[0]; + int i, chunkLen; + byte[] byteData; + String chunkName; + + for (i = 0; i < numChunks; i++) { + chunkName = "chunk-" + i; + chunkLen = CHUNK_SIZE; + if (i == numChunks - 1) { + chunkLen = CHUNK_SIZE / 2; + } + byteData = generateRandomData(chunkLen); ChunkInfo chunkInfo = ChunkInfo.newBuilder() .setChunkName(chunkName) .setOffset(0) - .setLen(CHUNK_SIZE) - .setChecksumData(dummyChecksumData) + .setLen(chunkLen) + .setChecksumData(checksum.computeChecksum( + byteData, 0, chunkLen).getProtoBufMessage()) .build(); - chunkList.add(chunkInfo); + + chunkDataMap.put(chunkName, byteData); + chunks.add(chunkInfo); + + blockSize += chunkLen; + blockData = Bytes.concat(blockData, byteData); } - ChunkInfo chunkInfo = ChunkInfo.newBuilder() - .setChunkName("chunk-" + i) - .setOffset(0) - .setLen(CHUNK_SIZE/2) - .setChecksumData(dummyChecksumData) - .build(); - chunkList.add(chunkInfo); - - return chunkList; } /** - * A dummy BlockInputStream to test the functionality of BlockInputStream. + * A dummy BlockInputStream to mock read block call to DN. */ - private static class DummyBlockInputStream extends BlockInputStream { + private class DummyBlockInputStream extends BlockInputStream { - DummyBlockInputStream(BlockID blockID, - XceiverClientManager xceiverClientManager, - XceiverClientSpi xceiverClient, - List chunks, - String traceID, + DummyBlockInputStream(BlockID blockId, + long blockLen, + Pipeline pipeline, + Token token, boolean verifyChecksum, - long initialPosition) throws IOException { - super(blockID, xceiverClientManager, xceiverClient, chunks, traceID, - verifyChecksum, initialPosition); + XceiverClientManager xceiverClientManager) { + super(blockId, blockLen, pipeline, token, verifyChecksum, + xceiverClientManager); } @Override - protected ByteString readChunk(final ChunkInfo chunkInfo, - List excludeDns, List dnListFromReply) - throws IOException { - return getByteString(chunkInfo.getChunkName(), (int) chunkInfo.getLen()); + protected List getChunkInfos() { + return chunks; } @Override - protected List getDatanodeList() { - // return an empty dummy list of size 10 - return new ArrayList<>(10); + protected void addStream(ChunkInfo chunkInfo) { + TestChunkInputStream testChunkInputStream = new TestChunkInputStream(); + getChunkStreams().add(testChunkInputStream.new DummyChunkInputStream( + chunkInfo, null, null, false, + chunkDataMap.get(chunkInfo.getChunkName()).clone())); } - /** - * Create ByteString with the input data to return when a readChunk call is - * placed. - */ - private static ByteString getByteString(String data, int length) { - while (data.length() < length) { - data = data + "0"; - } - return ByteString.copyFrom(data.getBytes(), 0, length); + @Override + protected synchronized void checkOpen() throws IOException { + // No action needed + } + } + + private void seekAndVerify(int pos) throws Exception { + blockStream.seek(pos); + Assert.assertEquals("Current position of buffer does not match with the " + + "seeked position", pos, blockStream.getPos()); + } + + /** + * Match readData with the chunkData byte-wise. + * @param readData Data read through ChunkInputStream + * @param inputDataStartIndex first index (inclusive) in chunkData to compare + * with read data + * @param length the number of bytes of data to match starting from + * inputDataStartIndex + */ + private void matchWithInputData(byte[] readData, int inputDataStartIndex, + int length) { + for (int i = inputDataStartIndex; i < inputDataStartIndex + length; i++) { + Assert.assertEquals(blockData[i], readData[i - inputDataStartIndex]); } } @@ -144,17 +164,26 @@ public void testSeek() throws Exception { int pos = 0; seekAndVerify(pos); Assert.assertEquals("ChunkIndex is incorrect", 0, - blockInputStream.getChunkIndex()); + blockStream.getChunkIndex()); + // Before BlockInputStream is initialized (initialization happens during + // read operation), seek should update the BlockInputStream#blockPosition pos = CHUNK_SIZE; seekAndVerify(pos); + Assert.assertEquals("ChunkIndex is incorrect", 0, + blockStream.getChunkIndex()); + Assert.assertEquals(pos, blockStream.getBlockPosition()); + + // Initialize the BlockInputStream. After initializtion, the chunkIndex + // should be updated to correspond to the seeked position. + blockStream.initialize(); Assert.assertEquals("ChunkIndex is incorrect", 1, - blockInputStream.getChunkIndex()); + blockStream.getChunkIndex()); - pos = (CHUNK_SIZE * 5) + 5; + pos = (CHUNK_SIZE * 4) + 5; seekAndVerify(pos); - Assert.assertEquals("ChunkIndex is incorrect", 5, - blockInputStream.getChunkIndex()); + Assert.assertEquals("ChunkIndex is incorrect", 4, + blockStream.getChunkIndex()); try { // Try seeking beyond the blockSize. @@ -162,7 +191,7 @@ public void testSeek() throws Exception { seekAndVerify(pos); Assert.fail("Seek to position beyond block size should fail."); } catch (EOFException e) { - // Expected + System.out.println(e); } // Seek to random positions between 0 and the block size. @@ -174,20 +203,32 @@ public void testSeek() throws Exception { } @Test - public void testBlockEOF() throws Exception { - // Seek to some position < blockSize and verify EOF is not reached. - seekAndVerify(CHUNK_SIZE); - Assert.assertFalse(blockInputStream.blockStreamEOF()); - - // Seek to blockSize-1 and verify that EOF is not reached as the chunk - // has not been read from container yet. - seekAndVerify(blockSize-1); - Assert.assertFalse(blockInputStream.blockStreamEOF()); + public void testRead() throws Exception { + // read 200 bytes of data starting from position 50. Chunk0 contains + // indices 0 to 99, chunk1 from 100 to 199 and chunk3 from 200 to 299. So + // the read should result in 3 ChunkInputStream reads + seekAndVerify(50); + byte[] b = new byte[200]; + blockStream.read(b, 0, 200); + matchWithInputData(b, 50, 200); + + // The new position of the blockInputStream should be the last index read + // + 1. + Assert.assertEquals(250, blockStream.getPos()); + Assert.assertEquals(2, blockStream.getChunkIndex()); } - private void seekAndVerify(int pos) throws Exception { - blockInputStream.seek(pos); - Assert.assertEquals("Current position of buffer does not match with the " + - "seeked position", pos, blockInputStream.getPos()); + @Test + public void testSeekAndRead() throws Exception { + // Seek to a position and read data + seekAndVerify(50); + byte[] b1 = new byte[100]; + blockStream.read(b1, 0, 100); + matchWithInputData(b1, 50, 100); + + // Next read should start from the position of the last read + 1 i.e. 100 + byte[] b2 = new byte[100]; + blockStream.read(b2, 0, 100); + matchWithInputData(b2, 150, 100); } } diff --git a/hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestChunkInputStream.java b/hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestChunkInputStream.java new file mode 100644 index 0000000000000..a5fe26b5619ab --- /dev/null +++ b/hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestChunkInputStream.java @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.scm.storage; + +import org.apache.hadoop.hdds.client.BlockID; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ChecksumType; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ChunkInfo; +import org.apache.hadoop.hdds.scm.XceiverClientSpi; +import org.apache.hadoop.ozone.OzoneConfigKeys; +import org.apache.hadoop.ozone.common.Checksum; +import org.apache.hadoop.test.GenericTestUtils; +import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.io.EOFException; +import java.util.ArrayList; +import java.util.List; +import java.util.Random; + +/** + * Tests for {@link ChunkInputStream}'s functionality. + */ +public class TestChunkInputStream { + + private static final int CHUNK_SIZE = 100; + private static final int BYTES_PER_CHECKSUM = 20; + private static final String CHUNK_NAME = "dummyChunk"; + private static final Random RANDOM = new Random(); + private static Checksum checksum; + + private DummyChunkInputStream chunkStream; + private ChunkInfo chunkInfo; + private byte[] chunkData; + + @Before + public void setup() throws Exception { + checksum = new Checksum(ChecksumType.valueOf( + OzoneConfigKeys.OZONE_CLIENT_CHECKSUM_TYPE_DEFAULT), + BYTES_PER_CHECKSUM); + + chunkData = generateRandomData(CHUNK_SIZE); + + chunkInfo = ChunkInfo.newBuilder() + .setChunkName(CHUNK_NAME) + .setOffset(0) + .setLen(CHUNK_SIZE) + .setChecksumData(checksum.computeChecksum( + chunkData, 0, CHUNK_SIZE).getProtoBufMessage()) + .build(); + + chunkStream = new DummyChunkInputStream(chunkInfo, null, null, true); + } + + static byte[] generateRandomData(int length) { + byte[] bytes = new byte[length]; + RANDOM.nextBytes(bytes); + return bytes; + } + + /** + * A dummy ChunkInputStream to mock read chunk calls to DN. + */ + public class DummyChunkInputStream extends ChunkInputStream { + + // Stores the read chunk data in each readChunk call + private List readByteBuffers = new ArrayList<>(); + + DummyChunkInputStream(ChunkInfo chunkInfo, + BlockID blockId, + XceiverClientSpi xceiverClient, + boolean verifyChecksum) { + super(chunkInfo, blockId, xceiverClient, verifyChecksum); + } + + public DummyChunkInputStream(ChunkInfo chunkInfo, + BlockID blockId, + XceiverClientSpi xceiverClient, + boolean verifyChecksum, + byte[] data) { + super(chunkInfo, blockId, xceiverClient, verifyChecksum); + chunkData = data; + } + + @Override + protected ByteString readChunk(ChunkInfo readChunkInfo) { + ByteString byteString = ByteString.copyFrom(chunkData, + (int) readChunkInfo.getOffset(), + (int) readChunkInfo.getLen()); + readByteBuffers.add(byteString); + return byteString; + } + + @Override + protected void checkOpen() { + // No action needed + } + } + + /** + * Match readData with the chunkData byte-wise. + * @param readData Data read through ChunkInputStream + * @param inputDataStartIndex first index (inclusive) in chunkData to compare + * with read data + * @param length the number of bytes of data to match starting from + * inputDataStartIndex + */ + private void matchWithInputData(byte[] readData, int inputDataStartIndex, + int length) { + for (int i = inputDataStartIndex; i < inputDataStartIndex + length; i++) { + Assert.assertEquals(chunkData[i], readData[i - inputDataStartIndex]); + } + } + + /** + * Seek to a position and verify through getPos(). + */ + private void seekAndVerify(int pos) throws Exception { + chunkStream.seek(pos); + Assert.assertEquals("Current position of buffer does not match with the " + + "seeked position", pos, chunkStream.getPos()); + } + + @Test + public void testFullChunkRead() throws Exception { + byte[] b = new byte[CHUNK_SIZE]; + chunkStream.read(b, 0, CHUNK_SIZE); + + matchWithInputData(b, 0, CHUNK_SIZE); + } + + @Test + public void testPartialChunkRead() throws Exception { + int len = CHUNK_SIZE / 2; + byte[] b = new byte[len]; + + chunkStream.read(b, 0, len); + + matchWithInputData(b, 0, len); + + // To read chunk data from index 0 to 49 (len = 50), we need to read + // chunk from offset 0 to 60 as the checksum boundary is at every 20 + // bytes. Verify that 60 bytes of chunk data are read and stored in the + // buffers. + matchWithInputData(chunkStream.readByteBuffers.get(0).toByteArray(), + 0, 60); + + } + + @Test + public void testSeek() throws Exception { + seekAndVerify(0); + + try { + seekAndVerify(CHUNK_SIZE); + Assert.fail("Seeking to Chunk Length should fail."); + } catch (EOFException e) { + GenericTestUtils.assertExceptionContains("EOF encountered at pos: " + + CHUNK_SIZE + " for chunk: " + CHUNK_NAME, e); + } + + // Seek before read should update the ChunkInputStream#chunkPosition + seekAndVerify(25); + Assert.assertEquals(25, chunkStream.getChunkPosition()); + + // Read from the seeked position. + // Reading from index 25 to 54 should result in the ChunkInputStream + // copying chunk data from index 20 to 59 into the buffers (checksum + // boundaries). + byte[] b = new byte[30]; + chunkStream.read(b, 0, 30); + matchWithInputData(b, 25, 30); + matchWithInputData(chunkStream.readByteBuffers.get(0).toByteArray(), + 20, 40); + + // After read, the position of the chunkStream is evaluated from the + // buffers and the chunkPosition should be reset to -1. + Assert.assertEquals(-1, chunkStream.getChunkPosition()); + + // Seek to a position within the current buffers. Current buffers contain + // data from index 20 to 59. ChunkPosition should still not be used to + // set the position. + seekAndVerify(35); + Assert.assertEquals(-1, chunkStream.getChunkPosition()); + + // Seek to a position outside the current buffers. In this case, the + // chunkPosition should be updated to the seeked position. + seekAndVerify(75); + Assert.assertEquals(75, chunkStream.getChunkPosition()); + } + + @Test + public void testSeekAndRead() throws Exception { + // Seek to a position and read data + seekAndVerify(50); + byte[] b1 = new byte[20]; + chunkStream.read(b1, 0, 20); + matchWithInputData(b1, 50, 20); + + // Next read should start from the position of the last read + 1 i.e. 70 + byte[] b2 = new byte[20]; + chunkStream.read(b2, 0, 20); + matchWithInputData(b2, 70, 20); + } +} \ No newline at end of file diff --git a/hadoop-hdds/common/dev-support/findbugsExcludeFile.xml b/hadoop-hdds/common/dev-support/findbugsExcludeFile.xml index c7db6794cc0e0..4441b69d8683e 100644 --- a/hadoop-hdds/common/dev-support/findbugsExcludeFile.xml +++ b/hadoop-hdds/common/dev-support/findbugsExcludeFile.xml @@ -25,4 +25,9 @@ + + + + + diff --git a/hadoop-hdds/common/pom.xml b/hadoop-hdds/common/pom.xml index 51560ca3af92d..9af807f8b9eb0 100644 --- a/hadoop-hdds/common/pom.xml +++ b/hadoop-hdds/common/pom.xml @@ -15,7 +15,7 @@ +https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -88,7 +88,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> org.rocksdb rocksdbjni - 5.14.2 + 6.0.1 org.apache.hadoop @@ -135,7 +135,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> io.jaegertracing jaeger-client - 0.33.1 + ${jaeger.version} io.opentracing @@ -258,9 +258,6 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> ${protobuf.version} ${protoc.path} - - ${basedir}/../../hadoop-common-project/hadoop-common/src/main/proto - ${basedir}/src/main/proto @@ -277,8 +274,8 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> - org.codehaus.mojo - findbugs-maven-plugin + com.github.spotbugs + spotbugs-maven-plugin ${basedir}/dev-support/findbugsExcludeFile.xml diff --git a/hadoop-hdds/common/src/main/bin/hadoop-config.cmd b/hadoop-hdds/common/src/main/bin/hadoop-config.cmd new file mode 100644 index 0000000000000..d77dc5346a1fc --- /dev/null +++ b/hadoop-hdds/common/src/main/bin/hadoop-config.cmd @@ -0,0 +1,317 @@ +@echo off +@rem Licensed to the Apache Software Foundation (ASF) under one or more +@rem contributor license agreements. See the NOTICE file distributed with +@rem this work for additional information regarding copyright ownership. +@rem The ASF licenses this file to You under the Apache License, Version 2.0 +@rem (the "License"); you may not use this file except in compliance with +@rem the License. You may obtain a copy of the License at +@rem +@rem http://www.apache.org/licenses/LICENSE-2.0 +@rem +@rem Unless required by applicable law or agreed to in writing, software +@rem distributed under the License is distributed on an "AS IS" BASIS, +@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +@rem See the License for the specific language governing permissions and +@rem limitations under the License. + +@rem included in all the hadoop scripts with source command +@rem should not be executable directly +@rem also should not be passed any arguments, since we need original %* + +if not defined HADOOP_COMMON_DIR ( + set HADOOP_COMMON_DIR=share\hadoop\common +) +if not defined HADOOP_COMMON_LIB_JARS_DIR ( + set HADOOP_COMMON_LIB_JARS_DIR=share\hadoop\common\lib +) +if not defined HADOOP_COMMON_LIB_NATIVE_DIR ( + set HADOOP_COMMON_LIB_NATIVE_DIR=lib\native +) +if not defined HDFS_DIR ( + set HDFS_DIR=share\hadoop\hdfs +) +if not defined HDFS_LIB_JARS_DIR ( + set HDFS_LIB_JARS_DIR=share\hadoop\hdfs\lib +) +if not defined YARN_DIR ( + set YARN_DIR=share\hadoop\yarn +) +if not defined YARN_LIB_JARS_DIR ( + set YARN_LIB_JARS_DIR=share\hadoop\yarn\lib +) +if not defined MAPRED_DIR ( + set MAPRED_DIR=share\hadoop\mapreduce +) +if not defined MAPRED_LIB_JARS_DIR ( + set MAPRED_LIB_JARS_DIR=share\hadoop\mapreduce\lib +) + +@rem the root of the Hadoop installation +set HADOOP_HOME=%~dp0 +for %%i in (%HADOOP_HOME%.) do ( + set HADOOP_HOME=%%~dpi +) +if "%HADOOP_HOME:~-1%" == "\" ( + set HADOOP_HOME=%HADOOP_HOME:~0,-1% +) + +if not exist %HADOOP_HOME%\share\hadoop\common\hadoop-common-*.jar ( + @echo +================================================================+ + @echo ^| Error: HADOOP_HOME is not set correctly ^| + @echo +----------------------------------------------------------------+ + @echo ^| Please set your HADOOP_HOME variable to the absolute path of ^| + @echo ^| the directory that contains the hadoop distribution ^| + @echo +================================================================+ + exit /b 1 +) + +if not defined HADOOP_CONF_DIR ( + set HADOOP_CONF_DIR=%HADOOP_HOME%\etc\hadoop +) + +@rem +@rem Allow alternate conf dir location. +@rem + +if "%1" == "--config" ( + set HADOOP_CONF_DIR=%2 + shift + shift +) + +@rem +@rem check to see it is specified whether to use the workers or the +@rem masters file +@rem + +if "%1" == "--hosts" ( + set HADOOP_WORKERS=%HADOOP_CONF_DIR%\%2 + shift + shift +) + +@rem +@rem Set log level. Default to INFO. +@rem + +if "%1" == "--loglevel" ( + set HADOOP_LOGLEVEL=%2 + shift + shift +) + +if exist %HADOOP_CONF_DIR%\hadoop-env.cmd ( + call %HADOOP_CONF_DIR%\hadoop-env.cmd +) + +@rem +@rem setup java environment variables +@rem + +if not defined JAVA_HOME ( + echo Error: JAVA_HOME is not set. + goto :eof +) + +if not exist %JAVA_HOME%\bin\java.exe ( + echo Error: JAVA_HOME is incorrectly set. + echo Please update %HADOOP_CONF_DIR%\hadoop-env.cmd + goto :eof +) + +set JAVA=%JAVA_HOME%\bin\java +@rem some Java parameters +set JAVA_HEAP_MAX=-Xmx1000m + +@rem +@rem check envvars which might override default args +@rem + +if defined HADOOP_HEAPSIZE ( + set JAVA_HEAP_MAX=-Xmx%HADOOP_HEAPSIZE%m +) + +@rem +@rem CLASSPATH initially contains %HADOOP_CONF_DIR% +@rem + +set CLASSPATH=%HADOOP_CONF_DIR% + +if not defined HADOOP_COMMON_HOME ( + if exist %HADOOP_HOME%\share\hadoop\common ( + set HADOOP_COMMON_HOME=%HADOOP_HOME% + ) +) + +@rem +@rem for releases, add core hadoop jar & webapps to CLASSPATH +@rem + +if exist %HADOOP_COMMON_HOME%\%HADOOP_COMMON_DIR%\webapps ( + set CLASSPATH=!CLASSPATH!;%HADOOP_COMMON_HOME%\%HADOOP_COMMON_DIR% +) + +if exist %HADOOP_COMMON_HOME%\%HADOOP_COMMON_LIB_JARS_DIR% ( + set CLASSPATH=!CLASSPATH!;%HADOOP_COMMON_HOME%\%HADOOP_COMMON_LIB_JARS_DIR%\* +) + +set CLASSPATH=!CLASSPATH!;%HADOOP_COMMON_HOME%\%HADOOP_COMMON_DIR%\* + +@rem +@rem default log directory % file +@rem + +if not defined HADOOP_LOG_DIR ( + set HADOOP_LOG_DIR=%HADOOP_HOME%\logs +) + +if not defined HADOOP_LOGFILE ( + set HADOOP_LOGFILE=hadoop.log +) + +if not defined HADOOP_LOGLEVEL ( + set HADOOP_LOGLEVEL=INFO +) + +if not defined HADOOP_ROOT_LOGGER ( + set HADOOP_ROOT_LOGGER=%HADOOP_LOGLEVEL%,console +) + +@rem +@rem default policy file for service-level authorization +@rem + +if not defined HADOOP_POLICYFILE ( + set HADOOP_POLICYFILE=hadoop-policy.xml +) + +@rem +@rem Determine the JAVA_PLATFORM +@rem + +for /f "delims=" %%A in ('%JAVA% -Xmx32m %HADOOP_JAVA_PLATFORM_OPTS% -classpath "%CLASSPATH%" org.apache.hadoop.util.PlatformName') do set JAVA_PLATFORM=%%A +@rem replace space with underscore +set JAVA_PLATFORM=%JAVA_PLATFORM: =_% + +@rem +@rem setup 'java.library.path' for native hadoop code if necessary +@rem + +@rem Check if we're running hadoop directly from the build +if exist %HADOOP_COMMON_HOME%\target\bin ( + if defined JAVA_LIBRARY_PATH ( + set JAVA_LIBRARY_PATH=%JAVA_LIBRARY_PATH%;%HADOOP_COMMON_HOME%\target\bin + ) else ( + set JAVA_LIBRARY_PATH=%HADOOP_COMMON_HOME%\target\bin + ) +) + +@rem For the distro case, check the bin folder +if exist %HADOOP_COMMON_HOME%\bin ( + if defined JAVA_LIBRARY_PATH ( + set JAVA_LIBRARY_PATH=%JAVA_LIBRARY_PATH%;%HADOOP_COMMON_HOME%\bin + ) else ( + set JAVA_LIBRARY_PATH=%HADOOP_COMMON_HOME%\bin + ) +) + +@rem +@rem setup a default TOOL_PATH +@rem +set TOOL_PATH=%HADOOP_HOME%\share\hadoop\tools\lib\* + +set HADOOP_OPTS=%HADOOP_OPTS% -Dhadoop.log.dir=%HADOOP_LOG_DIR% +set HADOOP_OPTS=%HADOOP_OPTS% -Dhadoop.log.file=%HADOOP_LOGFILE% +set HADOOP_OPTS=%HADOOP_OPTS% -Dhadoop.home.dir=%HADOOP_HOME% +set HADOOP_OPTS=%HADOOP_OPTS% -Dhadoop.id.str=%HADOOP_IDENT_STRING% +set HADOOP_OPTS=%HADOOP_OPTS% -Dhadoop.root.logger=%HADOOP_ROOT_LOGGER% + +if defined JAVA_LIBRARY_PATH ( + set HADOOP_OPTS=%HADOOP_OPTS% -Djava.library.path=%JAVA_LIBRARY_PATH% +) +set HADOOP_OPTS=%HADOOP_OPTS% -Dhadoop.policy.file=%HADOOP_POLICYFILE% + +@rem +@rem Disable ipv6 as it can cause issues +@rem + +set HADOOP_OPTS=%HADOOP_OPTS% -Djava.net.preferIPv4Stack=true + +@rem +@rem put hdfs in classpath if present +@rem + +if not defined HADOOP_HDFS_HOME ( + if exist %HADOOP_HOME%\%HDFS_DIR% ( + set HADOOP_HDFS_HOME=%HADOOP_HOME% + ) +) + +if exist %HADOOP_HDFS_HOME%\%HDFS_DIR%\webapps ( + set CLASSPATH=!CLASSPATH!;%HADOOP_HDFS_HOME%\%HDFS_DIR% +) + +if exist %HADOOP_HDFS_HOME%\%HDFS_LIB_JARS_DIR% ( + set CLASSPATH=!CLASSPATH!;%HADOOP_HDFS_HOME%\%HDFS_LIB_JARS_DIR%\* +) + +set CLASSPATH=!CLASSPATH!;%HADOOP_HDFS_HOME%\%HDFS_DIR%\* + +@rem +@rem put yarn in classpath if present +@rem + +if not defined HADOOP_YARN_HOME ( + if exist %HADOOP_HOME%\%YARN_DIR% ( + set HADOOP_YARN_HOME=%HADOOP_HOME% + ) +) + +if exist %HADOOP_YARN_HOME%\%YARN_DIR%\webapps ( + set CLASSPATH=!CLASSPATH!;%HADOOP_YARN_HOME%\%YARN_DIR% +) + +if exist %HADOOP_YARN_HOME%\%YARN_LIB_JARS_DIR% ( + set CLASSPATH=!CLASSPATH!;%HADOOP_YARN_HOME%\%YARN_LIB_JARS_DIR%\* +) + +set CLASSPATH=!CLASSPATH!;%HADOOP_YARN_HOME%\%YARN_DIR%\* + +@rem +@rem put mapred in classpath if present AND different from YARN +@rem + +if not defined HADOOP_MAPRED_HOME ( + if exist %HADOOP_HOME%\%MAPRED_DIR% ( + set HADOOP_MAPRED_HOME=%HADOOP_HOME% + ) +) + +if not "%HADOOP_MAPRED_HOME%\%MAPRED_DIR%" == "%HADOOP_YARN_HOME%\%YARN_DIR%" ( + + if exist %HADOOP_MAPRED_HOME%\%MAPRED_DIR%\webapps ( + set CLASSPATH=!CLASSPATH!;%HADOOP_MAPRED_HOME%\%MAPRED_DIR% + ) + + if exist %HADOOP_MAPRED_HOME%\%MAPRED_LIB_JARS_DIR% ( + set CLASSPATH=!CLASSPATH!;%HADOOP_MAPRED_HOME%\%MAPRED_LIB_JARS_DIR%\* + ) + + set CLASSPATH=!CLASSPATH!;%HADOOP_MAPRED_HOME%\%MAPRED_DIR%\* +) + +@rem +@rem add user-specified CLASSPATH last +@rem + +if defined HADOOP_CLASSPATH ( + if not defined HADOOP_USE_CLIENT_CLASSLOADER ( + if defined HADOOP_USER_CLASSPATH_FIRST ( + set CLASSPATH=%HADOOP_CLASSPATH%;%CLASSPATH%; + ) else ( + set CLASSPATH=%CLASSPATH%;%HADOOP_CLASSPATH%; + ) + ) +) + +:eof diff --git a/hadoop-hdds/common/src/main/bin/hadoop-config.sh b/hadoop-hdds/common/src/main/bin/hadoop-config.sh new file mode 100755 index 0000000000000..444b79a362953 --- /dev/null +++ b/hadoop-hdds/common/src/main/bin/hadoop-config.sh @@ -0,0 +1,165 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#### +# IMPORTANT +#### + +## The hadoop-config.sh tends to get executed by non-Hadoop scripts. +## Those parts expect this script to parse/manipulate $@. In order +## to maintain backward compatibility, this means a surprising +## lack of functions for bits that would be much better off in +## a function. +## +## In other words, yes, there is some bad things happen here and +## unless we break the rest of the ecosystem, we can't change it. :( + + +# included in all the hadoop scripts with source command +# should not be executable directly +# also should not be passed any arguments, since we need original $* +# +# after doing more config, caller should also exec finalize +# function to finish last minute/default configs for +# settings that might be different between daemons & interactive + +# you must be this high to ride the ride +if [[ -z "${BASH_VERSINFO[0]}" ]] \ + || [[ "${BASH_VERSINFO[0]}" -lt 3 ]] \ + || [[ "${BASH_VERSINFO[0]}" -eq 3 && "${BASH_VERSINFO[1]}" -lt 2 ]]; then + echo "bash v3.2+ is required. Sorry." + exit 1 +fi + +# In order to get partially bootstrapped, we need to figure out where +# we are located. Chances are good that our caller has already done +# this work for us, but just in case... + +if [[ -z "${HADOOP_LIBEXEC_DIR}" ]]; then + _hadoop_common_this="${BASH_SOURCE-$0}" + HADOOP_LIBEXEC_DIR=$(cd -P -- "$(dirname -- "${_hadoop_common_this}")" >/dev/null && pwd -P) +fi + +# get our functions defined for usage later +if [[ -n "${HADOOP_COMMON_HOME}" ]] && + [[ -e "${HADOOP_COMMON_HOME}/libexec/hadoop-functions.sh" ]]; then + # shellcheck source=./hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh + . "${HADOOP_COMMON_HOME}/libexec/hadoop-functions.sh" +elif [[ -e "${HADOOP_LIBEXEC_DIR}/hadoop-functions.sh" ]]; then + # shellcheck source=./hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh + . "${HADOOP_LIBEXEC_DIR}/hadoop-functions.sh" +else + echo "ERROR: Unable to exec ${HADOOP_LIBEXEC_DIR}/hadoop-functions.sh." 1>&2 + exit 1 +fi + +hadoop_deprecate_envvar HADOOP_PREFIX HADOOP_HOME + +# allow overrides of the above and pre-defines of the below +if [[ -n "${HADOOP_COMMON_HOME}" ]] && + [[ -e "${HADOOP_COMMON_HOME}/libexec/hadoop-layout.sh" ]]; then + # shellcheck source=./hadoop-common-project/hadoop-common/src/main/bin/hadoop-layout.sh.example + . "${HADOOP_COMMON_HOME}/libexec/hadoop-layout.sh" +elif [[ -e "${HADOOP_LIBEXEC_DIR}/hadoop-layout.sh" ]]; then + # shellcheck source=./hadoop-common-project/hadoop-common/src/main/bin/hadoop-layout.sh.example + . "${HADOOP_LIBEXEC_DIR}/hadoop-layout.sh" +fi + +# +# IMPORTANT! We are not executing user provided code yet! +# + +# Let's go! Base definitions so we can move forward +hadoop_bootstrap + +# let's find our conf. +# +# first, check and process params passed to us +# we process this in-line so that we can directly modify $@ +# if something downstream is processing that directly, +# we need to make sure our params have been ripped out +# note that we do many of them here for various utilities. +# this provides consistency and forces a more consistent +# user experience + + +# save these off in case our caller needs them +# shellcheck disable=SC2034 +HADOOP_USER_PARAMS=("$@") + +hadoop_parse_args "$@" +shift "${HADOOP_PARSE_COUNTER}" + +# +# Setup the base-line environment +# +hadoop_find_confdir +hadoop_exec_hadoopenv +hadoop_import_shellprofiles +hadoop_exec_userfuncs + +# +# IMPORTANT! User provided code is now available! +# + +hadoop_exec_user_hadoopenv +hadoop_verify_confdir + +hadoop_deprecate_envvar HADOOP_SLAVES HADOOP_WORKERS +hadoop_deprecate_envvar HADOOP_SLAVE_NAMES HADOOP_WORKER_NAMES +hadoop_deprecate_envvar HADOOP_SLAVE_SLEEP HADOOP_WORKER_SLEEP + +# do all the OS-specific startup bits here +# this allows us to get a decent JAVA_HOME, +# call crle for LD_LIBRARY_PATH, etc. +hadoop_os_tricks + +hadoop_java_setup + +hadoop_basic_init + +# inject any sub-project overrides, defaults, etc. +if declare -F hadoop_subproject_init >/dev/null ; then + hadoop_subproject_init +fi + +hadoop_shellprofiles_init + +# get the native libs in there pretty quick +hadoop_add_javalibpath "${HADOOP_HOME}/build/native" +hadoop_add_javalibpath "${HADOOP_HOME}/${HADOOP_COMMON_LIB_NATIVE_DIR}" + +hadoop_shellprofiles_nativelib + +# get the basic java class path for these subprojects +# in as quickly as possible since other stuff +# will definitely depend upon it. + +hadoop_add_common_to_classpath +hadoop_shellprofiles_classpath + +# user API commands can now be run since the runtime +# environment has been configured +hadoop_exec_hadooprc + +# +# backwards compatibility. new stuff should +# call this when they are ready +# +if [[ -z "${HADOOP_NEW_CONFIG}" ]]; then + hadoop_finalize +fi diff --git a/hadoop-hdds/common/src/main/bin/hadoop-daemons.sh b/hadoop-hdds/common/src/main/bin/hadoop-daemons.sh new file mode 100755 index 0000000000000..55304916ad1f7 --- /dev/null +++ b/hadoop-hdds/common/src/main/bin/hadoop-daemons.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Run a Hadoop command on all slave hosts. + +function hadoop_usage +{ + echo "Usage: hadoop-daemons.sh [--config confdir] [--hosts hostlistfile] (start|stop|status) " +} + +this="${BASH_SOURCE-$0}" +bin=$(cd -P -- "$(dirname -- "${this}")" >/dev/null && pwd -P) + +# let's locate libexec... +if [[ -n "${HADOOP_HOME}" ]]; then + HADOOP_DEFAULT_LIBEXEC_DIR="${HADOOP_HOME}/libexec" +else + HADOOP_DEFAULT_LIBEXEC_DIR="${bin}/../libexec" +fi + +HADOOP_LIBEXEC_DIR="${HADOOP_LIBEXEC_DIR:-$HADOOP_DEFAULT_LIBEXEC_DIR}" +# shellcheck disable=SC2034 +HADOOP_NEW_CONFIG=true +if [[ -f "${HADOOP_LIBEXEC_DIR}/hdfs-config.sh" ]]; then + . "${HADOOP_LIBEXEC_DIR}/hdfs-config.sh" +else + echo "ERROR: Cannot execute ${HADOOP_LIBEXEC_DIR}/hdfs-config.sh." 2>&1 + exit 1 +fi + +if [[ $# = 0 ]]; then + hadoop_exit_with_usage 1 +fi + +daemonmode=$1 +shift + +if [[ -z "${HADOOP_HDFS_HOME}" ]]; then + hdfsscript="${HADOOP_HOME}/bin/hdfs" +else + hdfsscript="${HADOOP_HDFS_HOME}/bin/hdfs" +fi + +hadoop_error "WARNING: Use of this script to ${daemonmode} HDFS daemons is deprecated." +hadoop_error "WARNING: Attempting to execute replacement \"hdfs --workers --daemon ${daemonmode}\" instead." + +# +# Original input was usually: +# hadoop-daemons.sh (shell options) (start|stop) (datanode|...) (daemon options) +# we're going to turn this into +# hdfs --workers --daemon (start|stop) (rest of options) +# +for (( i = 0; i < ${#HADOOP_USER_PARAMS[@]}; i++ )) +do + if [[ "${HADOOP_USER_PARAMS[$i]}" =~ ^start$ ]] || + [[ "${HADOOP_USER_PARAMS[$i]}" =~ ^stop$ ]] || + [[ "${HADOOP_USER_PARAMS[$i]}" =~ ^status$ ]]; then + unset HADOOP_USER_PARAMS[$i] + fi +done + +${hdfsscript} --workers --daemon "${daemonmode}" "${HADOOP_USER_PARAMS[@]}" diff --git a/hadoop-hdds/common/src/main/bin/hadoop-functions.sh b/hadoop-hdds/common/src/main/bin/hadoop-functions.sh new file mode 100755 index 0000000000000..484fe2302f9ba --- /dev/null +++ b/hadoop-hdds/common/src/main/bin/hadoop-functions.sh @@ -0,0 +1,2732 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# we need to declare this globally as an array, which can only +# be done outside of a function +declare -a HADOOP_SUBCMD_USAGE +declare -a HADOOP_OPTION_USAGE +declare -a HADOOP_SUBCMD_USAGE_TYPES + +## @description Print a message to stderr +## @audience public +## @stability stable +## @replaceable no +## @param string +function hadoop_error +{ + echo "$*" 1>&2 +} + +## @description Print a message to stderr if --debug is turned on +## @audience public +## @stability stable +## @replaceable no +## @param string +function hadoop_debug +{ + if [[ -n "${HADOOP_SHELL_SCRIPT_DEBUG}" ]]; then + echo "DEBUG: $*" 1>&2 + fi +} + +## @description Given a filename or dir, return the absolute version of it +## @description This works as an alternative to readlink, which isn't +## @description portable. +## @audience public +## @stability stable +## @param fsobj +## @replaceable no +## @return 0 success +## @return 1 failure +## @return stdout abspath +function hadoop_abs +{ + declare obj=$1 + declare dir + declare fn + declare dirret + + if [[ ! -e ${obj} ]]; then + return 1 + elif [[ -d ${obj} ]]; then + dir=${obj} + else + dir=$(dirname -- "${obj}") + fn=$(basename -- "${obj}") + fn="/${fn}" + fi + + dir=$(cd -P -- "${dir}" >/dev/null 2>/dev/null && pwd -P) + dirret=$? + if [[ ${dirret} = 0 ]]; then + echo "${dir}${fn}" + return 0 + fi + return 1 +} + +## @description Given variable $1 delete $2 from it +## @audience public +## @stability stable +## @replaceable no +function hadoop_delete_entry +{ + if [[ ${!1} =~ \ ${2}\ ]] ; then + hadoop_debug "Removing ${2} from ${1}" + eval "${1}"=\""${!1// ${2} }"\" + fi +} + +## @description Given variable $1 add $2 to it +## @audience public +## @stability stable +## @replaceable no +function hadoop_add_entry +{ + if [[ ! ${!1} =~ \ ${2}\ ]] ; then + hadoop_debug "Adding ${2} to ${1}" + #shellcheck disable=SC2140 + eval "${1}"=\""${!1} ${2} "\" + fi +} + +## @description Given variable $1 determine if $2 is in it +## @audience public +## @stability stable +## @replaceable no +## @return 0 = yes, 1 = no +function hadoop_verify_entry +{ + # this unfortunately can't really be tested by bats. :( + # so if this changes, be aware that unit tests effectively + # do this function in them + [[ ${!1} =~ \ ${2}\ ]] +} + +## @description Check if an array has a given value +## @audience public +## @stability stable +## @replaceable yes +## @param element +## @param array +## @returns 0 = yes +## @returns 1 = no +function hadoop_array_contains +{ + declare element=$1 + shift + declare val + + if [[ "$#" -eq 0 ]]; then + return 1 + fi + + for val in "${@}"; do + if [[ "${val}" == "${element}" ]]; then + return 0 + fi + done + return 1 +} + +## @description Add the `appendstring` if `checkstring` is not +## @description present in the given array +## @audience public +## @stability stable +## @replaceable yes +## @param envvar +## @param appendstring +function hadoop_add_array_param +{ + declare arrname=$1 + declare add=$2 + + declare arrref="${arrname}[@]" + declare array=("${!arrref}") + + if ! hadoop_array_contains "${add}" "${array[@]}"; then + #shellcheck disable=SC1083,SC2086 + eval ${arrname}=\(\"\${array[@]}\" \"${add}\" \) + hadoop_debug "$1 accepted $2" + else + hadoop_debug "$1 declined $2" + fi +} + +## @description Sort an array (must not contain regexps) +## @description present in the given array +## @audience public +## @stability stable +## @replaceable yes +## @param arrayvar +function hadoop_sort_array +{ + declare arrname=$1 + declare arrref="${arrname}[@]" + declare array=("${!arrref}") + declare oifs + + declare globstatus + declare -a sa + + globstatus=$(set -o | grep noglob | awk '{print $NF}') + + set -f + oifs=${IFS} + + # shellcheck disable=SC2034 + IFS=$'\n' sa=($(sort <<<"${array[*]}")) + + # shellcheck disable=SC1083 + eval "${arrname}"=\(\"\${sa[@]}\"\) + + IFS=${oifs} + if [[ "${globstatus}" = off ]]; then + set +f + fi +} + +## @description Check if we are running with priv +## @description by default, this implementation looks for +## @description EUID=0. For OSes that have true priv +## @description separation, this should be something more complex +## @audience private +## @stability evolving +## @replaceable yes +## @return 1 = no priv +## @return 0 = priv +function hadoop_privilege_check +{ + [[ "${EUID}" = 0 ]] +} + +## @description Execute a command via su when running as root +## @description if the given user is found or exit with +## @description failure if not. +## @description otherwise just run it. (This is intended to +## @description be used by the start-*/stop-* scripts.) +## @audience private +## @stability evolving +## @replaceable yes +## @param user +## @param commandstring +## @return exitstatus +function hadoop_su +{ + declare user=$1 + shift + + if hadoop_privilege_check; then + if hadoop_verify_user_resolves user; then + su -l "${user}" -- "$@" + else + hadoop_error "ERROR: Refusing to run as root: ${user} account is not found. Aborting." + return 1 + fi + else + "$@" + fi +} + +## @description Execute a command via su when running as root +## @description with extra support for commands that might +## @description legitimately start as root (e.g., datanode) +## @description (This is intended to +## @description be used by the start-*/stop-* scripts.) +## @audience private +## @stability evolving +## @replaceable no +## @param user +## @param commandstring +## @return exitstatus +function hadoop_uservar_su +{ + + ## startup matrix: + # + # if $EUID != 0, then exec + # if $EUID =0 then + # if hdfs_subcmd_user is defined, call hadoop_su to exec + # if hdfs_subcmd_user is not defined, error + # + # For secure daemons, this means both the secure and insecure env vars need to be + # defined. e.g., HDFS_DATANODE_USER=root HDFS_DATANODE_SECURE_USER=hdfs + # This function will pick up the "normal" var, switch to that user, then + # execute the command which will then pick up the "secure" version. + # + + declare program=$1 + declare command=$2 + shift 2 + + declare uprogram + declare ucommand + declare uvar + declare svar + + if hadoop_privilege_check; then + uvar=$(hadoop_build_custom_subcmd_var "${program}" "${command}" USER) + + svar=$(hadoop_build_custom_subcmd_var "${program}" "${command}" SECURE_USER) + + if [[ -n "${!uvar}" ]]; then + hadoop_su "${!uvar}" "$@" + elif [[ -n "${!svar}" ]]; then + ## if we are here, then SECURE_USER with no USER defined + ## we are already privileged, so just run the command and hope + ## for the best + "$@" + else + hadoop_error "ERROR: Attempting to operate on ${program} ${command} as root" + hadoop_error "ERROR: but there is no ${uvar} defined. Aborting operation." + return 1 + fi + else + "$@" + fi +} + +## @description Add a subcommand to the usage output +## @audience private +## @stability evolving +## @replaceable no +## @param subcommand +## @param subcommandtype +## @param subcommanddesc +function hadoop_add_subcommand +{ + declare subcmd=$1 + declare subtype=$2 + declare text=$3 + + hadoop_debug "${subcmd} as a ${subtype}" + + hadoop_add_array_param HADOOP_SUBCMD_USAGE_TYPES "${subtype}" + + # done in this order so that sort works later + HADOOP_SUBCMD_USAGE[${HADOOP_SUBCMD_USAGE_COUNTER}]="${subcmd}@${subtype}@${text}" + ((HADOOP_SUBCMD_USAGE_COUNTER=HADOOP_SUBCMD_USAGE_COUNTER+1)) +} + +## @description Add an option to the usage output +## @audience private +## @stability evolving +## @replaceable no +## @param subcommand +## @param subcommanddesc +function hadoop_add_option +{ + local option=$1 + local text=$2 + + HADOOP_OPTION_USAGE[${HADOOP_OPTION_USAGE_COUNTER}]="${option}@${text}" + ((HADOOP_OPTION_USAGE_COUNTER=HADOOP_OPTION_USAGE_COUNTER+1)) +} + +## @description Reset the usage information to blank +## @audience private +## @stability evolving +## @replaceable no +function hadoop_reset_usage +{ + HADOOP_SUBCMD_USAGE=() + HADOOP_OPTION_USAGE=() + HADOOP_SUBCMD_USAGE_TYPES=() + HADOOP_SUBCMD_USAGE_COUNTER=0 + HADOOP_OPTION_USAGE_COUNTER=0 +} + +## @description Print a screen-size aware two-column output +## @description if reqtype is not null, only print those requested +## @audience private +## @stability evolving +## @replaceable no +## @param reqtype +## @param array +function hadoop_generic_columnprinter +{ + declare reqtype=$1 + shift + declare -a input=("$@") + declare -i i=0 + declare -i counter=0 + declare line + declare text + declare option + declare giventext + declare -i maxoptsize + declare -i foldsize + declare -a tmpa + declare numcols + declare brup + + if [[ -n "${COLUMNS}" ]]; then + numcols=${COLUMNS} + else + numcols=$(tput cols) 2>/dev/null + COLUMNS=${numcols} + fi + + if [[ -z "${numcols}" + || ! "${numcols}" =~ ^[0-9]+$ ]]; then + numcols=75 + else + ((numcols=numcols-5)) + fi + + while read -r line; do + tmpa[${counter}]=${line} + ((counter=counter+1)) + IFS='@' read -ra brup <<< "${line}" + option="${brup[0]}" + if [[ ${#option} -gt ${maxoptsize} ]]; then + maxoptsize=${#option} + fi + done < <(for text in "${input[@]}"; do + echo "${text}" + done | sort) + + i=0 + ((foldsize=numcols-maxoptsize)) + + until [[ $i -eq ${#tmpa[@]} ]]; do + IFS='@' read -ra brup <<< "${tmpa[$i]}" + + option="${brup[0]}" + cmdtype="${brup[1]}" + giventext="${brup[2]}" + + if [[ -n "${reqtype}" ]]; then + if [[ "${cmdtype}" != "${reqtype}" ]]; then + ((i=i+1)) + continue + fi + fi + + if [[ -z "${giventext}" ]]; then + giventext=${cmdtype} + fi + + while read -r line; do + printf "%-${maxoptsize}s %-s\n" "${option}" "${line}" + option=" " + done < <(echo "${giventext}"| fold -s -w ${foldsize}) + ((i=i+1)) + done +} + +## @description generate standard usage output +## @description and optionally takes a class +## @audience private +## @stability evolving +## @replaceable no +## @param execname +## @param true|false +## @param [text to use in place of SUBCOMMAND] +function hadoop_generate_usage +{ + declare cmd=$1 + declare takesclass=$2 + declare subcmdtext=${3:-"SUBCOMMAND"} + declare haveoptions + declare optstring + declare havesubs + declare subcmdstring + declare cmdtype + + cmd=${cmd##*/} + + if [[ -n "${HADOOP_OPTION_USAGE_COUNTER}" + && "${HADOOP_OPTION_USAGE_COUNTER}" -gt 0 ]]; then + haveoptions=true + optstring=" [OPTIONS]" + fi + + if [[ -n "${HADOOP_SUBCMD_USAGE_COUNTER}" + && "${HADOOP_SUBCMD_USAGE_COUNTER}" -gt 0 ]]; then + havesubs=true + subcmdstring=" ${subcmdtext} [${subcmdtext} OPTIONS]" + fi + + echo "Usage: ${cmd}${optstring}${subcmdstring}" + if [[ ${takesclass} = true ]]; then + echo " or ${cmd}${optstring} CLASSNAME [CLASSNAME OPTIONS]" + echo " where CLASSNAME is a user-provided Java class" + fi + + if [[ "${haveoptions}" = true ]]; then + echo "" + echo " OPTIONS is none or any of:" + echo "" + + hadoop_generic_columnprinter "" "${HADOOP_OPTION_USAGE[@]}" + fi + + if [[ "${havesubs}" = true ]]; then + echo "" + echo " ${subcmdtext} is one of:" + echo "" + + if [[ "${#HADOOP_SUBCMD_USAGE_TYPES[@]}" -gt 0 ]]; then + + hadoop_sort_array HADOOP_SUBCMD_USAGE_TYPES + for subtype in "${HADOOP_SUBCMD_USAGE_TYPES[@]}"; do + #shellcheck disable=SC2086 + cmdtype="$(tr '[:lower:]' '[:upper:]' <<< ${subtype:0:1})${subtype:1}" + printf "\n %s Commands:\n\n" "${cmdtype}" + hadoop_generic_columnprinter "${subtype}" "${HADOOP_SUBCMD_USAGE[@]}" + done + else + hadoop_generic_columnprinter "" "${HADOOP_SUBCMD_USAGE[@]}" + fi + echo "" + echo "${subcmdtext} may print help when invoked w/o parameters or with -h." + fi +} + +## @description Replace `oldvar` with `newvar` if `oldvar` exists. +## @audience public +## @stability stable +## @replaceable yes +## @param oldvar +## @param newvar +function hadoop_deprecate_envvar +{ + local oldvar=$1 + local newvar=$2 + local oldval=${!oldvar} + local newval=${!newvar} + + if [[ -n "${oldval}" ]]; then + hadoop_error "WARNING: ${oldvar} has been replaced by ${newvar}. Using value of ${oldvar}." + # shellcheck disable=SC2086 + eval ${newvar}=\"${oldval}\" + + # shellcheck disable=SC2086 + newval=${oldval} + + # shellcheck disable=SC2086 + eval ${newvar}=\"${newval}\" + fi +} + +## @description Declare `var` being used and print its value. +## @audience public +## @stability stable +## @replaceable yes +## @param var +function hadoop_using_envvar +{ + local var=$1 + local val=${!var} + + if [[ -n "${val}" ]]; then + hadoop_debug "${var} = ${val}" + fi +} + +## @description Create the directory 'dir'. +## @audience public +## @stability stable +## @replaceable yes +## @param dir +function hadoop_mkdir +{ + local dir=$1 + + if [[ ! -w "${dir}" ]] && [[ ! -d "${dir}" ]]; then + hadoop_error "WARNING: ${dir} does not exist. Creating." + if ! mkdir -p "${dir}"; then + hadoop_error "ERROR: Unable to create ${dir}. Aborting." + exit 1 + fi + fi +} + +## @description Bootstraps the Hadoop shell environment +## @audience private +## @stability evolving +## @replaceable no +function hadoop_bootstrap +{ + # the root of the Hadoop installation + # See HADOOP-6255 for the expected directory structure layout + + if [[ -n "${DEFAULT_LIBEXEC_DIR}" ]]; then + hadoop_error "WARNING: DEFAULT_LIBEXEC_DIR ignored. It has been replaced by HADOOP_DEFAULT_LIBEXEC_DIR." + fi + + # By now, HADOOP_LIBEXEC_DIR should have been defined upstream + # We can piggyback off of that to figure out where the default + # HADOOP_FREFIX should be. This allows us to run without + # HADOOP_HOME ever being defined by a human! As a consequence + # HADOOP_LIBEXEC_DIR now becomes perhaps the single most powerful + # env var within Hadoop. + if [[ -z "${HADOOP_LIBEXEC_DIR}" ]]; then + hadoop_error "HADOOP_LIBEXEC_DIR is not defined. Exiting." + exit 1 + fi + HADOOP_DEFAULT_PREFIX=$(cd -P -- "${HADOOP_LIBEXEC_DIR}/.." >/dev/null && pwd -P) + HADOOP_HOME=${HADOOP_HOME:-$HADOOP_DEFAULT_PREFIX} + export HADOOP_HOME + + # + # short-cuts. vendors may redefine these as well, preferably + # in hadoop-layout.sh + # + HADOOP_COMMON_DIR=${HADOOP_COMMON_DIR:-"share/hadoop/common"} + HADOOP_COMMON_LIB_JARS_DIR=${HADOOP_COMMON_LIB_JARS_DIR:-"share/hadoop/common/lib"} + HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_COMMON_LIB_NATIVE_DIR:-"lib/native"} + HDFS_DIR=${HDFS_DIR:-"share/hadoop/hdfs"} + HDFS_LIB_JARS_DIR=${HDFS_LIB_JARS_DIR:-"share/hadoop/hdfs/lib"} + YARN_DIR=${YARN_DIR:-"share/hadoop/yarn"} + YARN_LIB_JARS_DIR=${YARN_LIB_JARS_DIR:-"share/hadoop/yarn/lib"} + MAPRED_DIR=${MAPRED_DIR:-"share/hadoop/mapreduce"} + MAPRED_LIB_JARS_DIR=${MAPRED_LIB_JARS_DIR:-"share/hadoop/mapreduce/lib"} + HDDS_DIR=${HDDS_DIR:-"share/hadoop/hdds"} + HDDS_LIB_JARS_DIR=${HDDS_LIB_JARS_DIR:-"share/hadoop/hdds/lib"} + OZONE_DIR=${OZONE_DIR:-"share/hadoop/ozone"} + OZONE_LIB_JARS_DIR=${OZONE_LIB_JARS_DIR:-"share/hadoop/ozone/lib"} + OZONEFS_DIR=${OZONEFS_DIR:-"share/hadoop/ozonefs"} + + HADOOP_TOOLS_HOME=${HADOOP_TOOLS_HOME:-${HADOOP_HOME}} + HADOOP_TOOLS_DIR=${HADOOP_TOOLS_DIR:-"share/hadoop/tools"} + HADOOP_TOOLS_LIB_JARS_DIR=${HADOOP_TOOLS_LIB_JARS_DIR:-"${HADOOP_TOOLS_DIR}/lib"} + + # by default, whatever we are about to run doesn't support + # daemonization + HADOOP_SUBCMD_SUPPORTDAEMONIZATION=false + + # by default, we have not been self-re-execed + HADOOP_REEXECED_CMD=false + + HADOOP_SUBCMD_SECURESERVICE=false + + # This is the default we claim in hadoop-env.sh + JSVC_HOME=${JSVC_HOME:-"/usr/bin"} + + # usage output set to zero + hadoop_reset_usage + + export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)} + + # defaults + export HADOOP_OPTS=${HADOOP_OPTS:-"-Djava.net.preferIPv4Stack=true"} + hadoop_debug "Initial HADOOP_OPTS=${HADOOP_OPTS}" +} + +## @description Locate Hadoop's configuration directory +## @audience private +## @stability evolving +## @replaceable no +function hadoop_find_confdir +{ + local conf_dir + + # An attempt at compatibility with some Hadoop 1.x + # installs. + if [[ -e "${HADOOP_HOME}/conf/hadoop-env.sh" ]]; then + conf_dir="conf" + else + conf_dir="etc/hadoop" + fi + export HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-${HADOOP_HOME}/${conf_dir}}" + + hadoop_debug "HADOOP_CONF_DIR=${HADOOP_CONF_DIR}" +} + +## @description Validate ${HADOOP_CONF_DIR} +## @audience public +## @stability stable +## @replaceable yes +## @return will exit on failure conditions +function hadoop_verify_confdir +{ + # Check only log4j.properties by default. + # --loglevel does not work without logger settings in log4j.log4j.properties. + if [[ ! -f "${HADOOP_CONF_DIR}/log4j.properties" ]]; then + hadoop_error "WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete." + fi +} + +## @description Import the hadoop-env.sh settings +## @audience private +## @stability evolving +## @replaceable no +function hadoop_exec_hadoopenv +{ + if [[ -z "${HADOOP_ENV_PROCESSED}" ]]; then + if [[ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]]; then + export HADOOP_ENV_PROCESSED=true + # shellcheck source=./hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh + . "${HADOOP_CONF_DIR}/hadoop-env.sh" + fi + fi +} + +## @description Import the replaced functions +## @audience private +## @stability evolving +## @replaceable no +function hadoop_exec_userfuncs +{ + if [[ -e "${HADOOP_CONF_DIR}/hadoop-user-functions.sh" ]]; then + # shellcheck disable=SC1090 + . "${HADOOP_CONF_DIR}/hadoop-user-functions.sh" + fi +} + +## @description Read the user's settings. This provides for users to +## @description override and/or append hadoop-env.sh. It is not meant +## @description as a complete system override. +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_exec_user_hadoopenv +{ + if [[ -f "${HOME}/.hadoop-env" ]]; then + hadoop_debug "Applying the user's .hadoop-env" + # shellcheck disable=SC1090 + . "${HOME}/.hadoop-env" + fi +} + +## @description Read the user's settings. This provides for users to +## @description run Hadoop Shell API after system bootstrap +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_exec_hadooprc +{ + if [[ -f "${HOME}/.hadooprc" ]]; then + hadoop_debug "Applying the user's .hadooprc" + # shellcheck disable=SC1090 + . "${HOME}/.hadooprc" + fi +} + +## @description Import shellprofile.d content +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_import_shellprofiles +{ + local i + local files1 + local files2 + + if [[ -d "${HADOOP_LIBEXEC_DIR}/shellprofile.d" ]]; then + files1=(${HADOOP_LIBEXEC_DIR}/shellprofile.d/*.sh) + hadoop_debug "shellprofiles: ${files1[*]}" + else + hadoop_error "WARNING: ${HADOOP_LIBEXEC_DIR}/shellprofile.d doesn't exist. Functionality may not work." + fi + + if [[ -d "${HADOOP_CONF_DIR}/shellprofile.d" ]]; then + files2=(${HADOOP_CONF_DIR}/shellprofile.d/*.sh) + fi + + # enable bundled shellprofiles that come + # from hadoop-tools. This converts the user-facing HADOOP_OPTIONAL_TOOLS + # to the HADOOP_TOOLS_OPTIONS that the shell profiles expect. + # See dist-tools-hooks-maker for how the example HADOOP_OPTIONAL_TOOLS + # gets populated into hadoop-env.sh + + for i in ${HADOOP_OPTIONAL_TOOLS//,/ }; do + hadoop_add_entry HADOOP_TOOLS_OPTIONS "${i}" + done + + for i in "${files1[@]}" "${files2[@]}" + do + if [[ -n "${i}" + && -f "${i}" ]]; then + hadoop_debug "Profiles: importing ${i}" + # shellcheck disable=SC1090 + . "${i}" + fi + done +} + +## @description Initialize the registered shell profiles +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_shellprofiles_init +{ + local i + + for i in ${HADOOP_SHELL_PROFILES} + do + if declare -F _${i}_hadoop_init >/dev/null ; then + hadoop_debug "Profiles: ${i} init" + # shellcheck disable=SC2086 + _${i}_hadoop_init + fi + done +} + +## @description Apply the shell profile classpath additions +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_shellprofiles_classpath +{ + local i + + for i in ${HADOOP_SHELL_PROFILES} + do + if declare -F _${i}_hadoop_classpath >/dev/null ; then + hadoop_debug "Profiles: ${i} classpath" + # shellcheck disable=SC2086 + _${i}_hadoop_classpath + fi + done +} + +## @description Apply the shell profile native library additions +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_shellprofiles_nativelib +{ + local i + + for i in ${HADOOP_SHELL_PROFILES} + do + if declare -F _${i}_hadoop_nativelib >/dev/null ; then + hadoop_debug "Profiles: ${i} nativelib" + # shellcheck disable=SC2086 + _${i}_hadoop_nativelib + fi + done +} + +## @description Apply the shell profile final configuration +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_shellprofiles_finalize +{ + local i + + for i in ${HADOOP_SHELL_PROFILES} + do + if declare -F _${i}_hadoop_finalize >/dev/null ; then + hadoop_debug "Profiles: ${i} finalize" + # shellcheck disable=SC2086 + _${i}_hadoop_finalize + fi + done +} + +## @description Initialize the Hadoop shell environment, now that +## @description user settings have been imported +## @audience private +## @stability evolving +## @replaceable no +function hadoop_basic_init +{ + # Some of these are also set in hadoop-env.sh. + # we still set them here just in case hadoop-env.sh is + # broken in some way, set up defaults, etc. + # + # but it is important to note that if you update these + # you also need to update hadoop-env.sh as well!!! + + CLASSPATH="" + hadoop_debug "Initialize CLASSPATH" + + if [[ -z "${HADOOP_COMMON_HOME}" ]] && + [[ -d "${HADOOP_HOME}/${HADOOP_COMMON_DIR}" ]]; then + export HADOOP_COMMON_HOME="${HADOOP_HOME}" + fi + + # default policy file for service-level authorization + HADOOP_POLICYFILE=${HADOOP_POLICYFILE:-"hadoop-policy.xml"} + + # define HADOOP_HDFS_HOME + if [[ -z "${HADOOP_HDFS_HOME}" ]] && + [[ -d "${HADOOP_HOME}/${HDFS_DIR}" ]]; then + export HADOOP_HDFS_HOME="${HADOOP_HOME}" + fi + + # define HADOOP_YARN_HOME + if [[ -z "${HADOOP_YARN_HOME}" ]] && + [[ -d "${HADOOP_HOME}/${YARN_DIR}" ]]; then + export HADOOP_YARN_HOME="${HADOOP_HOME}" + fi + + # define HADOOP_MAPRED_HOME + if [[ -z "${HADOOP_MAPRED_HOME}" ]] && + [[ -d "${HADOOP_HOME}/${MAPRED_DIR}" ]]; then + export HADOOP_MAPRED_HOME="${HADOOP_HOME}" + fi + + if [[ ! -d "${HADOOP_COMMON_HOME}" ]]; then + hadoop_error "ERROR: Invalid HADOOP_COMMON_HOME" + exit 1 + fi + + if [[ ! -d "${HADOOP_HDFS_HOME}" ]]; then + hadoop_error "ERROR: Invalid HADOOP_HDFS_HOME" + exit 1 + fi + + if [[ ! -d "${HADOOP_YARN_HOME}" ]]; then + hadoop_error "ERROR: Invalid HADOOP_YARN_HOME" + exit 1 + fi + + if [[ ! -d "${HADOOP_MAPRED_HOME}" ]]; then + hadoop_error "ERROR: Invalid HADOOP_MAPRED_HOME" + exit 1 + fi + + # if for some reason the shell doesn't have $USER defined + # (e.g., ssh'd in to execute a command) + # let's get the effective username and use that + USER=${USER:-$(id -nu)} + HADOOP_IDENT_STRING=${HADOOP_IDENT_STRING:-$USER} + HADOOP_LOG_DIR=${HADOOP_LOG_DIR:-"${HADOOP_HOME}/logs"} + HADOOP_LOGFILE=${HADOOP_LOGFILE:-hadoop.log} + HADOOP_LOGLEVEL=${HADOOP_LOGLEVEL:-INFO} + HADOOP_NICENESS=${HADOOP_NICENESS:-0} + HADOOP_STOP_TIMEOUT=${HADOOP_STOP_TIMEOUT:-5} + HADOOP_PID_DIR=${HADOOP_PID_DIR:-/tmp} + HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console} + HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA} + HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender} + HADOOP_SSH_OPTS=${HADOOP_SSH_OPTS-"-o BatchMode=yes -o StrictHostKeyChecking=no -o ConnectTimeout=10s"} + HADOOP_SECURE_LOG_DIR=${HADOOP_SECURE_LOG_DIR:-${HADOOP_LOG_DIR}} + HADOOP_SECURE_PID_DIR=${HADOOP_SECURE_PID_DIR:-${HADOOP_PID_DIR}} + HADOOP_SSH_PARALLEL=${HADOOP_SSH_PARALLEL:-10} +} + +## @description Set the worker support information to the contents +## @description of `filename` +## @audience public +## @stability stable +## @replaceable no +## @param filename +## @return will exit if file does not exist +function hadoop_populate_workers_file +{ + local workersfile=$1 + shift + if [[ -f "${workersfile}" ]]; then + HADOOP_WORKERS="${workersfile}" + elif [[ -f "${HADOOP_CONF_DIR}/${workersfile}" ]]; then + HADOOP_WORKERS="${HADOOP_CONF_DIR}/${workersfile}" + else + hadoop_error "ERROR: Cannot find hosts file \"${workersfile}\"" + hadoop_exit_with_usage 1 + fi +} + +## @description Rotates the given `file` until `number` of +## @description files exist. +## @audience public +## @stability stable +## @replaceable no +## @param filename +## @param [number] +## @return $? will contain last mv's return value +function hadoop_rotate_log +{ + # + # Users are likely to replace this one for something + # that gzips or uses dates or who knows what. + # + # be aware that &1 and &2 might go through here + # so don't do anything too crazy... + # + local log=$1; + local num=${2:-5}; + + if [[ -f "${log}" ]]; then # rotate logs + while [[ ${num} -gt 1 ]]; do + #shellcheck disable=SC2086 + let prev=${num}-1 + if [[ -f "${log}.${prev}" ]]; then + mv "${log}.${prev}" "${log}.${num}" + fi + num=${prev} + done + mv "${log}" "${log}.${num}" + fi +} + +## @description Via ssh, log into `hostname` and run `command` +## @audience private +## @stability evolving +## @replaceable yes +## @param hostname +## @param command +## @param [...] +function hadoop_actual_ssh +{ + # we are passing this function to xargs + # should get hostname followed by rest of command line + local worker=$1 + shift + + # shellcheck disable=SC2086 + ssh ${HADOOP_SSH_OPTS} ${worker} $"${@// /\\ }" 2>&1 | sed "s/^/$worker: /" +} + +## @description Connect to ${HADOOP_WORKERS} or ${HADOOP_WORKER_NAMES} +## @description and execute command. +## @audience private +## @stability evolving +## @replaceable yes +## @param command +## @param [...] +function hadoop_connect_to_hosts +{ + # shellcheck disable=SC2124 + local params="$@" + local worker_file + local tmpslvnames + + # + # ssh (or whatever) to a host + # + # User can specify hostnames or a file where the hostnames are (not both) + if [[ -n "${HADOOP_WORKERS}" && -n "${HADOOP_WORKER_NAMES}" ]] ; then + hadoop_error "ERROR: Both HADOOP_WORKERS and HADOOP_WORKER_NAMES were defined. Aborting." + exit 1 + elif [[ -z "${HADOOP_WORKER_NAMES}" ]]; then + if [[ -n "${HADOOP_WORKERS}" ]]; then + worker_file=${HADOOP_WORKERS} + elif [[ -f "${HADOOP_CONF_DIR}/workers" ]]; then + worker_file=${HADOOP_CONF_DIR}/workers + elif [[ -f "${HADOOP_CONF_DIR}/slaves" ]]; then + hadoop_error "WARNING: 'slaves' file has been deprecated. Please use 'workers' file instead." + worker_file=${HADOOP_CONF_DIR}/slaves + fi + fi + + # if pdsh is available, let's use it. otherwise default + # to a loop around ssh. (ugh) + if [[ -e '/usr/bin/pdsh' ]]; then + if [[ -z "${HADOOP_WORKER_NAMES}" ]] ; then + # if we were given a file, just let pdsh deal with it. + # shellcheck disable=SC2086 + PDSH_SSH_ARGS_APPEND="${HADOOP_SSH_OPTS}" pdsh \ + -f "${HADOOP_SSH_PARALLEL}" -w ^"${worker_file}" $"${@// /\\ }" 2>&1 + else + # no spaces allowed in the pdsh arg host list + # shellcheck disable=SC2086 + tmpslvnames=$(echo ${HADOOP_WORKER_NAMES} | tr -s ' ' ,) + PDSH_SSH_ARGS_APPEND="${HADOOP_SSH_OPTS}" pdsh \ + -f "${HADOOP_SSH_PARALLEL}" \ + -w "${tmpslvnames}" $"${@// /\\ }" 2>&1 + fi + else + if [[ -z "${HADOOP_WORKER_NAMES}" ]]; then + HADOOP_WORKER_NAMES=$(sed 's/#.*$//;/^$/d' "${worker_file}") + fi + hadoop_connect_to_hosts_without_pdsh "${params}" + fi +} + +## @description Connect to ${HADOOP_WORKER_NAMES} and execute command +## @description under the environment which does not support pdsh. +## @audience private +## @stability evolving +## @replaceable yes +## @param command +## @param [...] +function hadoop_connect_to_hosts_without_pdsh +{ + # shellcheck disable=SC2124 + local params="$@" + local workers=(${HADOOP_WORKER_NAMES}) + for (( i = 0; i < ${#workers[@]}; i++ )) + do + if (( i != 0 && i % HADOOP_SSH_PARALLEL == 0 )); then + wait + fi + # shellcheck disable=SC2086 + hadoop_actual_ssh "${workers[$i]}" ${params} & + done + wait +} + +## @description Utility routine to handle --workers mode +## @audience private +## @stability evolving +## @replaceable yes +## @param commandarray +function hadoop_common_worker_mode_execute +{ + # + # input should be the command line as given by the user + # in the form of an array + # + local argv=("$@") + + # if --workers is still on the command line, remove it + # to prevent loops + # Also remove --hostnames and --hosts along with arg values + local argsSize=${#argv[@]}; + for (( i = 0; i < argsSize; i++ )) + do + if [[ "${argv[$i]}" =~ ^--workers$ ]]; then + unset argv[$i] + elif [[ "${argv[$i]}" =~ ^--hostnames$ ]] || + [[ "${argv[$i]}" =~ ^--hosts$ ]]; then + unset argv[$i]; + let i++; + unset argv[$i]; + fi + done + if [[ ${QATESTMODE} = true ]]; then + echo "${argv[@]}" + return + fi + hadoop_connect_to_hosts -- "${argv[@]}" +} + +## @description Verify that a shell command was passed a valid +## @description class name +## @audience public +## @stability stable +## @replaceable yes +## @param classname +## @return 0 = success +## @return 1 = failure w/user message +function hadoop_validate_classname +{ + local class=$1 + shift 1 + + if [[ ! ${class} =~ \. ]]; then + # assuming the arg is typo of command if it does not conatain ".". + # class belonging to no package is not allowed as a result. + hadoop_error "ERROR: ${class} is not COMMAND nor fully qualified CLASSNAME." + return 1 + fi + return 0 +} + +## @description Append the `appendstring` if `checkstring` is not +## @description present in the given `envvar` +## @audience public +## @stability stable +## @replaceable yes +## @param envvar +## @param checkstring +## @param appendstring +function hadoop_add_param +{ + # + # general param dedupe.. + # $1 is what we are adding to + # $2 is the name of what we want to add (key) + # $3 is the key+value of what we're adding + # + # doing it this way allows us to support all sorts of + # different syntaxes, just so long as they are space + # delimited + # + if [[ ! ${!1} =~ $2 ]] ; then + #shellcheck disable=SC2140 + eval "$1"="'${!1} $3'" + if [[ ${!1:0:1} = ' ' ]]; then + #shellcheck disable=SC2140 + eval "$1"="'${!1# }'" + fi + hadoop_debug "$1 accepted $3" + else + hadoop_debug "$1 declined $3" + fi +} + +## @description Register the given `shellprofile` to the Hadoop +## @description shell subsystem +## @audience public +## @stability stable +## @replaceable yes +## @param shellprofile +function hadoop_add_profile +{ + # shellcheck disable=SC2086 + hadoop_add_param HADOOP_SHELL_PROFILES $1 $1 +} + +## @description Add a file system object (directory, file, +## @description wildcard, ...) to the classpath. Optionally provide +## @description a hint as to where in the classpath it should go. +## @audience public +## @stability stable +## @replaceable yes +## @param object +## @param [before|after] +## @return 0 = success (added or duplicate) +## @return 1 = failure (doesn't exist or some other reason) +function hadoop_add_classpath +{ + # However, with classpath (& JLP), we can do dedupe + # along with some sanity checking (e.g., missing directories) + # since we have a better idea of what is legal + # + # for wildcard at end, we can + # at least check the dir exists + if [[ $1 =~ ^.*\*$ ]]; then + local mp + mp=$(dirname "$1") + if [[ ! -d "${mp}" ]]; then + hadoop_debug "Rejected CLASSPATH: $1 (not a dir)" + return 1 + fi + + # no wildcard in the middle, so check existence + # (doesn't matter *what* it is) + elif [[ ! $1 =~ ^.*\*.*$ ]] && [[ ! -e "$1" ]]; then + hadoop_debug "Rejected CLASSPATH: $1 (does not exist)" + return 1 + fi + if [[ -z "${CLASSPATH}" ]]; then + CLASSPATH=$1 + hadoop_debug "Initial CLASSPATH=$1" + elif [[ ":${CLASSPATH}:" != *":$1:"* ]]; then + if [[ "$2" = "before" ]]; then + CLASSPATH="$1:${CLASSPATH}" + hadoop_debug "Prepend CLASSPATH: $1" + else + CLASSPATH+=:$1 + hadoop_debug "Append CLASSPATH: $1" + fi + else + hadoop_debug "Dupe CLASSPATH: $1" + fi + return 0 +} + +## @description Add a file system object (directory, file, +## @description wildcard, ...) to the colonpath. Optionally provide +## @description a hint as to where in the colonpath it should go. +## @description Prior to adding, objects are checked for duplication +## @description and check for existence. Many other functions use +## @description this function as their base implementation +## @description including `hadoop_add_javalibpath` and `hadoop_add_ldlibpath`. +## @audience public +## @stability stable +## @replaceable yes +## @param envvar +## @param object +## @param [before|after] +## @return 0 = success (added or duplicate) +## @return 1 = failure (doesn't exist or some other reason) +function hadoop_add_colonpath +{ + # this is CLASSPATH, JLP, etc but with dedupe but no + # other checking + if [[ -d "${2}" ]] && [[ ":${!1}:" != *":$2:"* ]]; then + if [[ -z "${!1}" ]]; then + # shellcheck disable=SC2086 + eval $1="'$2'" + hadoop_debug "Initial colonpath($1): $2" + elif [[ "$3" = "before" ]]; then + # shellcheck disable=SC2086 + eval $1="'$2:${!1}'" + hadoop_debug "Prepend colonpath($1): $2" + else + # shellcheck disable=SC2086 + eval $1+=":'$2'" + hadoop_debug "Append colonpath($1): $2" + fi + return 0 + fi + hadoop_debug "Rejected colonpath($1): $2" + return 1 +} + +## @description Add a file system object (directory, file, +## @description wildcard, ...) to the Java JNI path. Optionally +## @description provide a hint as to where in the Java JNI path +## @description it should go. +## @audience public +## @stability stable +## @replaceable yes +## @param object +## @param [before|after] +## @return 0 = success (added or duplicate) +## @return 1 = failure (doesn't exist or some other reason) +function hadoop_add_javalibpath +{ + # specialized function for a common use case + hadoop_add_colonpath JAVA_LIBRARY_PATH "$1" "$2" +} + +## @description Add a file system object (directory, file, +## @description wildcard, ...) to the LD_LIBRARY_PATH. Optionally +## @description provide a hint as to where in the LD_LIBRARY_PATH +## @description it should go. +## @audience public +## @stability stable +## @replaceable yes +## @param object +## @param [before|after] +## @return 0 = success (added or duplicate) +## @return 1 = failure (doesn't exist or some other reason) +function hadoop_add_ldlibpath +{ + local status + # specialized function for a common use case + hadoop_add_colonpath LD_LIBRARY_PATH "$1" "$2" + status=$? + + # note that we export this + export LD_LIBRARY_PATH + return ${status} +} + +## @description Add the common/core Hadoop components to the +## @description environment +## @audience private +## @stability evolving +## @replaceable yes +## @returns 1 on failure, may exit +## @returns 0 on success +function hadoop_add_common_to_classpath +{ + # + # get all of the common jars+config in the path + # + + if [[ -z "${HADOOP_COMMON_HOME}" + || -z "${HADOOP_COMMON_DIR}" + || -z "${HADOOP_COMMON_LIB_JARS_DIR}" ]]; then + hadoop_debug "COMMON_HOME=${HADOOP_COMMON_HOME}" + hadoop_debug "COMMON_DIR=${HADOOP_COMMON_DIR}" + hadoop_debug "COMMON_LIB_JARS_DIR=${HADOOP_COMMON_LIB_JARS_DIR}" + hadoop_error "ERROR: HADOOP_COMMON_HOME or related vars are not configured." + exit 1 + fi + + # developers + if [[ -n "${HADOOP_ENABLE_BUILD_PATHS}" ]]; then + hadoop_add_classpath "${HADOOP_COMMON_HOME}/hadoop-common/target/classes" + fi + + hadoop_add_classpath "${HADOOP_COMMON_HOME}/${HADOOP_COMMON_LIB_JARS_DIR}"'/*' + hadoop_add_classpath "${HADOOP_COMMON_HOME}/${HADOOP_COMMON_DIR}"'/*' +} + +## @description Run libexec/tools/module.sh to add to the classpath +## @description environment +## @audience private +## @stability evolving +## @replaceable yes +## @param module +function hadoop_add_to_classpath_tools +{ + declare module=$1 + + if [[ -f "${HADOOP_LIBEXEC_DIR}/tools/${module}.sh" ]]; then + # shellcheck disable=SC1090 + . "${HADOOP_LIBEXEC_DIR}/tools/${module}.sh" + else + hadoop_error "ERROR: Tools helper ${HADOOP_LIBEXEC_DIR}/tools/${module}.sh was not found." + fi + + if declare -f hadoop_classpath_tools_${module} >/dev/null 2>&1; then + "hadoop_classpath_tools_${module}" + fi +} + +## @description Add the user's custom classpath settings to the +## @description environment +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_add_to_classpath_userpath +{ + # Add the user-specified HADOOP_CLASSPATH to the + # official CLASSPATH env var if HADOOP_USE_CLIENT_CLASSLOADER + # is not set. + # Add it first or last depending on if user has + # set env-var HADOOP_USER_CLASSPATH_FIRST + # we'll also dedupe it, because we're cool like that. + # + declare -a array + declare -i c=0 + declare -i j + declare -i i + declare idx + + if [[ -n "${HADOOP_CLASSPATH}" ]]; then + # I wonder if Java runs on VMS. + for idx in $(echo "${HADOOP_CLASSPATH}" | tr : '\n'); do + array[${c}]=${idx} + ((c=c+1)) + done + + # bats gets confused by j getting set to 0 + ((j=c-1)) || ${QATESTMODE} + + if [[ -z "${HADOOP_USE_CLIENT_CLASSLOADER}" ]]; then + if [[ -z "${HADOOP_USER_CLASSPATH_FIRST}" ]]; then + for ((i=0; i<=j; i++)); do + hadoop_add_classpath "${array[$i]}" after + done + else + for ((i=j; i>=0; i--)); do + hadoop_add_classpath "${array[$i]}" before + done + fi + fi + fi +} + +## @description Routine to configure any OS-specific settings. +## @audience public +## @stability stable +## @replaceable yes +## @return may exit on failure conditions +function hadoop_os_tricks +{ + local bindv6only + + HADOOP_IS_CYGWIN=false + case ${HADOOP_OS_TYPE} in + Darwin) + if [[ -z "${JAVA_HOME}" ]]; then + if [[ -x /usr/libexec/java_home ]]; then + JAVA_HOME="$(/usr/libexec/java_home)" + export JAVA_HOME + else + JAVA_HOME=/Library/Java/Home + export JAVA_HOME + fi + fi + ;; + Linux) + + # Newer versions of glibc use an arena memory allocator that + # causes virtual # memory usage to explode. This interacts badly + # with the many threads that we use in Hadoop. Tune the variable + # down to prevent vmem explosion. + export MALLOC_ARENA_MAX=${MALLOC_ARENA_MAX:-4} + # we put this in QA test mode off so that non-Linux can test + if [[ "${QATESTMODE}" = true ]]; then + return + fi + + # NOTE! HADOOP_ALLOW_IPV6 is a developer hook. We leave it + # undocumented in hadoop-env.sh because we don't want users to + # shoot themselves in the foot while devs make IPv6 work. + + bindv6only=$(/sbin/sysctl -n net.ipv6.bindv6only 2> /dev/null) + + if [[ -n "${bindv6only}" ]] && + [[ "${bindv6only}" -eq "1" ]] && + [[ "${HADOOP_ALLOW_IPV6}" != "yes" ]]; then + hadoop_error "ERROR: \"net.ipv6.bindv6only\" is set to 1 " + hadoop_error "ERROR: Hadoop networking could be broken. Aborting." + hadoop_error "ERROR: For more info: http://wiki.apache.org/hadoop/HadoopIPv6" + exit 1 + fi + ;; + CYGWIN*) + # Flag that we're running on Cygwin to trigger path translation later. + HADOOP_IS_CYGWIN=true + ;; + esac +} + +## @description Configure/verify ${JAVA_HOME} +## @audience public +## @stability stable +## @replaceable yes +## @return may exit on failure conditions +function hadoop_java_setup +{ + # Bail if we did not detect it + if [[ -z "${JAVA_HOME}" ]]; then + hadoop_error "ERROR: JAVA_HOME is not set and could not be found." + exit 1 + fi + + if [[ ! -d "${JAVA_HOME}" ]]; then + hadoop_error "ERROR: JAVA_HOME ${JAVA_HOME} does not exist." + exit 1 + fi + + JAVA="${JAVA_HOME}/bin/java" + + if [[ ! -x "$JAVA" ]]; then + hadoop_error "ERROR: $JAVA is not executable." + exit 1 + fi +} + +## @description Finish Java JNI paths prior to execution +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_finalize_libpaths +{ + if [[ -n "${JAVA_LIBRARY_PATH}" ]]; then + hadoop_translate_cygwin_path JAVA_LIBRARY_PATH + hadoop_add_param HADOOP_OPTS java.library.path \ + "-Djava.library.path=${JAVA_LIBRARY_PATH}" + export LD_LIBRARY_PATH + fi +} + +## @description Finish Java heap parameters prior to execution +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_finalize_hadoop_heap +{ + if [[ -n "${HADOOP_HEAPSIZE_MAX}" ]]; then + if [[ "${HADOOP_HEAPSIZE_MAX}" =~ ^[0-9]+$ ]]; then + HADOOP_HEAPSIZE_MAX="${HADOOP_HEAPSIZE_MAX}m" + fi + hadoop_add_param HADOOP_OPTS Xmx "-Xmx${HADOOP_HEAPSIZE_MAX}" + fi + + # backwards compatibility + if [[ -n "${HADOOP_HEAPSIZE}" ]]; then + if [[ "${HADOOP_HEAPSIZE}" =~ ^[0-9]+$ ]]; then + HADOOP_HEAPSIZE="${HADOOP_HEAPSIZE}m" + fi + hadoop_add_param HADOOP_OPTS Xmx "-Xmx${HADOOP_HEAPSIZE}" + fi + + if [[ -n "${HADOOP_HEAPSIZE_MIN}" ]]; then + if [[ "${HADOOP_HEAPSIZE_MIN}" =~ ^[0-9]+$ ]]; then + HADOOP_HEAPSIZE_MIN="${HADOOP_HEAPSIZE_MIN}m" + fi + hadoop_add_param HADOOP_OPTS Xms "-Xms${HADOOP_HEAPSIZE_MIN}" + fi +} + +## @description Converts the contents of the variable name +## @description `varnameref` into the equivalent Windows path. +## @description If the second parameter is true, then `varnameref` +## @description is treated as though it was a path list. +## @audience public +## @stability stable +## @replaceable yes +## @param varnameref +## @param [true] +function hadoop_translate_cygwin_path +{ + if [[ "${HADOOP_IS_CYGWIN}" = "true" ]]; then + if [[ "$2" = "true" ]]; then + #shellcheck disable=SC2016 + eval "$1"='$(cygpath -p -w "${!1}" 2>/dev/null)' + else + #shellcheck disable=SC2016 + eval "$1"='$(cygpath -w "${!1}" 2>/dev/null)' + fi + fi +} + +## @description Adds the HADOOP_CLIENT_OPTS variable to +## @description HADOOP_OPTS if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is false +## @audience public +## @stability stable +## @replaceable yes +function hadoop_add_client_opts +{ + if [[ "${HADOOP_SUBCMD_SUPPORTDAEMONIZATION}" = false + || -z "${HADOOP_SUBCMD_SUPPORTDAEMONIZATION}" ]]; then + hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS" + HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}" + fi +} + +## @description Finish configuring Hadoop specific system properties +## @description prior to executing Java +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_finalize_hadoop_opts +{ + hadoop_translate_cygwin_path HADOOP_LOG_DIR + hadoop_add_param HADOOP_OPTS hadoop.log.dir "-Dhadoop.log.dir=${HADOOP_LOG_DIR}" + hadoop_add_param HADOOP_OPTS hadoop.log.file "-Dhadoop.log.file=${HADOOP_LOGFILE}" + hadoop_translate_cygwin_path HADOOP_HOME + export HADOOP_HOME + hadoop_add_param HADOOP_OPTS hadoop.home.dir "-Dhadoop.home.dir=${HADOOP_HOME}" + hadoop_add_param HADOOP_OPTS hadoop.id.str "-Dhadoop.id.str=${HADOOP_IDENT_STRING}" + hadoop_add_param HADOOP_OPTS hadoop.root.logger "-Dhadoop.root.logger=${HADOOP_ROOT_LOGGER}" + hadoop_add_param HADOOP_OPTS hadoop.policy.file "-Dhadoop.policy.file=${HADOOP_POLICYFILE}" + hadoop_add_param HADOOP_OPTS hadoop.security.logger "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER}" +} + +## @description Finish Java classpath prior to execution +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_finalize_classpath +{ + hadoop_add_classpath "${HADOOP_CONF_DIR}" before + + # user classpath gets added at the last minute. this allows + # override of CONF dirs and more + hadoop_add_to_classpath_userpath + hadoop_translate_cygwin_path CLASSPATH true +} + +## @description Finish all the remaining environment settings prior +## @description to executing Java. This is a wrapper that calls +## @description the other `finalize` routines. +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_finalize +{ + hadoop_shellprofiles_finalize + + hadoop_finalize_classpath + hadoop_finalize_libpaths + hadoop_finalize_hadoop_heap + hadoop_finalize_hadoop_opts + + hadoop_translate_cygwin_path HADOOP_HOME + hadoop_translate_cygwin_path HADOOP_CONF_DIR + hadoop_translate_cygwin_path HADOOP_COMMON_HOME + hadoop_translate_cygwin_path HADOOP_HDFS_HOME + hadoop_translate_cygwin_path HADOOP_YARN_HOME + hadoop_translate_cygwin_path HADOOP_MAPRED_HOME +} + +## @description Print usage information and exit with the passed +## @description `exitcode` +## @audience public +## @stability stable +## @replaceable no +## @param exitcode +## @return This function will always exit. +function hadoop_exit_with_usage +{ + local exitcode=$1 + if [[ -z $exitcode ]]; then + exitcode=1 + fi + # shellcheck disable=SC2034 + if declare -F hadoop_usage >/dev/null ; then + hadoop_usage + elif [[ -x /usr/bin/cowsay ]]; then + /usr/bin/cowsay -f elephant "Sorry, no help available." + else + hadoop_error "Sorry, no help available." + fi + exit $exitcode +} + +## @description Verify that prerequisites have been met prior to +## @description excuting a privileged program. +## @audience private +## @stability evolving +## @replaceable yes +## @return This routine may exit. +function hadoop_verify_secure_prereq +{ + # if you are on an OS like Illumos that has functional roles + # and you are using pfexec, you'll probably want to change + # this. + + if ! hadoop_privilege_check && [[ -z "${HADOOP_SECURE_COMMAND}" ]]; then + hadoop_error "ERROR: You must be a privileged user in order to run a secure service." + exit 1 + else + return 0 + fi +} + +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_setup_secure_service +{ + # need a more complicated setup? replace me! + + HADOOP_PID_DIR=${HADOOP_SECURE_PID_DIR} + HADOOP_LOG_DIR=${HADOOP_SECURE_LOG_DIR} +} + +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_verify_piddir +{ + if [[ -z "${HADOOP_PID_DIR}" ]]; then + hadoop_error "No pid directory defined." + exit 1 + fi + hadoop_mkdir "${HADOOP_PID_DIR}" + touch "${HADOOP_PID_DIR}/$$" >/dev/null 2>&1 + if [[ $? -gt 0 ]]; then + hadoop_error "ERROR: Unable to write in ${HADOOP_PID_DIR}. Aborting." + exit 1 + fi + rm "${HADOOP_PID_DIR}/$$" >/dev/null 2>&1 +} + +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_verify_logdir +{ + if [[ -z "${HADOOP_LOG_DIR}" ]]; then + hadoop_error "No log directory defined." + exit 1 + fi + hadoop_mkdir "${HADOOP_LOG_DIR}" + touch "${HADOOP_LOG_DIR}/$$" >/dev/null 2>&1 + if [[ $? -gt 0 ]]; then + hadoop_error "ERROR: Unable to write in ${HADOOP_LOG_DIR}. Aborting." + exit 1 + fi + rm "${HADOOP_LOG_DIR}/$$" >/dev/null 2>&1 +} + +## @description Determine the status of the daemon referenced +## @description by `pidfile` +## @audience public +## @stability stable +## @replaceable yes +## @param pidfile +## @return (mostly) LSB 4.1.0 compatible status +function hadoop_status_daemon +{ + # + # LSB 4.1.0 compatible status command (1) + # + # 0 = program is running + # 1 = dead, but still a pid (2) + # 2 = (not used by us) + # 3 = not running + # + # 1 - this is not an endorsement of the LSB + # + # 2 - technically, the specification says /var/run/pid, so + # we should never return this value, but we're giving + # them the benefit of a doubt and returning 1 even if + # our pid is not in in /var/run . + # + + local pidfile=$1 + shift + + local pid + local pspid + + if [[ -f "${pidfile}" ]]; then + pid=$(cat "${pidfile}") + if pspid=$(ps -o args= -p"${pid}" 2>/dev/null); then + # this is to check that the running process we found is actually the same + # daemon that we're interested in + if [[ ${pspid} =~ -Dproc_${daemonname} ]]; then + return 0 + fi + fi + return 1 + fi + return 3 +} + +## @description Execute the Java `class`, passing along any `options`. +## @description Additionally, set the Java property -Dproc_`command`. +## @audience public +## @stability stable +## @replaceable yes +## @param command +## @param class +## @param [options] +function hadoop_java_exec +{ + # run a java command. this is used for + # non-daemons + + local command=$1 + local class=$2 + shift 2 + + hadoop_debug "Final CLASSPATH: ${CLASSPATH}" + hadoop_debug "Final HADOOP_OPTS: ${HADOOP_OPTS}" + hadoop_debug "Final JAVA_HOME: ${JAVA_HOME}" + hadoop_debug "java: ${JAVA}" + hadoop_debug "Class name: ${class}" + hadoop_debug "Command line options: $*" + + export CLASSPATH + #shellcheck disable=SC2086 + exec "${JAVA}" "-Dproc_${command}" ${HADOOP_OPTS} "${class}" "$@" +} + +## @description Start a non-privileged daemon in the foreground. +## @audience private +## @stability evolving +## @replaceable yes +## @param command +## @param class +## @param pidfile +## @param [options] +function hadoop_start_daemon +{ + # this is our non-privileged daemon starter + # that fires up a daemon in the *foreground* + # so complex! so wow! much java! + local command=$1 + local class=$2 + local pidfile=$3 + shift 3 + + hadoop_debug "Final CLASSPATH: ${CLASSPATH}" + hadoop_debug "Final HADOOP_OPTS: ${HADOOP_OPTS}" + hadoop_debug "Final JAVA_HOME: ${JAVA_HOME}" + hadoop_debug "java: ${JAVA}" + hadoop_debug "Class name: ${class}" + hadoop_debug "Command line options: $*" + + # this is for the non-daemon pid creation + #shellcheck disable=SC2086 + echo $$ > "${pidfile}" 2>/dev/null + if [[ $? -gt 0 ]]; then + hadoop_error "ERROR: Cannot write ${command} pid ${pidfile}." + fi + + export CLASSPATH + #shellcheck disable=SC2086 + exec "${JAVA}" "-Dproc_${command}" ${HADOOP_OPTS} "${class}" "$@" +} + +## @description Start a non-privileged daemon in the background. +## @audience private +## @stability evolving +## @replaceable yes +## @param command +## @param class +## @param pidfile +## @param outfile +## @param [options] +function hadoop_start_daemon_wrapper +{ + local daemonname=$1 + local class=$2 + local pidfile=$3 + local outfile=$4 + shift 4 + + local counter + + hadoop_rotate_log "${outfile}" + + hadoop_start_daemon "${daemonname}" \ + "$class" \ + "${pidfile}" \ + "$@" >> "${outfile}" 2>&1 < /dev/null & + + # we need to avoid a race condition here + # so let's wait for the fork to finish + # before overriding with the daemonized pid + (( counter=0 )) + while [[ ! -f ${pidfile} && ${counter} -le 5 ]]; do + sleep 1 + (( counter++ )) + done + + # this is for daemon pid creation + #shellcheck disable=SC2086 + echo $! > "${pidfile}" 2>/dev/null + if [[ $? -gt 0 ]]; then + hadoop_error "ERROR: Cannot write ${daemonname} pid ${pidfile}." + fi + + # shellcheck disable=SC2086 + renice "${HADOOP_NICENESS}" $! >/dev/null 2>&1 + if [[ $? -gt 0 ]]; then + hadoop_error "ERROR: Cannot set priority of ${daemonname} process $!" + fi + + # shellcheck disable=SC2086 + disown %+ >/dev/null 2>&1 + if [[ $? -gt 0 ]]; then + hadoop_error "ERROR: Cannot disconnect ${daemonname} process $!" + fi + sleep 1 + + # capture the ulimit output + ulimit -a >> "${outfile}" 2>&1 + + # shellcheck disable=SC2086 + if ! ps -p $! >/dev/null 2>&1; then + return 1 + fi + return 0 +} + +## @description Start a privileged daemon in the foreground. +## @audience private +## @stability evolving +## @replaceable yes +## @param command +## @param class +## @param daemonpidfile +## @param daemonoutfile +## @param daemonerrfile +## @param wrapperpidfile +## @param [options] +function hadoop_start_secure_daemon +{ + # this is used to launch a secure daemon in the *foreground* + # + local daemonname=$1 + local class=$2 + + # pid file to create for our daemon + local daemonpidfile=$3 + + # where to send stdout. jsvc has bad habits so this *may* be &1 + # which means you send it to stdout! + local daemonoutfile=$4 + + # where to send stderr. same thing, except &2 = stderr + local daemonerrfile=$5 + local privpidfile=$6 + shift 6 + + hadoop_rotate_log "${daemonoutfile}" + hadoop_rotate_log "${daemonerrfile}" + + # shellcheck disable=SC2153 + jsvc="${JSVC_HOME}/jsvc" + if [[ ! -f "${jsvc}" ]]; then + hadoop_error "JSVC_HOME is not set or set incorrectly. jsvc is required to run secure" + hadoop_error "or privileged daemons. Please download and install jsvc from " + hadoop_error "http://archive.apache.org/dist/commons/daemon/binaries/ " + hadoop_error "and set JSVC_HOME to the directory containing the jsvc binary." + exit 1 + fi + + # note that shellcheck will throw a + # bogus for-our-use-case 2086 here. + # it doesn't properly support multi-line situations + + hadoop_debug "Final CLASSPATH: ${CLASSPATH}" + hadoop_debug "Final HADOOP_OPTS: ${HADOOP_OPTS}" + hadoop_debug "Final JSVC_HOME: ${JSVC_HOME}" + hadoop_debug "jsvc: ${jsvc}" + hadoop_debug "Final HADOOP_DAEMON_JSVC_EXTRA_OPTS: ${HADOOP_DAEMON_JSVC_EXTRA_OPTS}" + hadoop_debug "Class name: ${class}" + hadoop_debug "Command line options: $*" + + #shellcheck disable=SC2086 + echo $$ > "${privpidfile}" 2>/dev/null + if [[ $? -gt 0 ]]; then + hadoop_error "ERROR: Cannot write ${daemonname} pid ${privpidfile}." + fi + + # shellcheck disable=SC2086 + exec "${jsvc}" \ + "-Dproc_${daemonname}" \ + ${HADOOP_DAEMON_JSVC_EXTRA_OPTS} \ + -outfile "${daemonoutfile}" \ + -errfile "${daemonerrfile}" \ + -pidfile "${daemonpidfile}" \ + -nodetach \ + -user "${HADOOP_SECURE_USER}" \ + -cp "${CLASSPATH}" \ + ${HADOOP_OPTS} \ + "${class}" "$@" +} + +## @description Start a privileged daemon in the background. +## @audience private +## @stability evolving +## @replaceable yes +## @param command +## @param class +## @param daemonpidfile +## @param daemonoutfile +## @param wrapperpidfile +## @param warpperoutfile +## @param daemonerrfile +## @param [options] +function hadoop_start_secure_daemon_wrapper +{ + # this wraps hadoop_start_secure_daemon to take care + # of the dirty work to launch a daemon in the background! + local daemonname=$1 + local class=$2 + + # same rules as hadoop_start_secure_daemon except we + # have some additional parameters + + local daemonpidfile=$3 + + local daemonoutfile=$4 + + # the pid file of the subprocess that spawned our + # secure launcher + local jsvcpidfile=$5 + + # the output of the subprocess that spawned our secure + # launcher + local jsvcoutfile=$6 + + local daemonerrfile=$7 + shift 7 + + local counter + + hadoop_rotate_log "${jsvcoutfile}" + + hadoop_start_secure_daemon \ + "${daemonname}" \ + "${class}" \ + "${daemonpidfile}" \ + "${daemonoutfile}" \ + "${daemonerrfile}" \ + "${jsvcpidfile}" "$@" >> "${jsvcoutfile}" 2>&1 < /dev/null & + + # we need to avoid a race condition here + # so let's wait for the fork to finish + # before overriding with the daemonized pid + (( counter=0 )) + while [[ ! -f ${daemonpidfile} && ${counter} -le 5 ]]; do + sleep 1 + (( counter++ )) + done + + #shellcheck disable=SC2086 + if ! echo $! > "${jsvcpidfile}"; then + hadoop_error "ERROR: Cannot write ${daemonname} pid ${jsvcpidfile}." + fi + + sleep 1 + #shellcheck disable=SC2086 + renice "${HADOOP_NICENESS}" $! >/dev/null 2>&1 + if [[ $? -gt 0 ]]; then + hadoop_error "ERROR: Cannot set priority of ${daemonname} process $!" + fi + if [[ -f "${daemonpidfile}" ]]; then + #shellcheck disable=SC2046 + renice "${HADOOP_NICENESS}" $(cat "${daemonpidfile}" 2>/dev/null) >/dev/null 2>&1 + if [[ $? -gt 0 ]]; then + hadoop_error "ERROR: Cannot set priority of ${daemonname} process $(cat "${daemonpidfile}" 2>/dev/null)" + fi + fi + #shellcheck disable=SC2046 + disown %+ >/dev/null 2>&1 + if [[ $? -gt 0 ]]; then + hadoop_error "ERROR: Cannot disconnect ${daemonname} process $!" + fi + # capture the ulimit output + su "${HADOOP_SECURE_USER}" -c 'bash -c "ulimit -a"' >> "${jsvcoutfile}" 2>&1 + #shellcheck disable=SC2086 + if ! ps -p $! >/dev/null 2>&1; then + return 1 + fi + return 0 +} + +## @description Wait till process dies or till timeout +## @audience private +## @stability evolving +## @param pid +## @param timeout +function wait_process_to_die_or_timeout +{ + local pid=$1 + local timeout=$2 + + # Normalize timeout + # Round up or down + timeout=$(printf "%.0f\n" "${timeout}") + if [[ ${timeout} -lt 1 ]]; then + # minimum 1 second + timeout=1 + fi + + # Wait to see if it's still alive + for (( i=0; i < "${timeout}"; i++ )) + do + if kill -0 "${pid}" > /dev/null 2>&1; then + sleep 1 + else + break + fi + done +} + +## @description Stop the non-privileged `command` daemon with that +## @description that is running at `pidfile`. +## @audience public +## @stability stable +## @replaceable yes +## @param command +## @param pidfile +function hadoop_stop_daemon +{ + local cmd=$1 + local pidfile=$2 + shift 2 + + local pid + local cur_pid + + if [[ -f "${pidfile}" ]]; then + pid=$(cat "$pidfile") + + kill "${pid}" >/dev/null 2>&1 + + wait_process_to_die_or_timeout "${pid}" "${HADOOP_STOP_TIMEOUT}" + + if kill -0 "${pid}" > /dev/null 2>&1; then + hadoop_error "WARNING: ${cmd} did not stop gracefully after ${HADOOP_STOP_TIMEOUT} seconds: Trying to kill with kill -9" + kill -9 "${pid}" >/dev/null 2>&1 + fi + wait_process_to_die_or_timeout "${pid}" "${HADOOP_STOP_TIMEOUT}" + if ps -p "${pid}" > /dev/null 2>&1; then + hadoop_error "ERROR: Unable to kill ${pid}" + else + cur_pid=$(cat "$pidfile") + if [[ "${pid}" = "${cur_pid}" ]]; then + rm -f "${pidfile}" >/dev/null 2>&1 + else + hadoop_error "WARNING: pid has changed for ${cmd}, skip deleting pid file" + fi + fi + fi +} + +## @description Stop the privileged `command` daemon with that +## @description that is running at `daemonpidfile` and launched with +## @description the wrapper at `wrapperpidfile`. +## @audience public +## @stability stable +## @replaceable yes +## @param command +## @param daemonpidfile +## @param wrapperpidfile +function hadoop_stop_secure_daemon +{ + local command=$1 + local daemonpidfile=$2 + local privpidfile=$3 + shift 3 + local ret + + local daemon_pid + local priv_pid + local cur_daemon_pid + local cur_priv_pid + + daemon_pid=$(cat "$daemonpidfile") + priv_pid=$(cat "$privpidfile") + + hadoop_stop_daemon "${command}" "${daemonpidfile}" + ret=$? + + cur_daemon_pid=$(cat "$daemonpidfile") + cur_priv_pid=$(cat "$privpidfile") + + if [[ "${daemon_pid}" = "${cur_daemon_pid}" ]]; then + rm -f "${daemonpidfile}" >/dev/null 2>&1 + else + hadoop_error "WARNING: daemon pid has changed for ${command}, skip deleting daemon pid file" + fi + + if [[ "${priv_pid}" = "${cur_priv_pid}" ]]; then + rm -f "${privpidfile}" >/dev/null 2>&1 + else + hadoop_error "WARNING: priv pid has changed for ${command}, skip deleting priv pid file" + fi + return ${ret} +} + +## @description Manage a non-privileged daemon. +## @audience private +## @stability evolving +## @replaceable yes +## @param [start|stop|status|default] +## @param command +## @param class +## @param daemonpidfile +## @param daemonoutfile +## @param [options] +function hadoop_daemon_handler +{ + local daemonmode=$1 + local daemonname=$2 + local class=$3 + local daemon_pidfile=$4 + local daemon_outfile=$5 + shift 5 + + case ${daemonmode} in + status) + hadoop_status_daemon "${daemon_pidfile}" + exit $? + ;; + + stop) + hadoop_stop_daemon "${daemonname}" "${daemon_pidfile}" + exit $? + ;; + + ##COMPAT -- older hadoops would also start daemons by default + start|default) + hadoop_verify_piddir + hadoop_verify_logdir + hadoop_status_daemon "${daemon_pidfile}" + if [[ $? == 0 ]]; then + hadoop_error "${daemonname} is running as process $(cat "${daemon_pidfile}"). Stop it first." + exit 1 + else + # stale pid file, so just remove it and continue on + rm -f "${daemon_pidfile}" >/dev/null 2>&1 + fi + ##COMPAT - differenticate between --daemon start and nothing + # "nothing" shouldn't detach + if [[ "$daemonmode" = "default" ]]; then + hadoop_start_daemon "${daemonname}" "${class}" "${daemon_pidfile}" "$@" + else + hadoop_start_daemon_wrapper "${daemonname}" \ + "${class}" "${daemon_pidfile}" "${daemon_outfile}" "$@" + fi + ;; + esac +} + +## @description Manage a privileged daemon. +## @audience private +## @stability evolving +## @replaceable yes +## @param [start|stop|status|default] +## @param command +## @param class +## @param daemonpidfile +## @param daemonoutfile +## @param wrapperpidfile +## @param wrapperoutfile +## @param wrappererrfile +## @param [options] +function hadoop_secure_daemon_handler +{ + local daemonmode=$1 + local daemonname=$2 + local classname=$3 + local daemon_pidfile=$4 + local daemon_outfile=$5 + local priv_pidfile=$6 + local priv_outfile=$7 + local priv_errfile=$8 + shift 8 + + case ${daemonmode} in + status) + hadoop_status_daemon "${daemon_pidfile}" + exit $? + ;; + + stop) + hadoop_stop_secure_daemon "${daemonname}" \ + "${daemon_pidfile}" "${priv_pidfile}" + exit $? + ;; + + ##COMPAT -- older hadoops would also start daemons by default + start|default) + hadoop_verify_piddir + hadoop_verify_logdir + hadoop_status_daemon "${daemon_pidfile}" + if [[ $? == 0 ]]; then + hadoop_error "${daemonname} is running as process $(cat "${daemon_pidfile}"). Stop it first." + exit 1 + else + # stale pid file, so just remove it and continue on + rm -f "${daemon_pidfile}" >/dev/null 2>&1 + fi + + ##COMPAT - differenticate between --daemon start and nothing + # "nothing" shouldn't detach + if [[ "${daemonmode}" = "default" ]]; then + hadoop_start_secure_daemon "${daemonname}" "${classname}" \ + "${daemon_pidfile}" "${daemon_outfile}" \ + "${priv_errfile}" "${priv_pidfile}" "$@" + else + hadoop_start_secure_daemon_wrapper "${daemonname}" "${classname}" \ + "${daemon_pidfile}" "${daemon_outfile}" \ + "${priv_pidfile}" "${priv_outfile}" "${priv_errfile}" "$@" + fi + ;; + esac +} + +## @description autodetect whether this is a priv subcmd +## @description by whether or not a priv user var exists +## @description and if HADOOP_SECURE_CLASSNAME is defined +## @audience public +## @stability stable +## @replaceable yes +## @param command +## @param subcommand +## @return 1 = not priv +## @return 0 = priv +function hadoop_detect_priv_subcmd +{ + declare program=$1 + declare command=$2 + + if [[ -z "${HADOOP_SECURE_CLASSNAME}" ]]; then + hadoop_debug "No secure classname defined." + return 1 + fi + + uvar=$(hadoop_build_custom_subcmd_var "${program}" "${command}" SECURE_USER) + if [[ -z "${!uvar}" ]]; then + hadoop_debug "No secure user defined." + return 1 + fi + return 0 +} + +## @description Build custom subcommand var +## @audience public +## @stability stable +## @replaceable yes +## @param command +## @param subcommand +## @param customid +## @return string +function hadoop_build_custom_subcmd_var +{ + declare program=$1 + declare command=$2 + declare custom=$3 + declare uprogram + declare ucommand + + if [[ -z "${BASH_VERSINFO[0]}" ]] \ + || [[ "${BASH_VERSINFO[0]}" -lt 4 ]]; then + uprogram=$(echo "${program}" | tr '[:lower:]' '[:upper:]') + ucommand=$(echo "${command}" | tr '[:lower:]' '[:upper:]') + else + uprogram=${program^^} + ucommand=${command^^} + fi + + echo "${uprogram}_${ucommand}_${custom}" +} + +## @description Verify that username in a var converts to user id +## @audience public +## @stability stable +## @replaceable yes +## @param userstring +## @return 0 for success +## @return 1 for failure +function hadoop_verify_user_resolves +{ + declare userstr=$1 + + if [[ -z ${userstr} || -z ${!userstr} ]] ; then + return 1 + fi + + id -u "${!userstr}" >/dev/null 2>&1 +} + +## @description Verify that ${USER} is allowed to execute the +## @description given subcommand. +## @audience public +## @stability stable +## @replaceable yes +## @param command +## @param subcommand +## @return return 0 on success +## @return exit 1 on failure +function hadoop_verify_user_perm +{ + declare program=$1 + declare command=$2 + declare uvar + + if [[ ${command} =~ \. ]]; then + return 1 + fi + + uvar=$(hadoop_build_custom_subcmd_var "${program}" "${command}" USER) + + if [[ -n ${!uvar} ]]; then + if [[ ${!uvar} != "${USER}" ]]; then + hadoop_error "ERROR: ${command} can only be executed by ${!uvar}." + exit 1 + fi + fi + return 0 +} + +## @description Verify that ${USER} is allowed to execute the +## @description given subcommand. +## @audience public +## @stability stable +## @replaceable yes +## @param subcommand +## @return 1 on no re-exec needed +## @return 0 on need to re-exec +function hadoop_need_reexec +{ + declare program=$1 + declare command=$2 + declare uvar + + # we've already been re-execed, bail + + if [[ "${HADOOP_REEXECED_CMD}" = true ]]; then + return 1 + fi + + if [[ ${command} =~ \. ]]; then + return 1 + fi + + # if we have privilege, and the _USER is defined, and _USER is + # set to someone who isn't us, then yes, we should re-exec. + # otherwise no, don't re-exec and let the system deal with it. + + if hadoop_privilege_check; then + uvar=$(hadoop_build_custom_subcmd_var "${program}" "${command}" USER) + if [[ -n ${!uvar} ]]; then + if [[ ${!uvar} != "${USER}" ]]; then + return 0 + fi + fi + fi + return 1 +} + +## @description Add custom (program)_(command)_OPTS to HADOOP_OPTS. +## @description Also handles the deprecated cases from pre-3.x. +## @audience public +## @stability evolving +## @replaceable yes +## @param program +## @param subcommand +## @return will exit on failure conditions +function hadoop_subcommand_opts +{ + declare program=$1 + declare command=$2 + declare uvar + declare depvar + declare uprogram + declare ucommand + + if [[ -z "${program}" || -z "${command}" ]]; then + return 1 + fi + + if [[ ${command} =~ \. ]]; then + return 1 + fi + + # bash 4 and up have built-in ways to upper and lower + # case the contents of vars. This is faster than + # calling tr. + + ## We don't call hadoop_build_custom_subcmd_var here + ## since we need to construct this for the deprecation + ## cases. For Hadoop 4.x, this needs to get cleaned up. + + if [[ -z "${BASH_VERSINFO[0]}" ]] \ + || [[ "${BASH_VERSINFO[0]}" -lt 4 ]]; then + uprogram=$(echo "${program}" | tr '[:lower:]' '[:upper:]') + ucommand=$(echo "${command}" | tr '[:lower:]' '[:upper:]') + else + uprogram=${program^^} + ucommand=${command^^} + fi + + uvar="${uprogram}_${ucommand}_OPTS" + + # Let's handle all of the deprecation cases early + # HADOOP_NAMENODE_OPTS -> HDFS_NAMENODE_OPTS + + depvar="HADOOP_${ucommand}_OPTS" + + if [[ "${depvar}" != "${uvar}" ]]; then + if [[ -n "${!depvar}" ]]; then + hadoop_deprecate_envvar "${depvar}" "${uvar}" + fi + fi + + if [[ -n ${!uvar} ]]; then + hadoop_debug "Appending ${uvar} onto HADOOP_OPTS" + HADOOP_OPTS="${HADOOP_OPTS} ${!uvar}" + return 0 + fi +} + +## @description Add custom (program)_(command)_SECURE_EXTRA_OPTS to HADOOP_OPTS. +## @description This *does not* handle the pre-3.x deprecated cases +## @audience public +## @stability stable +## @replaceable yes +## @param program +## @param subcommand +## @return will exit on failure conditions +function hadoop_subcommand_secure_opts +{ + declare program=$1 + declare command=$2 + declare uvar + declare uprogram + declare ucommand + + if [[ -z "${program}" || -z "${command}" ]]; then + return 1 + fi + + # HDFS_DATANODE_SECURE_EXTRA_OPTS + # HDFS_NFS3_SECURE_EXTRA_OPTS + # ... + uvar=$(hadoop_build_custom_subcmd_var "${program}" "${command}" SECURE_EXTRA_OPTS) + + if [[ -n ${!uvar} ]]; then + hadoop_debug "Appending ${uvar} onto HADOOP_OPTS" + HADOOP_OPTS="${HADOOP_OPTS} ${!uvar}" + return 0 + fi +} + +## @description Perform the 'hadoop classpath', etc subcommand with the given +## @description parameters +## @audience private +## @stability evolving +## @replaceable yes +## @param [parameters] +## @return will print & exit with no params +function hadoop_do_classpath_subcommand +{ + if [[ "$#" -gt 1 ]]; then + eval "$1"=org.apache.hadoop.util.Classpath + else + hadoop_finalize + echo "${CLASSPATH}" + exit 0 + fi +} + +## @description generic shell script option parser. sets +## @description HADOOP_PARSE_COUNTER to set number the +## @description caller should shift +## @audience private +## @stability evolving +## @replaceable yes +## @param [parameters, typically "$@"] +function hadoop_parse_args +{ + HADOOP_DAEMON_MODE="default" + HADOOP_PARSE_COUNTER=0 + + # not all of the options supported here are supported by all commands + # however these are: + hadoop_add_option "--config dir" "Hadoop config directory" + hadoop_add_option "--debug" "turn on shell script debug mode" + hadoop_add_option "--help" "usage information" + + while true; do + hadoop_debug "hadoop_parse_args: processing $1" + case $1 in + --buildpaths) + HADOOP_ENABLE_BUILD_PATHS=true + shift + ((HADOOP_PARSE_COUNTER=HADOOP_PARSE_COUNTER+1)) + ;; + --config) + shift + confdir=$1 + shift + ((HADOOP_PARSE_COUNTER=HADOOP_PARSE_COUNTER+2)) + if [[ -d "${confdir}" ]]; then + HADOOP_CONF_DIR="${confdir}" + elif [[ -z "${confdir}" ]]; then + hadoop_error "ERROR: No parameter provided for --config " + hadoop_exit_with_usage 1 + else + hadoop_error "ERROR: Cannot find configuration directory \"${confdir}\"" + hadoop_exit_with_usage 1 + fi + ;; + --daemon) + shift + HADOOP_DAEMON_MODE=$1 + shift + ((HADOOP_PARSE_COUNTER=HADOOP_PARSE_COUNTER+2)) + if [[ -z "${HADOOP_DAEMON_MODE}" || \ + ! "${HADOOP_DAEMON_MODE}" =~ ^st(art|op|atus)$ ]]; then + hadoop_error "ERROR: --daemon must be followed by either \"start\", \"stop\", or \"status\"." + hadoop_exit_with_usage 1 + fi + ;; + --debug) + shift + HADOOP_SHELL_SCRIPT_DEBUG=true + ((HADOOP_PARSE_COUNTER=HADOOP_PARSE_COUNTER+1)) + ;; + --help|-help|-h|help|--h|--\?|-\?|\?) + hadoop_exit_with_usage 0 + ;; + --hostnames) + shift + HADOOP_WORKER_NAMES="$1" + shift + ((HADOOP_PARSE_COUNTER=HADOOP_PARSE_COUNTER+2)) + ;; + --hosts) + shift + hadoop_populate_workers_file "$1" + shift + ((HADOOP_PARSE_COUNTER=HADOOP_PARSE_COUNTER+2)) + ;; + --loglevel) + shift + # shellcheck disable=SC2034 + HADOOP_LOGLEVEL="$1" + shift + ((HADOOP_PARSE_COUNTER=HADOOP_PARSE_COUNTER+2)) + ;; + --reexec) + shift + if [[ "${HADOOP_REEXECED_CMD}" = true ]]; then + hadoop_error "ERROR: re-exec fork bomb prevention: --reexec already called" + exit 1 + fi + HADOOP_REEXECED_CMD=true + ((HADOOP_PARSE_COUNTER=HADOOP_PARSE_COUNTER+1)) + ;; + --workers) + shift + # shellcheck disable=SC2034 + HADOOP_WORKER_MODE=true + ((HADOOP_PARSE_COUNTER=HADOOP_PARSE_COUNTER+1)) + ;; + *) + break + ;; + esac + done + + hadoop_debug "hadoop_parse: asking caller to skip ${HADOOP_PARSE_COUNTER}" +} + +## @description Handle subcommands from main program entries +## @audience private +## @stability evolving +## @replaceable yes +function hadoop_generic_java_subcmd_handler +{ + declare priv_outfile + declare priv_errfile + declare priv_pidfile + declare daemon_outfile + declare daemon_pidfile + declare secureuser + + # The default/expected way to determine if a daemon is going to run in secure + # mode is defined by hadoop_detect_priv_subcmd. If this returns true + # then setup the secure user var and tell the world we're in secure mode + + if hadoop_detect_priv_subcmd "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"; then + HADOOP_SUBCMD_SECURESERVICE=true + secureuser=$(hadoop_build_custom_subcmd_var "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}" SECURE_USER) + + if ! hadoop_verify_user_resolves "${secureuser}"; then + hadoop_error "ERROR: User defined in ${secureuser} (${!secureuser}) does not exist. Aborting." + exit 1 + fi + + HADOOP_SECURE_USER="${!secureuser}" + fi + + # check if we're running in secure mode. + # breaking this up from the above lets 3rd parties + # do things a bit different + # secure services require some extra setup + # if yes, then we need to define all of the priv and daemon stuff + # if not, then we just need to define daemon stuff. + # note the daemon vars are purposefully different between the two + + if [[ "${HADOOP_SUBCMD_SECURESERVICE}" = true ]]; then + + hadoop_subcommand_secure_opts "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}" + + hadoop_verify_secure_prereq + hadoop_setup_secure_service + priv_outfile="${HADOOP_LOG_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out" + priv_errfile="${HADOOP_LOG_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.err" + priv_pidfile="${HADOOP_PID_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid" + daemon_outfile="${HADOOP_LOG_DIR}/hadoop-${HADOOP_SECURE_USER}-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out" + daemon_pidfile="${HADOOP_PID_DIR}/hadoop-${HADOOP_SECURE_USER}-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid" + else + daemon_outfile="${HADOOP_LOG_DIR}/hadoop-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out" + daemon_pidfile="${HADOOP_PID_DIR}/hadoop-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid" + fi + + # are we actually in daemon mode? + # if yes, use the daemon logger and the appropriate log file. + if [[ "${HADOOP_DAEMON_MODE}" != "default" ]]; then + HADOOP_ROOT_LOGGER="${HADOOP_DAEMON_ROOT_LOGGER}" + if [[ "${HADOOP_SUBCMD_SECURESERVICE}" = true ]]; then + HADOOP_LOGFILE="hadoop-${HADOOP_SECURE_USER}-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.log" + else + HADOOP_LOGFILE="hadoop-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.log" + fi + fi + + # finish defining the environment: system properties, env vars, class paths, etc. + hadoop_finalize + + # do the hard work of launching a daemon or just executing our interactive + # java class + if [[ "${HADOOP_SUBCMD_SUPPORTDAEMONIZATION}" = true ]]; then + if [[ "${HADOOP_SUBCMD_SECURESERVICE}" = true ]]; then + hadoop_secure_daemon_handler \ + "${HADOOP_DAEMON_MODE}" \ + "${HADOOP_SUBCMD}" \ + "${HADOOP_SECURE_CLASSNAME}" \ + "${daemon_pidfile}" \ + "${daemon_outfile}" \ + "${priv_pidfile}" \ + "${priv_outfile}" \ + "${priv_errfile}" \ + "${HADOOP_SUBCMD_ARGS[@]}" + else + hadoop_daemon_handler \ + "${HADOOP_DAEMON_MODE}" \ + "${HADOOP_SUBCMD}" \ + "${HADOOP_CLASSNAME}" \ + "${daemon_pidfile}" \ + "${daemon_outfile}" \ + "${HADOOP_SUBCMD_ARGS[@]}" + fi + exit $? + else + hadoop_java_exec "${HADOOP_SUBCMD}" "${HADOOP_CLASSNAME}" "${HADOOP_SUBCMD_ARGS[@]}" + fi +} diff --git a/hadoop-hdds/common/src/main/bin/workers.sh b/hadoop-hdds/common/src/main/bin/workers.sh new file mode 100755 index 0000000000000..05bc5fd8f0fe0 --- /dev/null +++ b/hadoop-hdds/common/src/main/bin/workers.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Run a shell command on all worker hosts. +# +# Environment Variables +# +# HADOOP_WORKERS File naming remote hosts. +# Default is ${HADOOP_CONF_DIR}/workers. +# HADOOP_CONF_DIR Alternate conf dir. Default is ${HADOOP_HOME}/conf. +# HADOOP_WORKER_SLEEP Seconds to sleep between spawning remote commands. +# HADOOP_SSH_OPTS Options passed to ssh when running remote commands. +## + +function hadoop_usage +{ + echo "Usage: workers.sh [--config confdir] command..." +} + +# let's locate libexec... +if [[ -n "${HADOOP_HOME}" ]]; then + HADOOP_DEFAULT_LIBEXEC_DIR="${HADOOP_HOME}/libexec" +else + this="${BASH_SOURCE-$0}" + bin=$(cd -P -- "$(dirname -- "${this}")" >/dev/null && pwd -P) + HADOOP_DEFAULT_LIBEXEC_DIR="${bin}/../libexec" +fi + +HADOOP_LIBEXEC_DIR="${HADOOP_LIBEXEC_DIR:-$HADOOP_DEFAULT_LIBEXEC_DIR}" +# shellcheck disable=SC2034 +HADOOP_NEW_CONFIG=true +if [[ -f "${HADOOP_LIBEXEC_DIR}/hadoop-config.sh" ]]; then + . "${HADOOP_LIBEXEC_DIR}/hadoop-config.sh" +else + echo "ERROR: Cannot execute ${HADOOP_LIBEXEC_DIR}/hadoop-config.sh." 2>&1 + exit 1 +fi + +# if no args specified, show usage +if [[ $# -le 0 ]]; then + hadoop_exit_with_usage 1 +fi + +hadoop_connect_to_hosts "$@" diff --git a/hadoop-hdds/common/src/main/conf/core-site.xml b/hadoop-hdds/common/src/main/conf/core-site.xml new file mode 100644 index 0000000000000..d2ddf893e49eb --- /dev/null +++ b/hadoop-hdds/common/src/main/conf/core-site.xml @@ -0,0 +1,20 @@ + + + + + + + + diff --git a/hadoop-hdds/common/src/main/conf/hadoop-env.cmd b/hadoop-hdds/common/src/main/conf/hadoop-env.cmd new file mode 100644 index 0000000000000..971869597f529 --- /dev/null +++ b/hadoop-hdds/common/src/main/conf/hadoop-env.cmd @@ -0,0 +1,90 @@ +@echo off +@rem Licensed to the Apache Software Foundation (ASF) under one or more +@rem contributor license agreements. See the NOTICE file distributed with +@rem this work for additional information regarding copyright ownership. +@rem The ASF licenses this file to You under the Apache License, Version 2.0 +@rem (the "License"); you may not use this file except in compliance with +@rem the License. You may obtain a copy of the License at +@rem +@rem http://www.apache.org/licenses/LICENSE-2.0 +@rem +@rem Unless required by applicable law or agreed to in writing, software +@rem distributed under the License is distributed on an "AS IS" BASIS, +@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +@rem See the License for the specific language governing permissions and +@rem limitations under the License. + +@rem Set Hadoop-specific environment variables here. + +@rem The only required environment variable is JAVA_HOME. All others are +@rem optional. When running a distributed configuration it is best to +@rem set JAVA_HOME in this file, so that it is correctly defined on +@rem remote nodes. + +@rem The java implementation to use. Required. +set JAVA_HOME=%JAVA_HOME% + +@rem The jsvc implementation to use. Jsvc is required to run secure datanodes. +@rem set JSVC_HOME=%JSVC_HOME% + +@rem set HADOOP_CONF_DIR= + +@rem Extra Java CLASSPATH elements. Automatically insert capacity-scheduler. +if exist %HADOOP_HOME%\contrib\capacity-scheduler ( + if not defined HADOOP_CLASSPATH ( + set HADOOP_CLASSPATH=%HADOOP_HOME%\contrib\capacity-scheduler\*.jar + ) else ( + set HADOOP_CLASSPATH=%HADOOP_CLASSPATH%;%HADOOP_HOME%\contrib\capacity-scheduler\*.jar + ) +) + +@rem The maximum amount of heap to use, in MB. Default is 1000. +@rem set HADOOP_HEAPSIZE= +@rem set HADOOP_NAMENODE_INIT_HEAPSIZE="" + +@rem Extra Java runtime options. Empty by default. +@rem set HADOOP_OPTS=%HADOOP_OPTS% -Djava.net.preferIPv4Stack=true + +@rem Command specific options appended to HADOOP_OPTS when specified +if not defined HADOOP_SECURITY_LOGGER ( + set HADOOP_SECURITY_LOGGER=INFO,RFAS +) +if not defined HDFS_AUDIT_LOGGER ( + set HDFS_AUDIT_LOGGER=INFO,NullAppender +) + +set HADOOP_NAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_NAMENODE_OPTS% +set HADOOP_DATANODE_OPTS=-Dhadoop.security.logger=ERROR,RFAS %HADOOP_DATANODE_OPTS% +set HADOOP_SECONDARYNAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_SECONDARYNAMENODE_OPTS% + +@rem The following applies to multiple commands (fs, dfs, fsck, distcp etc) +set HADOOP_CLIENT_OPTS=-Xmx512m %HADOOP_CLIENT_OPTS% +@rem set HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData %HADOOP_JAVA_PLATFORM_OPTS%" + +@rem On secure datanodes, user to run the datanode as after dropping privileges +set HADOOP_SECURE_DN_USER=%HADOOP_SECURE_DN_USER% + +@rem Where log files are stored. %HADOOP_HOME%/logs by default. +@rem set HADOOP_LOG_DIR=%HADOOP_LOG_DIR%\%USERNAME% + +@rem Where log files are stored in the secure data environment. +set HADOOP_SECURE_DN_LOG_DIR=%HADOOP_LOG_DIR%\%HADOOP_HDFS_USER% + +@rem +@rem Router-based HDFS Federation specific parameters +@rem Specify the JVM options to be used when starting the RBF Routers. +@rem These options will be appended to the options specified as HADOOP_OPTS +@rem and therefore may override any similar flags set in HADOOP_OPTS +@rem +@rem set HADOOP_DFSROUTER_OPTS="" +@rem + +@rem The directory where pid files are stored. /tmp by default. +@rem NOTE: this should be set to a directory that can only be written to by +@rem the user that will run the hadoop daemons. Otherwise there is the +@rem potential for a symlink attack. +set HADOOP_PID_DIR=%HADOOP_PID_DIR% +set HADOOP_SECURE_DN_PID_DIR=%HADOOP_PID_DIR% + +@rem A string representing this instance of hadoop. %USERNAME% by default. +set HADOOP_IDENT_STRING=%USERNAME% diff --git a/hadoop-hdds/common/src/main/conf/hadoop-env.sh b/hadoop-hdds/common/src/main/conf/hadoop-env.sh new file mode 100644 index 0000000000000..e43cd95b047ee --- /dev/null +++ b/hadoop-hdds/common/src/main/conf/hadoop-env.sh @@ -0,0 +1,439 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Set Hadoop-specific environment variables here. + +## +## THIS FILE ACTS AS THE MASTER FILE FOR ALL HADOOP PROJECTS. +## SETTINGS HERE WILL BE READ BY ALL HADOOP COMMANDS. THEREFORE, +## ONE CAN USE THIS FILE TO SET YARN, HDFS, AND MAPREDUCE +## CONFIGURATION OPTIONS INSTEAD OF xxx-env.sh. +## +## Precedence rules: +## +## {yarn-env.sh|hdfs-env.sh} > hadoop-env.sh > hard-coded defaults +## +## {YARN_xyz|HDFS_xyz} > HADOOP_xyz > hard-coded defaults +## + +# Many of the options here are built from the perspective that users +# may want to provide OVERWRITING values on the command line. +# For example: +# +# JAVA_HOME=/usr/java/testing hdfs dfs -ls +# +# Therefore, the vast majority (BUT NOT ALL!) of these defaults +# are configured for substitution and not append. If append +# is preferable, modify this file accordingly. + +### +# Generic settings for HADOOP +### + +# Technically, the only required environment variable is JAVA_HOME. +# All others are optional. However, the defaults are probably not +# preferred. Many sites configure these options outside of Hadoop, +# such as in /etc/profile.d + +# The java implementation to use. By default, this environment +# variable is REQUIRED on ALL platforms except OS X! +# export JAVA_HOME= + +# Location of Hadoop. By default, Hadoop will attempt to determine +# this location based upon its execution path. +# export HADOOP_HOME= + +# Location of Hadoop's configuration information. i.e., where this +# file is living. If this is not defined, Hadoop will attempt to +# locate it based upon its execution path. +# +# NOTE: It is recommend that this variable not be set here but in +# /etc/profile.d or equivalent. Some options (such as +# --config) may react strangely otherwise. +# +# export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop + +# The maximum amount of heap to use (Java -Xmx). If no unit +# is provided, it will be converted to MB. Daemons will +# prefer any Xmx setting in their respective _OPT variable. +# There is no default; the JVM will autoscale based upon machine +# memory size. +# export HADOOP_HEAPSIZE_MAX= + +# The minimum amount of heap to use (Java -Xms). If no unit +# is provided, it will be converted to MB. Daemons will +# prefer any Xms setting in their respective _OPT variable. +# There is no default; the JVM will autoscale based upon machine +# memory size. +# export HADOOP_HEAPSIZE_MIN= + +# Enable extra debugging of Hadoop's JAAS binding, used to set up +# Kerberos security. +# export HADOOP_JAAS_DEBUG=true + +# Extra Java runtime options for all Hadoop commands. We don't support +# IPv6 yet/still, so by default the preference is set to IPv4. +# export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true" +# For Kerberos debugging, an extended option set logs more information +# export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug" + +# Some parts of the shell code may do special things dependent upon +# the operating system. We have to set this here. See the next +# section as to why.... +export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)} + +# Extra Java runtime options for some Hadoop commands +# and clients (i.e., hdfs dfs -blah). These get appended to HADOOP_OPTS for +# such commands. In most cases, # this should be left empty and +# let users supply it on the command line. +# export HADOOP_CLIENT_OPTS="" + +# +# A note about classpaths. +# +# By default, Apache Hadoop overrides Java's CLASSPATH +# environment variable. It is configured such +# that it starts out blank with new entries added after passing +# a series of checks (file/dir exists, not already listed aka +# de-deduplication). During de-deduplication, wildcards and/or +# directories are *NOT* expanded to keep it simple. Therefore, +# if the computed classpath has two specific mentions of +# awesome-methods-1.0.jar, only the first one added will be seen. +# If two directories are in the classpath that both contain +# awesome-methods-1.0.jar, then Java will pick up both versions. + +# An additional, custom CLASSPATH. Site-wide configs should be +# handled via the shellprofile functionality, utilizing the +# hadoop_add_classpath function for greater control and much +# harder for apps/end-users to accidentally override. +# Similarly, end users should utilize ${HOME}/.hadooprc . +# This variable should ideally only be used as a short-cut, +# interactive way for temporary additions on the command line. +# export HADOOP_CLASSPATH="/some/cool/path/on/your/machine" + +# Should HADOOP_CLASSPATH be first in the official CLASSPATH? +# export HADOOP_USER_CLASSPATH_FIRST="yes" + +# If HADOOP_USE_CLIENT_CLASSLOADER is set, the classpath along +# with the main jar are handled by a separate isolated +# client classloader when 'hadoop jar', 'yarn jar', or 'mapred job' +# is utilized. If it is set, HADOOP_CLASSPATH and +# HADOOP_USER_CLASSPATH_FIRST are ignored. +# export HADOOP_USE_CLIENT_CLASSLOADER=true + +# HADOOP_CLIENT_CLASSLOADER_SYSTEM_CLASSES overrides the default definition of +# system classes for the client classloader when HADOOP_USE_CLIENT_CLASSLOADER +# is enabled. Names ending in '.' (period) are treated as package names, and +# names starting with a '-' are treated as negative matches. For example, +# export HADOOP_CLIENT_CLASSLOADER_SYSTEM_CLASSES="-org.apache.hadoop.UserClass,java.,javax.,org.apache.hadoop." + +# Enable optional, bundled Hadoop features +# This is a comma delimited list. It may NOT be overridden via .hadooprc +# Entries may be added/removed as needed. +# export HADOOP_OPTIONAL_TOOLS="@@@HADOOP_OPTIONAL_TOOLS@@@" + +### +# Options for remote shell connectivity +### + +# There are some optional components of hadoop that allow for +# command and control of remote hosts. For example, +# start-dfs.sh will attempt to bring up all NNs, DNS, etc. + +# Options to pass to SSH when one of the "log into a host and +# start/stop daemons" scripts is executed +# export HADOOP_SSH_OPTS="-o BatchMode=yes -o StrictHostKeyChecking=no -o ConnectTimeout=10s" + +# The built-in ssh handler will limit itself to 10 simultaneous connections. +# For pdsh users, this sets the fanout size ( -f ) +# Change this to increase/decrease as necessary. +# export HADOOP_SSH_PARALLEL=10 + +# Filename which contains all of the hosts for any remote execution +# helper scripts # such as workers.sh, start-dfs.sh, etc. +# export HADOOP_WORKERS="${HADOOP_CONF_DIR}/workers" + +### +# Options for all daemons +### +# + +# +# Many options may also be specified as Java properties. It is +# very common, and in many cases, desirable, to hard-set these +# in daemon _OPTS variables. Where applicable, the appropriate +# Java property is also identified. Note that many are re-used +# or set differently in certain contexts (e.g., secure vs +# non-secure) +# + +# Where (primarily) daemon log files are stored. +# ${HADOOP_HOME}/logs by default. +# Java property: hadoop.log.dir +# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs + +# A string representing this instance of hadoop. $USER by default. +# This is used in writing log and pid files, so keep that in mind! +# Java property: hadoop.id.str +# export HADOOP_IDENT_STRING=$USER + +# How many seconds to pause after stopping a daemon +# export HADOOP_STOP_TIMEOUT=5 + +# Where pid files are stored. /tmp by default. +# export HADOOP_PID_DIR=/tmp + +# Default log4j setting for interactive commands +# Java property: hadoop.root.logger +# export HADOOP_ROOT_LOGGER=INFO,console + +# Default log4j setting for daemons spawned explicitly by +# --daemon option of hadoop, hdfs, mapred and yarn command. +# Java property: hadoop.root.logger +# export HADOOP_DAEMON_ROOT_LOGGER=INFO,RFA + +# Default log level and output location for security-related messages. +# You will almost certainly want to change this on a per-daemon basis via +# the Java property (i.e., -Dhadoop.security.logger=foo). (Note that the +# defaults for the NN and 2NN override this by default.) +# Java property: hadoop.security.logger +# export HADOOP_SECURITY_LOGGER=INFO,NullAppender + +# Default process priority level +# Note that sub-processes will also run at this level! +# export HADOOP_NICENESS=0 + +# Default name for the service level authorization file +# Java property: hadoop.policy.file +# export HADOOP_POLICYFILE="hadoop-policy.xml" + +# +# NOTE: this is not used by default! <----- +# You can define variables right here and then re-use them later on. +# For example, it is common to use the same garbage collection settings +# for all the daemons. So one could define: +# +# export HADOOP_GC_SETTINGS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps" +# +# .. and then use it as per the b option under the namenode. + +### +# Secure/privileged execution +### + +# +# Out of the box, Hadoop uses jsvc from Apache Commons to launch daemons +# on privileged ports. This functionality can be replaced by providing +# custom functions. See hadoop-functions.sh for more information. +# + +# The jsvc implementation to use. Jsvc is required to run secure datanodes +# that bind to privileged ports to provide authentication of data transfer +# protocol. Jsvc is not required if SASL is configured for authentication of +# data transfer protocol using non-privileged ports. +# export JSVC_HOME=/usr/bin + +# +# This directory contains pids for secure and privileged processes. +#export HADOOP_SECURE_PID_DIR=${HADOOP_PID_DIR} + +# +# This directory contains the logs for secure and privileged processes. +# Java property: hadoop.log.dir +# export HADOOP_SECURE_LOG=${HADOOP_LOG_DIR} + +# +# When running a secure daemon, the default value of HADOOP_IDENT_STRING +# ends up being a bit bogus. Therefore, by default, the code will +# replace HADOOP_IDENT_STRING with HADOOP_xx_SECURE_USER. If one wants +# to keep HADOOP_IDENT_STRING untouched, then uncomment this line. +# export HADOOP_SECURE_IDENT_PRESERVE="true" + +### +# NameNode specific parameters +### + +# Default log level and output location for file system related change +# messages. For non-namenode daemons, the Java property must be set in +# the appropriate _OPTS if one wants something other than INFO,NullAppender +# Java property: hdfs.audit.logger +# export HDFS_AUDIT_LOGGER=INFO,NullAppender + +# Specify the JVM options to be used when starting the NameNode. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# a) Set JMX options +# export HDFS_NAMENODE_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1026" +# +# b) Set garbage collection logs +# export HDFS_NAMENODE_OPTS="${HADOOP_GC_SETTINGS} -Xloggc:${HADOOP_LOG_DIR}/gc-rm.log-$(date +'%Y%m%d%H%M')" +# +# c) ... or set them directly +# export HDFS_NAMENODE_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:${HADOOP_LOG_DIR}/gc-rm.log-$(date +'%Y%m%d%H%M')" + +# this is the default: +# export HDFS_NAMENODE_OPTS="-Dhadoop.security.logger=INFO,RFAS" + +### +# SecondaryNameNode specific parameters +### +# Specify the JVM options to be used when starting the SecondaryNameNode. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# This is the default: +# export HDFS_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=INFO,RFAS" + +### +# DataNode specific parameters +### +# Specify the JVM options to be used when starting the DataNode. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# This is the default: +# export HDFS_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS" + +# On secure datanodes, user to run the datanode as after dropping privileges. +# This **MUST** be uncommented to enable secure HDFS if using privileged ports +# to provide authentication of data transfer protocol. This **MUST NOT** be +# defined if SASL is configured for authentication of data transfer protocol +# using non-privileged ports. +# This will replace the hadoop.id.str Java property in secure mode. +# export HDFS_DATANODE_SECURE_USER=hdfs + +# Supplemental options for secure datanodes +# By default, Hadoop uses jsvc which needs to know to launch a +# server jvm. +# export HDFS_DATANODE_SECURE_EXTRA_OPTS="-jvm server" + +### +# NFS3 Gateway specific parameters +### +# Specify the JVM options to be used when starting the NFS3 Gateway. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# export HDFS_NFS3_OPTS="" + +# Specify the JVM options to be used when starting the Hadoop portmapper. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# export HDFS_PORTMAP_OPTS="-Xmx512m" + +# Supplemental options for priviliged gateways +# By default, Hadoop uses jsvc which needs to know to launch a +# server jvm. +# export HDFS_NFS3_SECURE_EXTRA_OPTS="-jvm server" + +# On privileged gateways, user to run the gateway as after dropping privileges +# This will replace the hadoop.id.str Java property in secure mode. +# export HDFS_NFS3_SECURE_USER=nfsserver + +### +# ZKFailoverController specific parameters +### +# Specify the JVM options to be used when starting the ZKFailoverController. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# export HDFS_ZKFC_OPTS="" + +### +# QuorumJournalNode specific parameters +### +# Specify the JVM options to be used when starting the QuorumJournalNode. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# export HDFS_JOURNALNODE_OPTS="" + +### +# HDFS Balancer specific parameters +### +# Specify the JVM options to be used when starting the HDFS Balancer. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# export HDFS_BALANCER_OPTS="" + +### +# HDFS Mover specific parameters +### +# Specify the JVM options to be used when starting the HDFS Mover. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# export HDFS_MOVER_OPTS="" + +### +# Router-based HDFS Federation specific parameters +# Specify the JVM options to be used when starting the RBF Routers. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# export HDFS_DFSROUTER_OPTS="" + +### +# Ozone Manager specific parameters +### +# Specify the JVM options to be used when starting the Ozone Manager. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# export HDFS_OM_OPTS="" + +### +# HDFS StorageContainerManager specific parameters +### +# Specify the JVM options to be used when starting the HDFS Storage Container Manager. +# These options will be appended to the options specified as HADOOP_OPTS +# and therefore may override any similar flags set in HADOOP_OPTS +# +# export HDFS_STORAGECONTAINERMANAGER_OPTS="" + +### +# Advanced Users Only! +### + +# +# When building Hadoop, one can add the class paths to the commands +# via this special env var: +# export HADOOP_ENABLE_BUILD_PATHS="true" + +# +# To prevent accidents, shell commands be (superficially) locked +# to only allow certain users to execute certain subcommands. +# It uses the format of (command)_(subcommand)_USER. +# +# For example, to limit who can execute the namenode command, +# export HDFS_NAMENODE_USER=hdfs + + +### +# Registry DNS specific parameters +### +# For privileged registry DNS, user to run as after dropping privileges +# This will replace the hadoop.id.str Java property in secure mode. +# export HADOOP_REGISTRYDNS_SECURE_USER=yarn + +# Supplemental options for privileged registry DNS +# By default, Hadoop uses jsvc which needs to know to launch a +# server jvm. +# export HADOOP_REGISTRYDNS_SECURE_EXTRA_OPTS="-jvm server" diff --git a/hadoop-hdds/common/src/main/conf/hadoop-metrics2.properties b/hadoop-hdds/common/src/main/conf/hadoop-metrics2.properties new file mode 100644 index 0000000000000..f67bf8e4c5b1f --- /dev/null +++ b/hadoop-hdds/common/src/main/conf/hadoop-metrics2.properties @@ -0,0 +1,99 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# syntax: [prefix].[source|sink].[instance].[options] +# See javadoc of package-info.java for org.apache.hadoop.metrics2 for details + +*.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink +# default sampling period, in seconds +*.period=10 + +# The namenode-metrics.out will contain metrics from all context +#namenode.sink.file.filename=namenode-metrics.out +# Specifying a special sampling period for namenode: +#namenode.sink.*.period=8 + +#datanode.sink.file.filename=datanode-metrics.out + +#resourcemanager.sink.file.filename=resourcemanager-metrics.out + +#nodemanager.sink.file.filename=nodemanager-metrics.out + +#mrappmaster.sink.file.filename=mrappmaster-metrics.out + +#jobhistoryserver.sink.file.filename=jobhistoryserver-metrics.out + +# the following example split metrics of different +# context to different sinks (in this case files) +#nodemanager.sink.file_jvm.class=org.apache.hadoop.metrics2.sink.FileSink +#nodemanager.sink.file_jvm.context=jvm +#nodemanager.sink.file_jvm.filename=nodemanager-jvm-metrics.out +#nodemanager.sink.file_mapred.class=org.apache.hadoop.metrics2.sink.FileSink +#nodemanager.sink.file_mapred.context=mapred +#nodemanager.sink.file_mapred.filename=nodemanager-mapred-metrics.out + +# +# Below are for sending metrics to Ganglia +# +# for Ganglia 3.0 support +# *.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink30 +# +# for Ganglia 3.1 support +# *.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31 + +# *.sink.ganglia.period=10 + +# default for supportsparse is false +# *.sink.ganglia.supportsparse=true + +#*.sink.ganglia.slope=jvm.metrics.gcCount=zero,jvm.metrics.memHeapUsedM=both +#*.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40 + +# Tag values to use for the ganglia prefix. If not defined no tags are used. +# If '*' all tags are used. If specifying multiple tags separate them with +# commas. Note that the last segment of the property name is the context name. +# +# A typical use of tags is separating the metrics by the HDFS rpc port +# and HDFS service rpc port. +# For example: +# With following HDFS configuration: +# dfs.namenode.rpc-address is set as namenodeAddress:9110 +# dfs.namenode.servicerpc-address is set as namenodeAddress:9111 +# If no tags are used, following metric would be gathered: +# rpc.rpc.NumOpenConnections +# If using "*.sink.ganglia.tagsForPrefix.rpc=port", +# following metrics would be gathered: +# rpc.rpc.port=9110.NumOpenConnections +# rpc.rpc.port=9111.NumOpenConnections +# +#*.sink.ganglia.tagsForPrefix.jvm=ProcessName +#*.sink.ganglia.tagsForPrefix.dfs=HAState,IsOutOfSync +#*.sink.ganglia.tagsForPrefix.rpc=port +#*.sink.ganglia.tagsForPrefix.rpcdetailed=port +#*.sink.ganglia.tagsForPrefix.metricssystem=* +#*.sink.ganglia.tagsForPrefix.ugi=* +#*.sink.ganglia.tagsForPrefix.mapred= + +#namenode.sink.ganglia.servers=yourgangliahost_1:8649,yourgangliahost_2:8649 + +#datanode.sink.ganglia.servers=yourgangliahost_1:8649,yourgangliahost_2:8649 + +#resourcemanager.sink.ganglia.servers=yourgangliahost_1:8649,yourgangliahost_2:8649 + +#nodemanager.sink.ganglia.servers=yourgangliahost_1:8649,yourgangliahost_2:8649 + +#mrappmaster.sink.ganglia.servers=yourgangliahost_1:8649,yourgangliahost_2:8649 + +#jobhistoryserver.sink.ganglia.servers=yourgangliahost_1:8649,yourgangliahost_2:8649 diff --git a/hadoop-hdds/common/src/main/conf/hadoop-policy.xml b/hadoop-hdds/common/src/main/conf/hadoop-policy.xml new file mode 100644 index 0000000000000..85e4975a78628 --- /dev/null +++ b/hadoop-hdds/common/src/main/conf/hadoop-policy.xml @@ -0,0 +1,275 @@ + + + + + + + + + security.client.protocol.acl + * + ACL for ClientProtocol, which is used by user code + via the DistributedFileSystem. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.client.datanode.protocol.acl + * + ACL for ClientDatanodeProtocol, the client-to-datanode protocol + for block recovery. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.datanode.protocol.acl + * + ACL for DatanodeProtocol, which is used by datanodes to + communicate with the namenode. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.inter.datanode.protocol.acl + * + ACL for InterDatanodeProtocol, the inter-datanode protocol + for updating generation timestamp. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.namenode.protocol.acl + * + ACL for NamenodeProtocol, the protocol used by the secondary + namenode to communicate with the namenode. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.admin.operations.protocol.acl + * + ACL for AdminOperationsProtocol. Used for admin commands. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.refresh.user.mappings.protocol.acl + * + ACL for RefreshUserMappingsProtocol. Used to refresh + users mappings. The ACL is a comma-separated list of user and + group names. The user and group list is separated by a blank. For + e.g. "alice,bob users,wheel". A special value of "*" means all + users are allowed. + + + + security.refresh.policy.protocol.acl + * + ACL for RefreshAuthorizationPolicyProtocol, used by the + dfsadmin and mradmin commands to refresh the security policy in-effect. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.ha.service.protocol.acl + * + ACL for HAService protocol used by HAAdmin to manage the + active and stand-by states of namenode. + + + + security.router.admin.protocol.acl + * + ACL for RouterAdmin Protocol. The ACL is a comma-separated + list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + + security.zkfc.protocol.acl + * + ACL for access to the ZK Failover Controller + + + + + security.qjournal.service.protocol.acl + * + ACL for QJournalProtocol, used by the NN to communicate with + JNs when using the QuorumJournalManager for edit logs. + + + + security.interqjournal.service.protocol.acl + * + ACL for InterQJournalProtocol, used by the JN to + communicate with other JN + + + + + security.mrhs.client.protocol.acl + * + ACL for HSClientProtocol, used by job clients to + communciate with the MR History Server job status etc. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + + + security.resourcetracker.protocol.acl + * + ACL for ResourceTrackerProtocol, used by the + ResourceManager and NodeManager to communicate with each other. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.resourcemanager-administration.protocol.acl + * + ACL for ResourceManagerAdministrationProtocol, for admin commands. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.applicationclient.protocol.acl + * + ACL for ApplicationClientProtocol, used by the ResourceManager + and applications submission clients to communicate with each other. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.applicationmaster.protocol.acl + * + ACL for ApplicationMasterProtocol, used by the ResourceManager + and ApplicationMasters to communicate with each other. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.containermanagement.protocol.acl + * + ACL for ContainerManagementProtocol protocol, used by the NodeManager + and ApplicationMasters to communicate with each other. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.resourcelocalizer.protocol.acl + * + ACL for ResourceLocalizer protocol, used by the NodeManager + and ResourceLocalizer to communicate with each other. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.job.task.protocol.acl + * + ACL for TaskUmbilicalProtocol, used by the map and reduce + tasks to communicate with the parent tasktracker. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.job.client.protocol.acl + * + ACL for MRClientProtocol, used by job clients to + communciate with the MR ApplicationMaster to query job status etc. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.applicationhistory.protocol.acl + * + ACL for ApplicationHistoryProtocol, used by the timeline + server and the generic history service client to communicate with each other. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.collector-nodemanager.protocol.acl + * + ACL for CollectorNodemanagerProtocol, used by nodemanager + if timeline service v2 is enabled, for the timeline collector and nodemanager + to communicate with each other. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.applicationmaster-nodemanager.applicationmaster.protocol.acl + * + ACL for ApplicationMasterProtocol, used by the Nodemanager + and ApplicationMasters to communicate. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + + + security.distributedscheduling.protocol.acl + * + ACL for DistributedSchedulingAMProtocol, used by the Nodemanager + and Resourcemanager to communicate. + The ACL is a comma-separated list of user and group names. The user and + group list is separated by a blank. For e.g. "alice,bob users,wheel". + A special value of "*" means all users are allowed. + + diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java index 83e270b3bb309..99972ae900389 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java @@ -16,7 +16,7 @@ */ package org.apache.hadoop.hdds; -import org.apache.hadoop.utils.db.DBProfile; +import org.apache.hadoop.hdds.utils.db.DBProfile; /** * This class contains constants for configuration keys and default values @@ -65,15 +65,12 @@ public final class HddsConfigKeys { public static final float HDDS_CONTAINER_CLOSE_THRESHOLD_DEFAULT = 0.9f; public static final String HDDS_SCM_SAFEMODE_ENABLED = "hdds.scm.safemode.enabled"; - public static final String HDDS_CONTAINERSCRUB_ENABLED = - "hdds.containerscrub.enabled"; - public static final boolean HDDS_CONTAINERSCRUB_ENABLED_DEFAULT = false; + public static final boolean HDDS_SCM_SAFEMODE_ENABLED_DEFAULT = true; public static final String HDDS_SCM_SAFEMODE_MIN_DATANODE = "hdds.scm.safemode.min.datanode"; public static final int HDDS_SCM_SAFEMODE_MIN_DATANODE_DEFAULT = 1; - public static final String HDDS_SCM_WAIT_TIME_AFTER_SAFE_MODE_EXIT = "hdds.scm.wait.time.after.safemode.exit"; @@ -179,34 +176,18 @@ public final class HddsConfigKeys { private HddsConfigKeys() { } + // Enable TLS for GRPC clients/server in ozone. public static final String HDDS_GRPC_TLS_ENABLED = "hdds.grpc.tls.enabled"; public static final boolean HDDS_GRPC_TLS_ENABLED_DEFAULT = false; - public static final String HDDS_GRPC_MUTUAL_TLS_REQUIRED = - "hdds.grpc.mutual.tls.required"; - public static final boolean HDDS_GRPC_MUTUAL_TLS_REQUIRED_DEFAULT = false; - + // Choose TLS provider the default is set to OPENSSL for better performance. public static final String HDDS_GRPC_TLS_PROVIDER = "hdds.grpc.tls.provider"; public static final String HDDS_GRPC_TLS_PROVIDER_DEFAULT = "OPENSSL"; - public static final String HDDS_TRUST_STORE_FILE_NAME = - "hdds.trust.cert.collection.file.name"; - public static final String HDDS_TRUST_STORE_FILE_NAME_DEFAULT = "ca.crt"; - - public static final String - HDDS_SERVER_CERTIFICATE_CHAIN_FILE_NAME = - "hdds.server.cert.chain.file.name"; - public static final String - HDDS_SERVER_CERTIFICATE_CHAIN_FILE_NAME_DEFAULT = "server.crt"; - - public static final String - HDDS_CLIENT_CERTIFICATE_CHAIN_FILE_NAME = - "hdds.client.cert.chain.file.name"; - public static final String - HDDS_CLIENT_CERTIFICATE_CHAIN_FILE_NAME_DEFAULT = "client.crt"; - + // Test only settings for using test signed certificate, authority assume to + // be localhost. public static final String HDDS_GRPC_TLS_TEST_CERT = "hdds.grpc.tls" + - ".test_cert"; + ".test.cert"; public static final boolean HDDS_GRPC_TLS_TEST_CERT_DEFAULT = false; // Comma separated acls (users, groups) allowing clients accessing @@ -238,6 +219,16 @@ private HddsConfigKeys() { public static final String HDDS_SECURITY_CLIENT_SCM_CERTIFICATE_PROTOCOL_ACL = "hdds.security.client.scm.certificate.protocol.acl"; + // Determines if the Container Chunk Manager will write user data to disk + // Set to false only for specific performance tests + public static final String HDDS_CONTAINER_PERSISTDATA = + "hdds.container.chunk.persistdata"; + public static final boolean HDDS_CONTAINER_PERSISTDATA_DEFAULT = true; + + public static final String HDDS_CONTAINER_SCRUB_ENABLED = + "hdds.container.scrub.enabled"; + public static final boolean HDDS_CONTAINER_SCRUB_ENABLED_DEFAULT = false; + public static final String HDDS_DATANODE_HTTP_ENABLED_KEY = "hdds.datanode.http.enabled"; public static final String HDDS_DATANODE_HTTP_BIND_HOST_KEY = diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java index 92ed9b61630c9..d7b20fdd9172c 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java @@ -30,6 +30,7 @@ import java.util.Map; import java.util.Optional; import java.util.TimeZone; +import java.util.concurrent.TimeUnit; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; @@ -42,9 +43,15 @@ import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdds.protocol.SCMSecurityProtocol; import org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolPB; +import org.apache.hadoop.hdfs.DFSConfigKeys; +import org.apache.hadoop.io.retry.RetryPolicies; +import org.apache.hadoop.io.retry.RetryPolicy; import org.apache.hadoop.ipc.Client; import org.apache.hadoop.ipc.ProtobufRpcEngine; import org.apache.hadoop.ipc.RPC; +import org.apache.hadoop.metrics2.MetricsSystem; +import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; +import org.apache.hadoop.metrics2.source.JvmMetrics; import org.apache.hadoop.metrics2.util.MBeans; import org.apache.hadoop.net.DNS; import org.apache.hadoop.net.NetUtils; @@ -173,23 +180,27 @@ public static InetSocketAddress getScmAddressForBlockClients( /** * Create a scm security client. * @param conf - Ozone configuration. - * @param address - inet socket address of scm. * * @return {@link SCMSecurityProtocol} * @throws IOException */ - public static SCMSecurityProtocol getScmSecurityClient( - OzoneConfiguration conf, InetSocketAddress address) throws IOException { + public static SCMSecurityProtocolClientSideTranslatorPB getScmSecurityClient( + OzoneConfiguration conf) throws IOException { RPC.setProtocolEngine(conf, SCMSecurityProtocolPB.class, ProtobufRpcEngine.class); long scmVersion = RPC.getProtocolVersion(ScmBlockLocationProtocolPB.class); + InetSocketAddress address = + getScmAddressForSecurityProtocol(conf); + RetryPolicy retryPolicy = + RetryPolicies.retryForeverWithFixedSleep( + 1000, TimeUnit.MILLISECONDS); SCMSecurityProtocolClientSideTranslatorPB scmSecurityClient = new SCMSecurityProtocolClientSideTranslatorPB( - RPC.getProxy(SCMSecurityProtocolPB.class, scmVersion, + RPC.getProtocolProxy(SCMSecurityProtocolPB.class, scmVersion, address, UserGroupInformation.getCurrentUser(), conf, NetUtils.getDefaultSocketFactory(conf), - Client.getRpcTimeout(conf))); + Client.getRpcTimeout(conf), retryPolicy).getProxy()); return scmSecurityClient; } @@ -226,7 +237,12 @@ public static Optional getHostName(String value) { if ((value == null) || value.isEmpty()) { return Optional.empty(); } - return Optional.of(HostAndPort.fromString(value).getHostText()); + String hostname = value.replaceAll("\\:[0-9]+$", ""); + if (hostname.length() == 0) { + return Optional.empty(); + } else { + return Optional.of(hostname); + } } /** @@ -407,8 +423,10 @@ public static ObjectName registerWithJmxProperties( InvocationTargetException e) { // Fallback - LOG.trace("Registering MBean {} without additional properties {}", - mBeanName, jmxProperties); + if (LOG.isTraceEnabled()) { + LOG.trace("Registering MBean {} without additional properties {}", + mBeanName, jmxProperties); + } return MBeans.register(serviceName, mBeanName, mBean); } } @@ -470,4 +488,18 @@ public static InetSocketAddress getScmAddressForSecurityProtocol( .orElse(ScmConfigKeys.OZONE_SCM_SECURITY_SERVICE_PORT_DEFAULT)); } + /** + * Initialize hadoop metrics system for Ozone servers. + * @param configuration OzoneConfiguration to use. + * @param serverName The logical name of the server components. + * @return + */ + public static MetricsSystem initializeMetrics( + OzoneConfiguration configuration, String serverName) { + MetricsSystem metricsSystem = DefaultMetricsSystem.initialize(serverName); + JvmMetrics.create(serverName, + configuration.get(DFSConfigKeys.DFS_METRICS_SESSION_ID_KEY), + DefaultMetricsSystem.instance()); + return metricsSystem; + } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/cli/HddsVersionProvider.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/cli/HddsVersionProvider.java index ef7a56e6b2b6b..2f4ac4f170a83 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/cli/HddsVersionProvider.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/cli/HddsVersionProvider.java @@ -17,7 +17,7 @@ */ package org.apache.hadoop.hdds.cli; -import org.apache.hadoop.utils.HddsVersionInfo; +import org.apache.hadoop.hdds.utils.HddsVersionInfo; import picocli.CommandLine.IVersionProvider; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/ReplicationFactor.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/ReplicationFactor.java index 0215964ab8e77..044bd6f8334cd 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/ReplicationFactor.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/ReplicationFactor.java @@ -18,6 +18,8 @@ package org.apache.hadoop.hdds.client; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; + /** * The replication factor to be used while writing key into ozone. */ @@ -53,6 +55,22 @@ public static ReplicationFactor valueOf(int value) { throw new IllegalArgumentException("Unsupported value: " + value); } + public static ReplicationFactor fromProto( + HddsProtos.ReplicationFactor replicationFactor) { + if (replicationFactor == null) { + return null; + } + switch (replicationFactor) { + case ONE: + return ReplicationFactor.ONE; + case THREE: + return ReplicationFactor.THREE; + default: + throw new IllegalArgumentException( + "Unsupported ProtoBuf replication factor: " + replicationFactor); + } + } + /** * Returns integer representation of ReplicationFactor. * @return replication value diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/ReplicationType.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/ReplicationType.java index 259a1a2931742..c63896e9e1d13 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/ReplicationType.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/ReplicationType.java @@ -18,11 +18,31 @@ package org.apache.hadoop.hdds.client; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; + /** * The replication type to be used while writing key into ozone. */ public enum ReplicationType { RATIS, STAND_ALONE, - CHAINED + CHAINED; + + public static ReplicationType fromProto( + HddsProtos.ReplicationType replicationType) { + if (replicationType == null) { + return null; + } + switch (replicationType) { + case RATIS: + return ReplicationType.RATIS; + case STAND_ALONE: + return ReplicationType.STAND_ALONE; + case CHAINED: + return ReplicationType.CHAINED; + default: + throw new IllegalArgumentException( + "Unsupported ProtoBuf replication type: " + replicationType); + } + } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/HddsConfServlet.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/HddsConfServlet.java index 712e8d37e0462..8beac1663b2b7 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/HddsConfServlet.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/HddsConfServlet.java @@ -171,7 +171,9 @@ private void processConfigTagRequest(HttpServletRequest request, Properties properties = config.getAllPropertiesByTag(tag); propMap.put(tag, properties); } else { - LOG.debug("Not a valid tag" + tag); + if (LOG.isDebugEnabled()) { + LOG.debug("Not a valid tag" + tag); + } } } out.write(gson.toJsonTree(propMap).toString()); diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java index b32ad63353553..c0486335cdd2a 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java @@ -34,6 +34,7 @@ import java.util.List; import java.util.Properties; +import com.google.common.base.Preconditions; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.conf.Configuration; @@ -46,6 +47,14 @@ public class OzoneConfiguration extends Configuration { activate(); } + public static OzoneConfiguration of(Configuration conf) { + Preconditions.checkNotNull(conf); + + return conf instanceof OzoneConfiguration + ? (OzoneConfiguration) conf + : new OzoneConfiguration(conf); + } + public OzoneConfiguration() { OzoneConfiguration.activate(); loadDefaults(); diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/function/FunctionWithServiceException.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/function/FunctionWithServiceException.java new file mode 100644 index 0000000000000..b9d7bceb48f95 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/function/FunctionWithServiceException.java @@ -0,0 +1,36 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

    + * http://www.apache.org/licenses/LICENSE-2.0 + *

    + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.function; + +import com.google.protobuf.ServiceException; + +/** + * Functional interface like java.util.function.Function but with + * checked exception. + */ +@FunctionalInterface +public interface FunctionWithServiceException { + + /** + * Applies this function to the given argument. + * + * @param t the function argument + * @return the function result + */ + R apply(T t) throws ServiceException; +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/function/package-info.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/function/package-info.java new file mode 100644 index 0000000000000..915fe3557e2ce --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/function/package-info.java @@ -0,0 +1,22 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * Functional interfaces for ozone, similar to java.util.function. + */ +package org.apache.hadoop.hdds.function; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java index 1dfeecd47ac9c..698a443fc6b44 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java @@ -19,9 +19,12 @@ package org.apache.hadoop.hdds.protocol; import com.google.common.base.Preconditions; +import com.google.common.base.Strings; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.hdds.protocol.proto.HddsProtos; +import org.apache.hadoop.hdds.scm.net.NetConstants; +import org.apache.hadoop.hdds.scm.net.NodeImpl; import java.util.ArrayList; import java.util.List; @@ -35,9 +38,9 @@ */ @InterfaceAudience.Private @InterfaceStability.Evolving -public class DatanodeDetails implements Comparable { - - /** +public class DatanodeDetails extends NodeImpl implements + Comparable { +/** * DataNode's unique identifier in the cluster. */ private final UUID uuid; @@ -47,18 +50,19 @@ public class DatanodeDetails implements Comparable { private List ports; private String certSerialId; - /** * Constructs DatanodeDetails instance. DatanodeDetails.Builder is used * for instantiating DatanodeDetails. * @param uuid DataNode's UUID * @param ipAddress IP Address of this DataNode * @param hostName DataNode's hostname + * @param networkLocation DataNode's network location path * @param ports Ports used by the DataNode * @param certSerialId serial id from SCM issued certificate. */ private DatanodeDetails(String uuid, String ipAddress, String hostName, - List ports, String certSerialId) { + String networkLocation, List ports, String certSerialId) { + super(hostName, networkLocation, NetConstants.NODE_COST_DEFAULT); this.uuid = UUID.fromString(uuid); this.ipAddress = ipAddress; this.hostName = hostName; @@ -67,10 +71,13 @@ private DatanodeDetails(String uuid, String ipAddress, String hostName, } protected DatanodeDetails(DatanodeDetails datanodeDetails) { + super(datanodeDetails.getHostName(), datanodeDetails.getNetworkLocation(), + datanodeDetails.getCost()); this.uuid = datanodeDetails.uuid; this.ipAddress = datanodeDetails.ipAddress; this.hostName = datanodeDetails.hostName; this.ports = datanodeDetails.ports; + this.setNetworkName(datanodeDetails.getNetworkName()); } /** @@ -187,6 +194,12 @@ public static DatanodeDetails getFromProtoBuf( builder.addPort(newPort( Port.Name.valueOf(port.getName().toUpperCase()), port.getValue())); } + if (datanodeDetailsProto.hasNetworkName()) { + builder.setNetworkName(datanodeDetailsProto.getNetworkName()); + } + if (datanodeDetailsProto.hasNetworkLocation()) { + builder.setNetworkLocation(datanodeDetailsProto.getNetworkLocation()); + } return builder.build(); } @@ -207,6 +220,13 @@ public HddsProtos.DatanodeDetailsProto getProtoBufMessage() { if (certSerialId != null) { builder.setCertSerialId(certSerialId); } + if (!Strings.isNullOrEmpty(getNetworkName())) { + builder.setNetworkName(getNetworkName()); + } + if (!Strings.isNullOrEmpty(getNetworkLocation())) { + builder.setNetworkLocation(getNetworkLocation()); + } + for (Port port : ports) { builder.addPorts(HddsProtos.Port.newBuilder() .setName(port.getName().toString()) @@ -223,6 +243,8 @@ public String toString() { ipAddress + ", host: " + hostName + + ", networkLocation: " + + getNetworkLocation() + ", certSerialId: " + certSerialId + "}"; } @@ -259,6 +281,8 @@ public static final class Builder { private String id; private String ipAddress; private String hostName; + private String networkName; + private String networkLocation; private List ports; private String certSerialId; @@ -303,6 +327,28 @@ public Builder setHostName(String host) { return this; } + /** + * Sets the network name of DataNode. + * + * @param name network name + * @return DatanodeDetails.Builder + */ + public Builder setNetworkName(String name) { + this.networkName = name; + return this; + } + + /** + * Sets the network location of DataNode. + * + * @param loc location + * @return DatanodeDetails.Builder + */ + public Builder setNetworkLocation(String loc) { + this.networkLocation = loc; + return this; + } + /** * Adds a DataNode Port. * @@ -334,9 +380,16 @@ public Builder setCertSerialId(String certId) { */ public DatanodeDetails build() { Preconditions.checkNotNull(id); - return new DatanodeDetails(id, ipAddress, hostName, ports, certSerialId); + if (networkLocation == null) { + networkLocation = NetConstants.DEFAULT_RACK; + } + DatanodeDetails dn = new DatanodeDetails(id, ipAddress, hostName, + networkLocation, ports, certSerialId); + if (networkName != null) { + dn.setNetworkName(networkName); + } + return dn; } - } /** @@ -437,5 +490,4 @@ public String getCertSerialId() { public void setCertSerialId(String certSerialId) { this.certSerialId = certSerialId; } - } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolClientSideTranslatorPB.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolClientSideTranslatorPB.java index 7cf9476ff20c9..efe79a76f31dd 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolClientSideTranslatorPB.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolClientSideTranslatorPB.java @@ -16,21 +16,29 @@ */ package org.apache.hadoop.hdds.protocolPB; -import com.google.protobuf.RpcController; -import com.google.protobuf.ServiceException; import java.io.Closeable; import java.io.IOException; +import java.util.function.Consumer; + +import org.apache.hadoop.hdds.protocol.SCMSecurityProtocol; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.DatanodeDetailsProto; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.OzoneManagerDetailsProto; +import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos; import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCACertificateRequestProto; +import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto; import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertificateRequestProto; -import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertificateRequestProto.Builder; import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetDataNodeCertRequestProto; -import org.apache.hadoop.hdds.protocol.SCMSecurityProtocol; +import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMSecurityRequest; +import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMSecurityRequest.Builder; +import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMSecurityResponse; +import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.Type; +import org.apache.hadoop.hdds.tracing.TracingUtil; import org.apache.hadoop.ipc.ProtobufHelper; import org.apache.hadoop.ipc.ProtocolTranslator; import org.apache.hadoop.ipc.RPC; +import com.google.protobuf.RpcController; +import com.google.protobuf.ServiceException; import static org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetOMCertRequestProto; /** @@ -51,6 +59,28 @@ public SCMSecurityProtocolClientSideTranslatorPB( this.rpcProxy = rpcProxy; } + /** + * Helper method to wrap the request and send the message. + */ + private SCMSecurityResponse submitRequest( + SCMSecurityProtocolProtos.Type type, + Consumer builderConsumer) throws IOException { + final SCMSecurityResponse response; + try { + + Builder builder = SCMSecurityRequest.newBuilder() + .setCmdType(type) + .setTraceID(TracingUtil.exportCurrentSpan()); + builderConsumer.accept(builder); + SCMSecurityRequest wrapper = builder.build(); + + response = rpcProxy.submitRequest(NULL_RPC_CONTROLLER, wrapper); + } catch (ServiceException ex) { + throw ProtobufHelper.getRemoteException(ex); + } + return response; + } + /** * Closes this stream and releases any system resources associated * with it. If the stream is already closed then invoking this @@ -79,60 +109,81 @@ public void close() throws IOException { @Override public String getDataNodeCertificate(DatanodeDetailsProto dataNodeDetails, String certSignReq) throws IOException { - SCMGetDataNodeCertRequestProto.Builder builder = - SCMGetDataNodeCertRequestProto - .newBuilder() - .setCSR(certSignReq) - .setDatanodeDetails(dataNodeDetails); - try { - return rpcProxy - .getDataNodeCertificate(NULL_RPC_CONTROLLER, builder.build()) - .getX509Certificate(); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + return getDataNodeCertificateChain(dataNodeDetails, certSignReq) + .getX509Certificate(); } /** * Get SCM signed certificate for OM. * - * @param omDetails - OzoneManager Details. - * @param certSignReq - Certificate signing request. + * @param omDetails - OzoneManager Details. + * @param certSignReq - Certificate signing request. * @return byte[] - SCM signed certificate. */ @Override public String getOMCertificate(OzoneManagerDetailsProto omDetails, String certSignReq) throws IOException { - SCMGetOMCertRequestProto.Builder builder = SCMGetOMCertRequestProto + return getOMCertChain(omDetails, certSignReq).getX509Certificate(); + } + + /** + * Get SCM signed certificate for OM. + * + * @param omDetails - OzoneManager Details. + * @param certSignReq - Certificate signing request. + * @return byte[] - SCM signed certificate. + */ + public SCMGetCertResponseProto getOMCertChain( + OzoneManagerDetailsProto omDetails, String certSignReq) + throws IOException { + SCMGetOMCertRequestProto request = SCMGetOMCertRequestProto .newBuilder() .setCSR(certSignReq) - .setOmDetails(omDetails); - try { - return rpcProxy.getOMCertificate(NULL_RPC_CONTROLLER, builder.build()) - .getX509Certificate(); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + .setOmDetails(omDetails) + .build(); + return submitRequest(Type.GetOMCertificate, + builder -> builder.setGetOMCertRequest(request)) + .getGetCertResponseProto(); } /** * Get SCM signed certificate with given serial id. Throws exception if * certificate is not found. * - * @param certSerialId - Certificate serial id. + * @param certSerialId - Certificate serial id. * @return string - pem encoded certificate. */ @Override public String getCertificate(String certSerialId) throws IOException { - Builder builder = SCMGetCertificateRequestProto + SCMGetCertificateRequestProto request = SCMGetCertificateRequestProto .newBuilder() - .setCertSerialId(certSerialId); - try { - return rpcProxy.getCertificate(NULL_RPC_CONTROLLER, builder.build()) - .getX509Certificate(); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + .setCertSerialId(certSerialId) + .build(); + return submitRequest(Type.GetCertificate, + builder -> builder.setGetCertificateRequest(request)) + .getGetCertResponseProto() + .getX509Certificate(); + } + + /** + * Get SCM signed certificate for Datanode. + * + * @param dnDetails - Datanode Details. + * @param certSignReq - Certificate signing request. + * @return byte[] - SCM signed certificate. + */ + public SCMGetCertResponseProto getDataNodeCertificateChain( + DatanodeDetailsProto dnDetails, String certSignReq) + throws IOException { + + SCMGetDataNodeCertRequestProto request = + SCMGetDataNodeCertRequestProto.newBuilder() + .setCSR(certSignReq) + .setDatanodeDetails(dnDetails) + .build(); + return submitRequest(Type.GetDataNodeCertificate, + builder -> builder.setGetDataNodeCertRequest(request)) + .getGetCertResponseProto(); } /** @@ -144,12 +195,10 @@ public String getCertificate(String certSerialId) throws IOException { public String getCACertificate() throws IOException { SCMGetCACertificateRequestProto protoIns = SCMGetCACertificateRequestProto .getDefaultInstance(); - try { - return rpcProxy.getCACertificate(NULL_RPC_CONTROLLER, protoIns) - .getX509Certificate(); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + return submitRequest(Type.GetCACertificate, + builder -> builder.setGetCACertificateRequest(protoIns)) + .getGetCertResponseProto().getX509Certificate(); + } /** diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolServerSideTranslatorPB.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolServerSideTranslatorPB.java deleted file mode 100644 index c7c4ff6b5988a..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolServerSideTranslatorPB.java +++ /dev/null @@ -1,129 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with this - * work for additional information regarding copyright ownership. The ASF - * licenses this file to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - *

    - * http://www.apache.org/licenses/LICENSE-2.0 - *

    - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - */ -package org.apache.hadoop.hdds.protocolPB; - -import com.google.protobuf.RpcController; -import com.google.protobuf.ServiceException; -import java.io.IOException; - -import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos; -import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto; -import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertificateRequestProto; -import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetDataNodeCertRequestProto; -import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto.ResponseCode; -import org.apache.hadoop.hdds.protocol.SCMSecurityProtocol; -import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetOMCertRequestProto; - -/** - * This class is the server-side translator that forwards requests received on - * {@link SCMSecurityProtocolPB} to the {@link - * SCMSecurityProtocol} server implementation. - */ -public class SCMSecurityProtocolServerSideTranslatorPB implements - SCMSecurityProtocolPB { - - private final SCMSecurityProtocol impl; - - public SCMSecurityProtocolServerSideTranslatorPB(SCMSecurityProtocol impl) { - this.impl = impl; - } - - /** - * Get SCM signed certificate for DataNode. - * - * @param controller - * @param request - * @return SCMGetDataNodeCertResponseProto. - */ - @Override - public SCMGetCertResponseProto getDataNodeCertificate( - RpcController controller, SCMGetDataNodeCertRequestProto request) - throws ServiceException { - try { - String certificate = impl - .getDataNodeCertificate(request.getDatanodeDetails(), - request.getCSR()); - SCMGetCertResponseProto.Builder builder = - SCMGetCertResponseProto - .newBuilder() - .setResponseCode(ResponseCode.success) - .setX509Certificate(certificate); - return builder.build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } - - /** - * Get SCM signed certificate for OzoneManager. - * - * @param controller - * @param request - * @return SCMGetCertResponseProto. - */ - @Override - public SCMGetCertResponseProto getOMCertificate( - RpcController controller, SCMGetOMCertRequestProto request) - throws ServiceException { - try { - String certificate = impl - .getOMCertificate(request.getOmDetails(), - request.getCSR()); - SCMGetCertResponseProto.Builder builder = - SCMGetCertResponseProto - .newBuilder() - .setResponseCode(ResponseCode.success) - .setX509Certificate(certificate); - return builder.build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } - - @Override - public SCMGetCertResponseProto getCertificate(RpcController controller, - SCMGetCertificateRequestProto request) throws ServiceException { - try { - String certificate = impl.getCertificate(request.getCertSerialId()); - SCMGetCertResponseProto.Builder builder = - SCMGetCertResponseProto - .newBuilder() - .setResponseCode(ResponseCode.success) - .setX509Certificate(certificate); - return builder.build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } - - @Override - public SCMGetCertResponseProto getCACertificate(RpcController controller, - SCMSecurityProtocolProtos.SCMGetCACertificateRequestProto request) - throws ServiceException { - try { - String certificate = impl.getCACertificate(); - SCMGetCertResponseProto.Builder builder = - SCMGetCertResponseProto - .newBuilder() - .setResponseCode(ResponseCode.success) - .setX509Certificate(certificate); - return builder.build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } -} \ No newline at end of file diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/ContainerCommandRequestMessage.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/ContainerCommandRequestMessage.java new file mode 100644 index 0000000000000..07a886a0f9c0d --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/ContainerCommandRequestMessage.java @@ -0,0 +1,107 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.ratis; + +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandRequestProto; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.PutSmallFileRequestProto; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.Type; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.WriteChunkRequestProto; +import org.apache.ratis.protocol.Message; +import org.apache.ratis.protocol.RaftGroupId; +import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; +import org.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException; +import org.apache.ratis.util.JavaUtils; + +import java.util.Objects; +import java.util.function.Supplier; + +/** + * Implementing the {@link Message} interface + * for {@link ContainerCommandRequestProto}. + */ +public final class ContainerCommandRequestMessage implements Message { + public static ContainerCommandRequestMessage toMessage( + ContainerCommandRequestProto request, String traceId) { + final ContainerCommandRequestProto.Builder b + = ContainerCommandRequestProto.newBuilder(request); + if (traceId != null) { + b.setTraceID(traceId); + } + + ByteString data = ByteString.EMPTY; + if (request.getCmdType() == Type.WriteChunk) { + final WriteChunkRequestProto w = request.getWriteChunk(); + data = w.getData(); + b.setWriteChunk(w.toBuilder().clearData()); + } else if (request.getCmdType() == Type.PutSmallFile) { + final PutSmallFileRequestProto p = request.getPutSmallFile(); + data = p.getData(); + b.setPutSmallFile(p.toBuilder().setData(ByteString.EMPTY)); + } + return new ContainerCommandRequestMessage(b.build(), data); + } + + public static ContainerCommandRequestProto toProto( + ByteString bytes, RaftGroupId groupId) + throws InvalidProtocolBufferException { + final int i = 4 + bytes.asReadOnlyByteBuffer().getInt(); + final ContainerCommandRequestProto header + = ContainerCommandRequestProto.parseFrom(bytes.substring(4, i)); + // TODO: setting pipeline id can be avoided if the client is sending it. + // In such case, just have to validate the pipeline id. + final ContainerCommandRequestProto.Builder b = header.toBuilder(); + if (groupId != null) { + b.setPipelineID(groupId.getUuid().toString()); + } + final ByteString data = bytes.substring(i); + if (header.getCmdType() == Type.WriteChunk) { + b.setWriteChunk(b.getWriteChunkBuilder().setData(data)); + } else if (header.getCmdType() == Type.PutSmallFile) { + b.setPutSmallFile(b.getPutSmallFileBuilder().setData(data)); + } + return b.build(); + } + + private final ContainerCommandRequestProto header; + private final ByteString data; + private final Supplier contentSupplier + = JavaUtils.memoize(this::buildContent); + + private ContainerCommandRequestMessage( + ContainerCommandRequestProto header, ByteString data) { + this.header = Objects.requireNonNull(header, "header == null"); + this.data = Objects.requireNonNull(data, "data == null"); + } + + private ByteString buildContent() { + final ByteString headerBytes = header.toByteString(); + return RatisHelper.int2ByteString(headerBytes.size()) + .concat(headerBytes) + .concat(data); + } + + @Override + public ByteString getContent() { + return contentSupplier.get(); + } + + @Override + public String toString() { + return header + ", data.size=" + data.size(); + } +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisHelper.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisHelper.java new file mode 100644 index 0000000000000..081b4fb766be8 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisHelper.java @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.ratis; + +import java.io.DataOutputStream; +import java.io.IOException; +import java.security.cert.CertificateException; +import java.security.cert.X509Certificate; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.List; +import java.util.UUID; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hdds.protocol.DatanodeDetails; +import org.apache.hadoop.hdds.scm.pipeline.Pipeline; +import org.apache.hadoop.hdds.security.exception.SCMSecurityException; +import org.apache.hadoop.hdds.security.x509.SecurityConfig; +import org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateServer; +import org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient; +import org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec; +import org.apache.hadoop.ozone.OzoneConfigKeys; +import org.apache.hadoop.ozone.OzoneConsts; + +import org.apache.ratis.RaftConfigKeys; +import org.apache.ratis.client.RaftClient; +import org.apache.ratis.client.RaftClientConfigKeys; +import org.apache.ratis.conf.RaftProperties; +import org.apache.ratis.grpc.GrpcConfigKeys; +import org.apache.ratis.grpc.GrpcFactory; +import org.apache.ratis.grpc.GrpcTlsConfig; +import org.apache.ratis.proto.RaftProtos; +import org.apache.ratis.protocol.RaftGroup; +import org.apache.ratis.protocol.RaftGroupId; +import org.apache.ratis.protocol.RaftPeer; +import org.apache.ratis.protocol.RaftPeerId; +import org.apache.ratis.retry.RetryPolicies; +import org.apache.ratis.retry.RetryPolicy; +import org.apache.ratis.rpc.RpcType; +import org.apache.ratis.rpc.SupportedRpcType; +import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; +import org.apache.ratis.util.SizeInBytes; +import org.apache.ratis.util.TimeDuration; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Ratis helper methods. + */ +public interface RatisHelper { + Logger LOG = LoggerFactory.getLogger(RatisHelper.class); + + static String toRaftPeerIdString(DatanodeDetails id) { + return id.getUuidString(); + } + + static UUID toDatanodeId(String peerIdString) { + return UUID.fromString(peerIdString); + } + + static UUID toDatanodeId(RaftPeerId peerId) { + return toDatanodeId(peerId.toString()); + } + + static UUID toDatanodeId(RaftProtos.RaftPeerProto peerId) { + return toDatanodeId(RaftPeerId.valueOf(peerId.getId())); + } + + static String toRaftPeerAddressString(DatanodeDetails id) { + return id.getIpAddress() + ":" + + id.getPort(DatanodeDetails.Port.Name.RATIS).getValue(); + } + + static RaftPeerId toRaftPeerId(DatanodeDetails id) { + return RaftPeerId.valueOf(toRaftPeerIdString(id)); + } + + static RaftPeer toRaftPeer(DatanodeDetails id) { + return new RaftPeer(toRaftPeerId(id), toRaftPeerAddressString(id)); + } + + static List toRaftPeers(Pipeline pipeline) { + return toRaftPeers(pipeline.getNodes()); + } + + static List toRaftPeers( + List datanodes) { + return datanodes.stream().map(RatisHelper::toRaftPeer) + .collect(Collectors.toList()); + } + + /* TODO: use a dummy id for all groups for the moment. + * It should be changed to a unique id for each group. + */ + RaftGroupId DUMMY_GROUP_ID = + RaftGroupId.valueOf(ByteString.copyFromUtf8("AOzoneRatisGroup")); + + RaftGroup EMPTY_GROUP = RaftGroup.valueOf(DUMMY_GROUP_ID, + Collections.emptyList()); + + static RaftGroup emptyRaftGroup() { + return EMPTY_GROUP; + } + + static RaftGroup newRaftGroup(Collection peers) { + return peers.isEmpty()? emptyRaftGroup() + : RaftGroup.valueOf(DUMMY_GROUP_ID, peers); + } + + static RaftGroup newRaftGroup(RaftGroupId groupId, + Collection peers) { + final List newPeers = peers.stream() + .map(RatisHelper::toRaftPeer) + .collect(Collectors.toList()); + return peers.isEmpty() ? RaftGroup.valueOf(groupId, Collections.emptyList()) + : RaftGroup.valueOf(groupId, newPeers); + } + + static RaftGroup newRaftGroup(Pipeline pipeline) { + return RaftGroup.valueOf(RaftGroupId.valueOf(pipeline.getId().getId()), + toRaftPeers(pipeline)); + } + + static RaftClient newRaftClient(RpcType rpcType, Pipeline pipeline, + RetryPolicy retryPolicy, int maxOutStandingRequest, + GrpcTlsConfig tlsConfig, TimeDuration timeout) throws IOException { + return newRaftClient(rpcType, toRaftPeerId(pipeline.getFirstNode()), + newRaftGroup(RaftGroupId.valueOf(pipeline.getId().getId()), + pipeline.getNodes()), retryPolicy, maxOutStandingRequest, tlsConfig, + timeout); + } + + static TimeDuration getClientRequestTimeout(Configuration conf) { + // Set the client requestTimeout + final TimeUnit timeUnit = + OzoneConfigKeys.DFS_RATIS_CLIENT_REQUEST_TIMEOUT_DURATION_DEFAULT + .getUnit(); + final long duration = conf.getTimeDuration( + OzoneConfigKeys.DFS_RATIS_CLIENT_REQUEST_TIMEOUT_DURATION_KEY, + OzoneConfigKeys.DFS_RATIS_CLIENT_REQUEST_TIMEOUT_DURATION_DEFAULT + .getDuration(), timeUnit); + final TimeDuration clientRequestTimeout = + TimeDuration.valueOf(duration, timeUnit); + return clientRequestTimeout; + } + + static RaftClient newRaftClient(RpcType rpcType, RaftPeer leader, + RetryPolicy retryPolicy, int maxOutstandingRequests, + GrpcTlsConfig tlsConfig, TimeDuration clientRequestTimeout) { + return newRaftClient(rpcType, leader.getId(), + newRaftGroup(new ArrayList<>(Arrays.asList(leader))), retryPolicy, + maxOutstandingRequests, tlsConfig, clientRequestTimeout); + } + + static RaftClient newRaftClient(RpcType rpcType, RaftPeer leader, + RetryPolicy retryPolicy, int maxOutstandingRequests, + TimeDuration clientRequestTimeout) { + return newRaftClient(rpcType, leader.getId(), + newRaftGroup(new ArrayList<>(Arrays.asList(leader))), retryPolicy, + maxOutstandingRequests, null, clientRequestTimeout); + } + + static RaftClient newRaftClient(RpcType rpcType, RaftPeerId leader, + RaftGroup group, RetryPolicy retryPolicy, int maxOutStandingRequest, + GrpcTlsConfig tlsConfig, TimeDuration clientRequestTimeout) { + if (LOG.isTraceEnabled()) { + LOG.trace("newRaftClient: {}, leader={}, group={}", + rpcType, leader, group); + } + final RaftProperties properties = new RaftProperties(); + RaftConfigKeys.Rpc.setType(properties, rpcType); + RaftClientConfigKeys.Rpc + .setRequestTimeout(properties, clientRequestTimeout); + + GrpcConfigKeys.setMessageSizeMax(properties, + SizeInBytes.valueOf(OzoneConsts.OZONE_SCM_CHUNK_MAX_SIZE)); + GrpcConfigKeys.OutputStream.setOutstandingAppendsMax(properties, + maxOutStandingRequest); + + RaftClient.Builder builder = RaftClient.newBuilder() + .setRaftGroup(group) + .setLeaderId(leader) + .setProperties(properties) + .setRetryPolicy(retryPolicy); + + // TODO: GRPC TLS only for now, netty/hadoop RPC TLS support later. + if (tlsConfig != null && rpcType == SupportedRpcType.GRPC) { + builder.setParameters(GrpcFactory.newRaftParameters(tlsConfig)); + } + return builder.build(); + } + + // For External gRPC client to server with gRPC TLS. + // No mTLS for external client as SCM CA does not issued certificates for them + static GrpcTlsConfig createTlsClientConfig(SecurityConfig conf, + X509Certificate caCert) { + GrpcTlsConfig tlsConfig = null; + if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) { + tlsConfig = new GrpcTlsConfig(null, null, + caCert, false); + } + return tlsConfig; + } + + // For Internal gRPC client from SCM to DN with gRPC TLS + static GrpcTlsConfig createTlsClientConfigForSCM(SecurityConfig conf, + CertificateServer certificateServer) throws IOException { + if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) { + try { + X509Certificate caCert = + CertificateCodec.getX509Certificate( + certificateServer.getCACertificate()); + return new GrpcTlsConfig(null, null, + caCert, false); + } catch (CertificateException ex) { + throw new SCMSecurityException("Fail to find SCM CA certificate.", ex); + } + } + return null; + } + + // For gRPC server running DN container service with gPRC TLS + // No mTLS as the channel is shared for for external client, which + // does not have SCM CA issued certificates. + // In summary: + // authenticate from server to client is via TLS. + // authenticate from client to server is via block token (or container token). + static GrpcTlsConfig createTlsServerConfigForDN(SecurityConfig conf, + CertificateClient caClient) { + if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) { + return new GrpcTlsConfig( + caClient.getPrivateKey(), caClient.getCertificate(), + null, false); + } + return null; + } + + static RetryPolicy createRetryPolicy(Configuration conf) { + int maxRetryCount = + conf.getInt(OzoneConfigKeys.DFS_RATIS_CLIENT_REQUEST_MAX_RETRIES_KEY, + OzoneConfigKeys. + DFS_RATIS_CLIENT_REQUEST_MAX_RETRIES_DEFAULT); + long retryInterval = conf.getTimeDuration(OzoneConfigKeys. + DFS_RATIS_CLIENT_REQUEST_RETRY_INTERVAL_KEY, OzoneConfigKeys. + DFS_RATIS_CLIENT_REQUEST_RETRY_INTERVAL_DEFAULT + .toIntExact(TimeUnit.MILLISECONDS), TimeUnit.MILLISECONDS); + TimeDuration sleepDuration = + TimeDuration.valueOf(retryInterval, TimeUnit.MILLISECONDS); + RetryPolicy retryPolicy = RetryPolicies + .retryUpToMaximumCountWithFixedSleep(maxRetryCount, sleepDuration); + return retryPolicy; + } + + static Long getMinReplicatedIndex( + Collection commitInfos) { + return commitInfos.stream().map(RaftProtos.CommitInfoProto::getCommitIndex) + .min(Long::compareTo).orElse(null); + } + + static ByteString int2ByteString(int n) { + final ByteString.Output out = ByteString.newOutput(); + try(DataOutputStream dataOut = new DataOutputStream(out)) { + dataOut.writeInt(n); + } catch (IOException e) { + throw new IllegalStateException( + "Failed to write integer n = " + n + " to a ByteString.", e); + } + return out.toByteString(); + } +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/package-info.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/package-info.java new file mode 100644 index 0000000000000..e52dc7ffc70bb --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/package-info.java @@ -0,0 +1,22 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.ratis; + +/** + * This package contains classes related to Apache Ratis. + */ diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ByteStringConversion.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ByteStringConversion.java new file mode 100644 index 0000000000000..4608df7612287 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ByteStringConversion.java @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.scm; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.ozone.OzoneConfigKeys; +import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; +import org.apache.ratis.thirdparty.com.google.protobuf.UnsafeByteOperations; + +import java.nio.ByteBuffer; +import java.util.function.Function; + +/** + * Helper class to create a conversion function from ByteBuffer to ByteString + * based on the property + * {@link OzoneConfigKeys#OZONE_UNSAFEBYTEOPERATIONS_ENABLED} in the + * Ozone configuration. + */ +public final class ByteStringConversion { + private ByteStringConversion(){} // no instantiation. + + /** + * Creates the conversion function to be used to convert ByteBuffers to + * ByteString instances to be used in protobuf messages. + * + * @param config the Ozone configuration + * @return the conversion function defined by + * {@link OzoneConfigKeys#OZONE_UNSAFEBYTEOPERATIONS_ENABLED} + * @see 

    ByteBuffer
    + */ + public static Function createByteBufferConversion( + Configuration config){ + boolean unsafeEnabled = + config!=null && config.getBoolean( + OzoneConfigKeys.OZONE_UNSAFEBYTEOPERATIONS_ENABLED, + OzoneConfigKeys.OZONE_UNSAFEBYTEOPERATIONS_ENABLED_DEFAULT); + if (unsafeEnabled) { + return buffer -> UnsafeByteOperations.unsafeWrap(buffer); + } else { + return buffer -> { + ByteString retval = ByteString.copyFrom(buffer); + buffer.flip(); + return retval; + }; + } + } +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ByteStringHelper.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ByteStringHelper.java deleted file mode 100644 index ccdf4fac42498..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ByteStringHelper.java +++ /dev/null @@ -1,69 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.hdds.scm; - -import com.google.common.base.Preconditions; -import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; -import org.apache.ratis.thirdparty.com.google.protobuf.UnsafeByteOperations; - -import java.nio.ByteBuffer; -import java.util.concurrent.atomic.AtomicBoolean; -/** - * Helper class to perform Unsafe ByteString conversion from byteBuffer or byte - * array depending on the config "ozone.UnsafeByteOperations.enabled". - */ -public final class ByteStringHelper { - private static final AtomicBoolean INITIALIZED = new AtomicBoolean(); - private static volatile boolean isUnsafeByteOperationsEnabled; - - /** - * There is no need to instantiate this class. - */ - private ByteStringHelper() { - } - - public static void init(boolean isUnsafeByteOperation) { - final boolean set = INITIALIZED.compareAndSet(false, true); - if (set) { - ByteStringHelper.isUnsafeByteOperationsEnabled = - isUnsafeByteOperation; - } else { - // already initialized, check values - Preconditions.checkState(isUnsafeByteOperationsEnabled - == isUnsafeByteOperation); - } - } - - private static ByteString copyFrom(ByteBuffer buffer) { - final ByteString bytes = ByteString.copyFrom(buffer); - // flip the buffer so as to read the data starting from pos 0 again - buffer.flip(); - return bytes; - } - - public static ByteString getByteString(ByteBuffer buffer) { - return isUnsafeByteOperationsEnabled ? - UnsafeByteOperations.unsafeWrap(buffer) : copyFrom(buffer); - } - - public static ByteString getByteString(byte[] bytes) { - return isUnsafeByteOperationsEnabled ? - UnsafeByteOperations.unsafeWrap(bytes) : ByteString.copyFrom(bytes); - } - -} \ No newline at end of file diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java index 4a423588f5f28..161780668ab0c 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java @@ -36,21 +36,6 @@ public final class ScmConfigKeys { // performance. public static final String OZONE_SCM_DB_DIRS = "ozone.scm.db.dirs"; - public static final String SCM_CONTAINER_CLIENT_STALE_THRESHOLD_KEY = - "scm.container.client.idle.threshold"; - public static final String SCM_CONTAINER_CLIENT_STALE_THRESHOLD_DEFAULT = - "10s"; - - public static final String SCM_CONTAINER_CLIENT_MAX_SIZE_KEY = - "scm.container.client.max.size"; - public static final int SCM_CONTAINER_CLIENT_MAX_SIZE_DEFAULT = - 256; - - public static final String SCM_CONTAINER_CLIENT_MAX_OUTSTANDING_REQUESTS = - "scm.container.client.max.outstanding.requests"; - public static final int SCM_CONTAINER_CLIENT_MAX_OUTSTANDING_REQUESTS_DEFAULT - = 100; - public static final String DFS_CONTAINER_RATIS_ENABLED_KEY = "dfs.container.ratis.enabled"; public static final boolean DFS_CONTAINER_RATIS_ENABLED_DEFAULT @@ -74,7 +59,7 @@ public final class ScmConfigKeys { public static final String DFS_CONTAINER_RATIS_SEGMENT_SIZE_KEY = "dfs.container.ratis.segment.size"; public static final String DFS_CONTAINER_RATIS_SEGMENT_SIZE_DEFAULT = - "16KB"; + "1MB"; public static final String DFS_CONTAINER_RATIS_SEGMENT_PREALLOCATED_SIZE_KEY = "dfs.container.ratis.segment.preallocated.size"; public static final String @@ -90,6 +75,14 @@ public final class ScmConfigKeys { "dfs.container.ratis.statemachinedata.sync.retries"; public static final int DFS_CONTAINER_RATIS_STATEMACHINEDATA_SYNC_RETRIES_DEFAULT = -1; + public static final String + DFS_CONTAINER_RATIS_STATEMACHINE_MAX_PENDING_APPLY_TXNS = + "dfs.container.ratis.statemachine.max.pending.apply-transactions"; + // The default value of maximum number of pending state machine apply + // transactions is kept same as default snapshot threshold. + public static final int + DFS_CONTAINER_RATIS_STATEMACHINE_MAX_PENDING_APPLY_TXNS_DEFAULT = + 100000; public static final String DFS_CONTAINER_RATIS_LOG_QUEUE_NUM_ELEMENTS = "dfs.container.ratis.log.queue.num-elements"; public static final int DFS_CONTAINER_RATIS_LOG_QUEUE_NUM_ELEMENTS_DEFAULT = @@ -107,6 +100,16 @@ public final class ScmConfigKeys { "dfs.container.ratis.log.appender.queue.byte-limit"; public static final String DFS_CONTAINER_RATIS_LOG_APPENDER_QUEUE_BYTE_LIMIT_DEFAULT = "32MB"; + public static final String DFS_CONTAINER_RATIS_LOG_PURGE_GAP = + "dfs.container.ratis.log.purge.gap"; + // TODO: Set to 1024 once RATIS issue around purge is fixed. + public static final int DFS_CONTAINER_RATIS_LOG_PURGE_GAP_DEFAULT = + 1000000; + + public static final String DFS_CONTAINER_RATIS_LEADER_NUM_PENDING_REQUESTS = + "dfs.container.ratis.leader.num.pending.requests"; + public static final int + DFS_CONTAINER_RATIS_LEADER_NUM_PENDING_REQUESTS_DEFAULT = 4096; // expiry interval stateMachineData cache entry inside containerStateMachine public static final String DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL = @@ -142,11 +145,11 @@ public final class ScmConfigKeys { "dfs.ratis.leader.election.minimum.timeout.duration"; public static final TimeDuration DFS_RATIS_LEADER_ELECTION_MINIMUM_TIMEOUT_DURATION_DEFAULT = - TimeDuration.valueOf(1, TimeUnit.SECONDS); + TimeDuration.valueOf(5, TimeUnit.SECONDS); public static final String DFS_RATIS_SNAPSHOT_THRESHOLD_KEY = "dfs.ratis.snapshot.threshold"; - public static final long DFS_RATIS_SNAPSHOT_THRESHOLD_DEFAULT = 10000; + public static final long DFS_RATIS_SNAPSHOT_THRESHOLD_DEFAULT = 100000; public static final String DFS_RATIS_SERVER_FAILURE_DURATION_KEY = "dfs.ratis.server.failure.duration"; @@ -352,7 +355,7 @@ public final class ScmConfigKeys { "hdds.scm.http.kerberos.principal"; public static final String HDDS_SCM_HTTP_KERBEROS_KEYTAB_FILE_KEY = - "hdds.scm.http.kerberos.keytab.file"; + "hdds.scm.http.kerberos.keytab"; // Network topology public static final String OZONE_SCM_NETWORK_TOPOLOGY_SCHEMA_FILE = diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientSpi.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientSpi.java index 1a183664a127f..5631badf44c93 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientSpi.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientSpi.java @@ -25,13 +25,13 @@ import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicInteger; -import org.apache.hadoop.hdds.protocol.DatanodeDetails; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandRequestProto; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandResponseProto; import org.apache.hadoop.hdds.protocol.proto.HddsProtos; import org.apache.hadoop.hdds.scm.pipeline.Pipeline; import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.hdds.scm.storage.CheckedBiFunction; /** * A Client for the storageContainer protocol. @@ -118,18 +118,21 @@ public ContainerCommandResponseProto sendCommand( * Sends a given command to server and gets the reply back along with * the server associated info. * @param request Request - * @param excludeDns list of servers on which the command won't be sent to. + * @param validators functions to validate the response * @return Response to the command * @throws IOException */ - public XceiverClientReply sendCommand( - ContainerCommandRequestProto request, List excludeDns) + public ContainerCommandResponseProto sendCommand( + ContainerCommandRequestProto request, List validators) throws IOException { try { XceiverClientReply reply; reply = sendCommandAsync(request); - reply.getResponse().get(); - return reply; + ContainerCommandResponseProto responseProto = reply.getResponse().get(); + for (CheckedBiFunction function : validators) { + function.apply(request, responseProto); + } + return responseProto; } catch (ExecutionException | InterruptedException e) { throw new IOException("Failed to command " + request, e); } @@ -156,7 +159,7 @@ public XceiverClientReply sendCommand( /** * Check if an specfic commitIndex is replicated to majority/all servers. * @param index index to watch for - * @param timeout timeout provided for the watch ipeartion to complete + * @param timeout timeout provided for the watch operation to complete * @return reply containing the min commit index replicated to all or majority * servers in case of a failure * @throws InterruptedException diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java index 85821ac92bc5d..226ceda9255ad 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java @@ -180,6 +180,22 @@ Pipeline createReplicationPipeline(HddsProtos.ReplicationType type, */ List listPipelines() throws IOException; + /** + * Activates the pipeline given a pipeline ID. + * + * @param pipelineID PipelineID to activate. + * @throws IOException In case of exception while activating the pipeline + */ + void activatePipeline(HddsProtos.PipelineID pipelineID) throws IOException; + + /** + * Deactivates the pipeline given a pipeline ID. + * + * @param pipelineID PipelineID to deactivate. + * @throws IOException In case of exception while deactivating the pipeline + */ + void deactivatePipeline(HddsProtos.PipelineID pipelineID) throws IOException; + /** * Closes the pipeline given a pipeline ID. * @@ -203,4 +219,23 @@ Pipeline createReplicationPipeline(HddsProtos.ReplicationType type, * @throws IOException */ boolean forceExitSafeMode() throws IOException; + + /** + * Start ReplicationManager. + */ + void startReplicationManager() throws IOException; + + /** + * Stop ReplicationManager. + */ + void stopReplicationManager() throws IOException; + + /** + * Returns ReplicationManager status. + * + * @return True if ReplicationManager is running, false otherwise. + */ + boolean getReplicationManagerStatus() throws IOException; + + } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerInfo.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerInfo.java index 7b5c467e67994..5c58e92d3c5d9 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerInfo.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerInfo.java @@ -54,7 +54,7 @@ public class ContainerInfo implements Comparator, mapper.setVisibility(PropertyAccessor.FIELD, JsonAutoDetect.Visibility.ANY); mapper .setVisibility(PropertyAccessor.GETTER, JsonAutoDetect.Visibility.NONE); - WRITER = mapper.writer(); + WRITER = mapper.writerWithDefaultPrettyPrinter(); } private HddsProtos.LifeCycleState state; @@ -467,4 +467,5 @@ public boolean isOpen() { return state == HddsProtos.LifeCycleState.OPEN || state == HddsProtos.LifeCycleState.CLOSING; } + } diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/ContainerPlacementPolicy.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/ContainerPlacementPolicy.java similarity index 92% rename from hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/ContainerPlacementPolicy.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/ContainerPlacementPolicy.java index 3336c8e80e74e..52ce7964b6769 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/ContainerPlacementPolicy.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/ContainerPlacementPolicy.java @@ -33,12 +33,13 @@ public interface ContainerPlacementPolicy { * that satisfy the nodes and size requirement. * * @param excludedNodes - list of nodes to be excluded. + * @param favoredNodes - list of nodes preferred. * @param nodesRequired - number of datanodes required. * @param sizeRequired - size required for the container or block. * @return list of datanodes chosen. * @throws IOException */ List chooseDatanodes(List excludedNodes, - int nodesRequired, long sizeRequired) + List favoredNodes, int nodesRequired, long sizeRequired) throws IOException; } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/package-info.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/package-info.java new file mode 100644 index 0000000000000..dac4752fe66fa --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/package-info.java @@ -0,0 +1,21 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.scm.container.placement.algorithms; +/** + Contains container placement policy interface definition. + **/ \ No newline at end of file diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/exceptions/SCMException.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions/SCMException.java similarity index 96% rename from hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/exceptions/SCMException.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions/SCMException.java index 01166ad5a765b..db1f82ae411d0 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/exceptions/SCMException.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions/SCMException.java @@ -95,7 +95,7 @@ public ResultCodes getResult() { * Error codes to make it easy to decode these exceptions. */ public enum ResultCodes { - SUCCEESS, + OK, FAILED_TO_LOAD_NODEPOOL, FAILED_TO_FIND_NODE_IN_POOL, FAILED_TO_FIND_HEALTHY_NODES, @@ -119,6 +119,9 @@ public enum ResultCodes { DUPLICATE_DATANODE, NO_SUCH_DATANODE, NO_REPLICA_FOUND, - FAILED_TO_FIND_ACTIVE_PIPELINE + FAILED_TO_FIND_ACTIVE_PIPELINE, + FAILED_TO_INIT_CONTAINER_PLACEMENT_POLICY, + FAILED_TO_ALLOCATE_ENOUGH_BLOCKS, + INTERNAL_ERROR } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions/package-info.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions/package-info.java new file mode 100644 index 0000000000000..721a32b48e219 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions/package-info.java @@ -0,0 +1,21 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.scm.exceptions; +/** + Exception objects for the SCM Server. + */ diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/InnerNode.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/InnerNode.java index a185b01f57403..6cf73bf54800d 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/InnerNode.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/InnerNode.java @@ -18,6 +18,7 @@ package org.apache.hadoop.hdds.scm.net; import java.util.Collection; +import java.util.List; /** * The interface defines an inner node in a network topology. @@ -72,13 +73,13 @@ N newInnerNode(String name, String location, InnerNode parent, int level, * * @param leafIndex ode's index, start from 0, skip the nodes in * excludedScope and excludedNodes with ancestorGen - * @param excludedScope the excluded scope + * @param excludedScopes the excluded scopes * @param excludedNodes nodes to be excluded. If ancestorGen is not 0, * the chosen node will not share same ancestor with * those in excluded nodes at the specified generation * @param ancestorGen ignored with value is 0 * @return the leaf node corresponding to the given index */ - Node getLeaf(int leafIndex, String excludedScope, + Node getLeaf(int leafIndex, List excludedScopes, Collection excludedNodes, int ancestorGen); } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/InnerNodeImpl.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/InnerNodeImpl.java index 3f1351d63e389..f2183fc9823fe 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/InnerNodeImpl.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/InnerNodeImpl.java @@ -22,6 +22,7 @@ import java.util.HashMap; import java.util.Iterator; import java.util.LinkedHashMap; +import java.util.List; import java.util.Map; import com.google.common.base.Preconditions; @@ -276,7 +277,7 @@ public Node getLeaf(int leafIndex) { * * @param leafIndex node's index, start from 0, skip the nodes in * excludedScope and excludedNodes with ancestorGen - * @param excludedScope the exclude scope + * @param excludedScopes the exclude scopes * @param excludedNodes nodes to be excluded from. If ancestorGen is not 0, * the chosen node will not share same ancestor with * those in excluded nodes at the specified generation @@ -300,7 +301,7 @@ public Node getLeaf(int leafIndex) { * * Input: * leafIndex = 2 - * excludedScope = /dc2 + * excludedScope = /dc2/rack2 * excludedNodes = {/dc1/rack1/n1} * ancestorGen = 1 * @@ -313,12 +314,12 @@ public Node getLeaf(int leafIndex) { * means picking the 3th available node, which is n5. * */ - public Node getLeaf(int leafIndex, String excludedScope, + public Node getLeaf(int leafIndex, List excludedScopes, Collection excludedNodes, int ancestorGen) { Preconditions.checkArgument(leafIndex >= 0 && ancestorGen >= 0); // come to leaf parent layer if (isLeafParent()) { - return getLeafOnLeafParent(leafIndex, excludedScope, excludedNodes); + return getLeafOnLeafParent(leafIndex, excludedScopes, excludedNodes); } int maxLevel = NodeSchemaManager.getInstance().getMaxLevel(); @@ -328,14 +329,16 @@ public Node getLeaf(int leafIndex, String excludedScope, Map countMap = getAncestorCountMap(excludedNodes, ancestorGen, currentGen); // nodes covered by excluded scope - int excludedNodeCount = getExcludedScopeNodeCount(excludedScope); + Map excludedNodeCount = + getExcludedScopeNodeCount(excludedScopes); - for(Node child : childrenMap.values()) { + for (Node child : childrenMap.values()) { int leafCount = child.getNumOfLeaves(); - // skip nodes covered by excluded scope - if (excludedScope != null && - excludedScope.startsWith(child.getNetworkFullPath())) { - leafCount -= excludedNodeCount; + // skip nodes covered by excluded scopes + for (Map.Entry entry: excludedNodeCount.entrySet()) { + if (entry.getKey().startsWith(child.getNetworkFullPath())) { + leafCount -= entry.getValue(); + } } // skip nodes covered by excluded nodes and ancestorGen Integer count = countMap.get(child); @@ -343,7 +346,7 @@ public Node getLeaf(int leafIndex, String excludedScope, leafCount -= count; } if (leafIndex < leafCount) { - return ((InnerNode)child).getLeaf(leafIndex, excludedScope, + return ((InnerNode)child).getLeaf(leafIndex, excludedScopes, excludedNodes, ancestorGen); } else { leafIndex -= leafCount; @@ -424,18 +427,22 @@ private Map getAncestorCountMap(Collection nodes, * Get the node with leafIndex, considering skip nodes in excludedScope * and in excludeNodes list. */ - private Node getLeafOnLeafParent(int leafIndex, String excludedScope, + private Node getLeafOnLeafParent(int leafIndex, List excludedScopes, Collection excludedNodes) { Preconditions.checkArgument(isLeafParent() && leafIndex >= 0); if (leafIndex >= getNumOfChildren()) { return null; } for(Node node : childrenMap.values()) { - if ((excludedNodes != null && (excludedNodes.contains(node))) || - (excludedScope != null && - (node.getNetworkFullPath().startsWith(excludedScope)))) { + if (excludedNodes != null && excludedNodes.contains(node)) { continue; } + if (excludedScopes != null && excludedScopes.size() > 0) { + if (excludedScopes.stream().anyMatch(scope -> + node.getNetworkFullPath().startsWith(scope))) { + continue; + } + } if (leafIndex == 0) { return node; } @@ -484,12 +491,19 @@ private Node getChildNode(int index) { return node; } - /** Get how many leaf nodes are covered by the excludedScope. */ - private int getExcludedScopeNodeCount(String excludedScope) { - if (excludedScope == null) { - return 0; + /** Get how many leaf nodes are covered by the excludedScopes(no overlap). */ + private Map getExcludedScopeNodeCount( + List excludedScopes) { + HashMap nodeCounts = new HashMap<>(); + if (excludedScopes == null || excludedScopes.isEmpty()) { + return nodeCounts; + } + + for (String scope: excludedScopes) { + Node excludedScopeNode = getNode(scope); + nodeCounts.put(scope, excludedScopeNode == null ? 0 : + excludedScopeNode.getNumOfLeaves()); } - Node excludedScopeNode = getNode(excludedScope); - return excludedScopeNode == null ? 0 : excludedScopeNode.getNumOfLeaves(); + return nodeCounts; } } \ No newline at end of file diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetUtils.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetUtils.java index 501a9ea3e52af..4019b1305f6a8 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetUtils.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetUtils.java @@ -17,6 +17,7 @@ */ package org.apache.hadoop.hdds.scm.net; +import org.apache.commons.collections.CollectionUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -24,6 +25,7 @@ import java.util.Collection; import java.util.Iterator; import java.util.List; +import java.util.stream.Collectors; /** * Utility class to facilitate network topology functions. @@ -71,18 +73,17 @@ public static int locationToDepth(String location) { * Remove node from mutableExcludedNodes if it's covered by excludedScope. * Please noted that mutableExcludedNodes content might be changed after the * function call. - * @return the new excludedScope */ - public static String removeDuplicate(NetworkTopology topology, - Collection mutableExcludedNodes, String excludedScope, + public static void removeDuplicate(NetworkTopology topology, + Collection mutableExcludedNodes, List mutableExcludedScopes, int ancestorGen) { - if (mutableExcludedNodes == null || mutableExcludedNodes.size() == 0 || - excludedScope == null || topology == null) { - return excludedScope; + if (CollectionUtils.isEmpty(mutableExcludedNodes) || + CollectionUtils.isEmpty(mutableExcludedScopes) || topology == null) { + return; } Iterator iterator = mutableExcludedNodes.iterator(); - while (iterator.hasNext()) { + while (iterator.hasNext() && (!mutableExcludedScopes.isEmpty())) { Node node = iterator.next(); Node ancestor = topology.getAncestor(node, ancestorGen); if (ancestor == null) { @@ -90,16 +91,20 @@ public static String removeDuplicate(NetworkTopology topology, " of node :" + node); continue; } - if (excludedScope.startsWith(ancestor.getNetworkFullPath())) { - // reset excludedScope if it's covered by exclude node's ancestor - return null; - } - if (ancestor.getNetworkFullPath().startsWith(excludedScope)) { - // remove exclude node if it's covered by excludedScope - iterator.remove(); - } + // excludedScope is child of ancestor + List duplicateList = mutableExcludedScopes.stream() + .filter(scope -> scope.startsWith(ancestor.getNetworkFullPath())) + .collect(Collectors.toList()); + mutableExcludedScopes.removeAll(duplicateList); + + // ancestor is covered by excludedScope + mutableExcludedScopes.stream().forEach(scope -> { + if (ancestor.getNetworkFullPath().startsWith(scope)) { + // remove exclude node if it's covered by excludedScope + iterator.remove(); + } + }); } - return excludedScope; } /** @@ -109,7 +114,7 @@ public static String removeDuplicate(NetworkTopology topology, */ public static void removeOutscope(Collection mutableExcludedNodes, String scope) { - if (mutableExcludedNodes == null || scope == null) { + if (CollectionUtils.isEmpty(mutableExcludedNodes) || scope == null) { return; } synchronized (mutableExcludedNodes) { @@ -134,7 +139,7 @@ public static void removeOutscope(Collection mutableExcludedNodes, public static List getAncestorList(NetworkTopology topology, Collection nodes, int generation) { List ancestorList = new ArrayList<>(); - if (topology == null ||nodes == null || nodes.size() == 0 || + if (topology == null || CollectionUtils.isEmpty(nodes) || generation == 0) { return ancestorList; } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetworkTopology.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetworkTopology.java index a3d3680e475e0..3a2c7c0f1a5ce 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetworkTopology.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetworkTopology.java @@ -18,6 +18,7 @@ package org.apache.hadoop.hdds.scm.net; import java.util.Collection; +import java.util.List; /** * The interface defines a network topology. @@ -38,7 +39,6 @@ public InvalidTopologyException(String msg) { */ void add(Node node); - /** * Remove a node from the network topology. This will be called when a * existing datanode is removed from the system. @@ -46,7 +46,6 @@ public InvalidTopologyException(String msg) { */ void remove(Node node); - /** * Check if the tree already contains node node. * @param node a node @@ -67,7 +66,6 @@ public InvalidTopologyException(String msg) { */ boolean isSameAncestor(Node node1, Node node2, int ancestorGen); - /** * Get the ancestor for node on generation ancestorGen. * @@ -118,11 +116,11 @@ public InvalidTopologyException(String msg) { * Randomly choose a node in the scope, ano not in the exclude scope. * @param scope range of nodes from which a node will be chosen. cannot start * with ~ - * @param excludedScope the chosen node cannot be in this range. cannot + * @param excludedScopes the chosen nodes cannot be in these ranges. cannot * starts with ~ * @return the chosen node */ - Node chooseRandom(String scope, String excludedScope); + Node chooseRandom(String scope, List excludedScopes); /** * Randomly choose a leaf node from scope. @@ -159,26 +157,6 @@ public InvalidTopologyException(String msg) { Node chooseRandom(String scope, Collection excludedNodes, int ancestorGen); - - /** - * Randomly choose a leaf node. - * - * @param scope range from which a node will be chosen, cannot start with ~ - * @param excludedNodes nodes to be excluded - * @param excludedScope excluded node range. Cannot start with ~ - * @param ancestorGen matters when excludeNodes is not null. It means the - * ancestor generation that's not allowed to share between chosen node and the - * excludedNodes. For example, if ancestorGen is 1, means chosen node - * cannot share the same parent with excludeNodes. If value is 2, cannot - * share the same grand parent, and so on. If ancestorGen is 0, then no - * effect. - * - * @return the chosen node - */ - Node chooseRandom(String scope, String excludedScope, - Collection excludedNodes, int ancestorGen); - - /** * Randomly choose one node from scope, share the same generation * ancestor with affinityNode, and exclude nodes in @@ -186,7 +164,7 @@ Node chooseRandom(String scope, String excludedScope, * * @param scope range of nodes from which a node will be chosen, cannot start * with ~ - * @param excludedScope range of nodes to be excluded, cannot start with ~ + * @param excludedScopes ranges of nodes to be excluded, cannot start with ~ * @param excludedNodes nodes to be excluded * @param affinityNode when not null, the chosen node should share the same * ancestor with this node at generation ancestorGen. @@ -197,7 +175,7 @@ Node chooseRandom(String scope, String excludedScope, * excludedNodes if affinityNode is null * @return the chosen node */ - Node chooseRandom(String scope, String excludedScope, + Node chooseRandom(String scope, List excludedScopes, Collection excludedNodes, Node affinityNode, int ancestorGen); /** @@ -209,7 +187,7 @@ Node chooseRandom(String scope, String excludedScope, * excludedNodes * @param scope range of nodes from which a node will be chosen, cannot start * with ~ - * @param excludedScope range of nodes to be excluded, cannot start with ~ + * @param excludedScopes ranges of nodes to be excluded, cannot start with ~ * @param excludedNodes nodes to be excluded * @param affinityNode when not null, the chosen node should share the same * ancestor with this node at generation ancestorGen. @@ -220,7 +198,7 @@ Node chooseRandom(String scope, String excludedScope, * excludedNodes if affinityNode is null * @return the chosen node */ - Node getNode(int leafIndex, String scope, String excludedScope, + Node getNode(int leafIndex, String scope, List excludedScopes, Collection excludedNodes, Node affinityNode, int ancestorGen); /** Return the distance cost between two nodes @@ -246,5 +224,6 @@ Node getNode(int leafIndex, String scope, String excludedScope, * @param nodes Available replicas with the requested data * @param activeLen Number of active nodes at the front of the array */ - void sortByDistanceCost(Node reader, Node[] nodes, int activeLen); + List sortByDistanceCost(Node reader, + List nodes, int activeLen); } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetworkTopologyImpl.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetworkTopologyImpl.java index d0b295f717875..579e5f71c7913 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetworkTopologyImpl.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetworkTopologyImpl.java @@ -20,9 +20,11 @@ import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; import com.google.common.collect.Lists; +import org.apache.commons.collections.CollectionUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.util.ArrayList; import java.util.Collection; import java.util.Collections; import java.util.List; @@ -107,7 +109,9 @@ public void add(Node node) { if (add) { LOG.info("Added a new node: " + node.getNetworkFullPath()); - LOG.debug("NetworkTopology became:\n{}", this); + if (LOG.isDebugEnabled()) { + LOG.debug("NetworkTopology became:\n{}", this); + } } } @@ -129,7 +133,9 @@ public void remove(Node node) { netlock.writeLock().unlock(); } LOG.info("Removed a node: " + node.getNetworkFullPath()); - LOG.debug("NetworkTopology became:\n{}", this); + if (LOG.isDebugEnabled()) { + LOG.debug("NetworkTopology became:\n{}", this); + } } /** @@ -282,7 +288,9 @@ public Node chooseRandom(String scope) { scope = ROOT; } if (scope.startsWith(SCOPE_REVERSE_STR)) { - return chooseRandom(ROOT, scope.substring(1), null, null, + ArrayList excludedScopes = new ArrayList(); + excludedScopes.add(scope.substring(1)); + return chooseRandom(ROOT, excludedScopes, null, null, ANCESTOR_GENERATION_DEFAULT); } else { return chooseRandom(scope, null, null, null, ANCESTOR_GENERATION_DEFAULT); @@ -293,12 +301,12 @@ public Node chooseRandom(String scope) { * Randomly choose a node in the scope, ano not in the exclude scope. * @param scope range of nodes from which a node will be chosen. cannot start * with ~ - * @param excludedScope the chosen node cannot be in this range. cannot + * @param excludedScopes the chosen node cannot be in these ranges. cannot * starts with ~ * @return the chosen node */ - public Node chooseRandom(String scope, String excludedScope) { - return chooseRandom(scope, excludedScope, null, null, + public Node chooseRandom(String scope, List excludedScopes) { + return chooseRandom(scope, excludedScopes, null, null, ANCESTOR_GENERATION_DEFAULT); } @@ -319,7 +327,9 @@ public Node chooseRandom(String scope, Collection excludedNodes) { scope = ROOT; } if (scope.startsWith(SCOPE_REVERSE_STR)) { - return chooseRandom(ROOT, scope.substring(1), excludedNodes, null, + ArrayList excludedScopes = new ArrayList(); + excludedScopes.add(scope.substring(1)); + return chooseRandom(ROOT, excludedScopes, excludedNodes, null, ANCESTOR_GENERATION_DEFAULT); } else { return chooseRandom(scope, null, excludedNodes, null, @@ -351,33 +361,15 @@ public Node chooseRandom(String scope, Collection excludedNodes, scope = ROOT; } if (scope.startsWith(SCOPE_REVERSE_STR)) { - return chooseRandom(ROOT, scope.substring(1), excludedNodes, null, + ArrayList excludedScopes = new ArrayList(); + excludedScopes.add(scope.substring(1)); + return chooseRandom(ROOT, excludedScopes, excludedNodes, null, ancestorGen); } else { return chooseRandom(scope, null, excludedNodes, null, ancestorGen); } } - /** - * Randomly choose a leaf node. - * - * @param scope range from which a node will be chosen, cannot start with ~ - * @param excludedNodes nodes to be excluded - * @param excludedScope excluded node range. Cannot start with ~ - * @param ancestorGen matters when excludeNodes is not null. It means the - * ancestor generation that's not allowed to share between chosen node and the - * excludedNodes. For example, if ancestorGen is 1, means chosen node - * cannot share the same parent with excludeNodes. If value is 2, cannot - * share the same grand parent, and so on. If ancestorGen is 0, then no - * effect. - * - * @return the chosen node - */ - public Node chooseRandom(String scope, String excludedScope, - Collection excludedNodes, int ancestorGen) { - return chooseRandom(scope, excludedScope, excludedNodes, null, ancestorGen); - } - /** * Randomly choose one leaf node from scope, share the same generation * ancestor with affinityNode, and exclude nodes in @@ -385,7 +377,7 @@ public Node chooseRandom(String scope, String excludedScope, * * @param scope range of nodes from which a node will be chosen, cannot start * with ~ - * @param excludedScope range of nodes to be excluded, cannot start with ~ + * @param excludedScopes ranges of nodes to be excluded, cannot start with ~ * @param excludedNodes nodes to be excluded * @param affinityNode when not null, the chosen node should share the same * ancestor with this node at generation ancestorGen. @@ -396,20 +388,20 @@ public Node chooseRandom(String scope, String excludedScope, * excludedNodes if affinityNode is null * @return the chosen node */ - public Node chooseRandom(String scope, String excludedScope, + public Node chooseRandom(String scope, List excludedScopes, Collection excludedNodes, Node affinityNode, int ancestorGen) { if (scope == null) { scope = ROOT; } checkScope(scope); - checkExcludedScope(excludedScope); + checkExcludedScopes(excludedScopes); checkAffinityNode(affinityNode); checkAncestorGen(ancestorGen); netlock.readLock().lock(); try { - return chooseNodeInternal(scope, -1, excludedScope, + return chooseNodeInternal(scope, -1, excludedScopes, excludedNodes, affinityNode, ancestorGen); } finally { netlock.readLock().unlock(); @@ -425,7 +417,7 @@ public Node chooseRandom(String scope, String excludedScope, * excludedNodes * @param scope range of nodes from which a node will be chosen, cannot start * with ~ - * @param excludedScope range of nodes to be excluded, cannot start with ~ + * @param excludedScopes ranges of nodes to be excluded, cannot start with ~ * @param excludedNodes nodes to be excluded * @param affinityNode when not null, the chosen node should share the same * ancestor with this node at generation ancestorGen. @@ -465,20 +457,20 @@ public Node chooseRandom(String scope, String excludedScope, * from subtree /dc1. LeafIndex 1, so we pick the 2nd available node n4. * */ - public Node getNode(int leafIndex, String scope, String excludedScope, + public Node getNode(int leafIndex, String scope, List excludedScopes, Collection excludedNodes, Node affinityNode, int ancestorGen) { Preconditions.checkArgument(leafIndex >= 0); if (scope == null) { scope = ROOT; } checkScope(scope); - checkExcludedScope(excludedScope); + checkExcludedScopes(excludedScopes); checkAffinityNode(affinityNode); checkAncestorGen(ancestorGen); netlock.readLock().lock(); try { - return chooseNodeInternal(scope, leafIndex, excludedScope, + return chooseNodeInternal(scope, leafIndex, excludedScopes, excludedNodes, affinityNode, ancestorGen); } finally { netlock.readLock().unlock(); @@ -486,8 +478,8 @@ public Node getNode(int leafIndex, String scope, String excludedScope, } private Node chooseNodeInternal(String scope, int leafIndex, - String excludedScope, Collection excludedNodes, Node affinityNode, - int ancestorGen) { + List excludedScopes, Collection excludedNodes, + Node affinityNode, int ancestorGen) { Preconditions.checkArgument(scope != null); String finalScope = scope; @@ -508,47 +500,51 @@ private Node chooseNodeInternal(String scope, int leafIndex, ancestorGen = 0; } - // check overlap of excludedScope and finalScope - if (excludedScope != null) { - // excludeScope covers finalScope - if (finalScope.startsWith(excludedScope)) { - return null; - } - // excludeScope and finalScope share nothing - if (!excludedScope.startsWith(finalScope)) { - excludedScope = null; + // check overlap of excludedScopes and finalScope + List mutableExcludedScopes = null; + if (excludedScopes != null && !excludedScopes.isEmpty()) { + mutableExcludedScopes = new ArrayList<>(); + for (String s: excludedScopes) { + // excludeScope covers finalScope + if (finalScope.startsWith(s)) { + return null; + } + // excludeScope and finalScope share nothing case + if (s.startsWith(finalScope)) { + if (!mutableExcludedScopes.stream().anyMatch( + e -> s.startsWith(e))) { + mutableExcludedScopes.add(s); + } + } } } // clone excludedNodes before remove duplicate in it Collection mutableExNodes = null; + + // Remove duplicate in excludedNodes if (excludedNodes != null) { - // Remove duplicate in excludedNodes mutableExNodes = excludedNodes.stream().distinct().collect(Collectors.toList()); } - // remove duplicate in mutableExNodes and excludedScope, given ancestorGen - excludedScope = NetUtils.removeDuplicate(this, mutableExNodes, - excludedScope, ancestorGen); + // remove duplicate in mutableExNodes and mutableExcludedScopes + NetUtils.removeDuplicate(this, mutableExNodes, mutableExcludedScopes, + ancestorGen); // calculate available node count Node scopeNode = getNode(finalScope); int availableNodes = getAvailableNodesCount( - scopeNode.getNetworkFullPath(), excludedScope, mutableExNodes, + scopeNode.getNetworkFullPath(), mutableExcludedScopes, mutableExNodes, ancestorGen); if (availableNodes <= 0) { LOG.warn("No available node in (scope=\"{}\" excludedScope=\"{}\" " + "excludedNodes=\"{}\" ancestorGen=\"{}\").", - scopeNode.getNetworkFullPath(), excludedScope, excludedNodes, + scopeNode.getNetworkFullPath(), excludedScopes, excludedNodes, ancestorGen); return null; } - LOG.debug("Choosing random from \"{}\" available nodes on node \"{}\"," + - " scope=\"{}\", excludedScope=\"{}\", excludeNodes=\"{}\".", - availableNodes, scopeNode, scopeNode.getNetworkFullPath(), - excludedScope, excludedNodes); // scope is a Leaf node if (!(scopeNode instanceof InnerNode)) { @@ -556,15 +552,24 @@ private Node chooseNodeInternal(String scope, int leafIndex, } Node ret; + int nodeIndex; if (leafIndex >= 0) { - ret = ((InnerNode)scopeNode).getLeaf(leafIndex % availableNodes, - excludedScope, mutableExNodes, ancestorGen); + nodeIndex = leafIndex % availableNodes; + ret = ((InnerNode)scopeNode).getLeaf(nodeIndex, mutableExcludedScopes, + mutableExNodes, ancestorGen); } else { - final int index = ThreadLocalRandom.current().nextInt(availableNodes); - ret = ((InnerNode)scopeNode).getLeaf(index, excludedScope, mutableExNodes, - ancestorGen); + nodeIndex = ThreadLocalRandom.current().nextInt(availableNodes); + ret = ((InnerNode)scopeNode).getLeaf(nodeIndex, mutableExcludedScopes, + mutableExNodes, ancestorGen); + } + if (LOG.isDebugEnabled()) { + LOG.debug("Choosing node[index={},random={}] from \"{}\" available " + + "nodes, scope=\"{}\", excludedScope=\"{}\", excludeNodes=\"{}\".", + nodeIndex, (leafIndex == -1 ? "true" : "false"), availableNodes, + scopeNode.getNetworkFullPath(), excludedScopes, excludedNodes); + LOG.debug("Chosen node = {}", (ret == null ? "not found" : + ret.toString())); } - LOG.debug("chooseRandom return {}", ret); return ret; } @@ -583,13 +588,16 @@ public int getDistanceCost(Node node1, Node node2) { (node1 == null && node2 == null)) { return 0; } + if (node1 == null || node2 == null) { + LOG.warn("One of the nodes is a null pointer"); + return Integer.MAX_VALUE; + } int cost = 0; netlock.readLock().lock(); try { - if (node1 == null || node2 == null || - (node1.getAncestor(maxLevel - 1) != clusterTree) || + if ((node1.getAncestor(maxLevel - 1) != clusterTree) || (node2.getAncestor(maxLevel - 1) != clusterTree)) { - LOG.warn("One of the nodes is a null pointer"); + LOG.debug("One of the nodes is outside of network topology"); return Integer.MAX_VALUE; } int level1 = node1.getLevel(); @@ -630,17 +638,21 @@ public int getDistanceCost(Node node1, Node node2) { * @param nodes Available replicas with the requested data * @param activeLen Number of active nodes at the front of the array */ - public void sortByDistanceCost(Node reader, Node[] nodes, int activeLen) { + public List sortByDistanceCost(Node reader, + List nodes, int activeLen) { /** Sort weights for the nodes array */ + if (reader == null) { + return nodes; + } int[] costs = new int[activeLen]; for (int i = 0; i < activeLen; i++) { - costs[i] = getDistanceCost(reader, nodes[i]); + costs[i] = getDistanceCost(reader, nodes.get(i)); } // Add cost/node pairs to a TreeMap to sort TreeMap> tree = new TreeMap>(); for (int i = 0; i < activeLen; i++) { int cost = costs[i]; - Node node = nodes[i]; + Node node = nodes.get(i); List list = tree.get(cost); if (list == null) { list = Lists.newArrayListWithExpectedSize(1); @@ -648,30 +660,33 @@ public void sortByDistanceCost(Node reader, Node[] nodes, int activeLen) { } list.add(node); } - int idx = 0; + + List ret = new ArrayList<>(); for (List list: tree.values()) { if (list != null) { Collections.shuffle(list); for (Node n: list) { - nodes[idx] = n; - idx++; + ret.add(n); } } } - Preconditions.checkState(idx == activeLen, "Wrong number of nodes sorted!"); + + Preconditions.checkState(ret.size() == activeLen, + "Wrong number of nodes sorted!"); + return ret; } /** * Return the number of leaves in scope but not in * excludedNodes and excludeScope. * @param scope the scope - * @param excludedScope excluded scope + * @param excludedScopes excluded scopes * @param mutableExcludedNodes a list of excluded nodes, content might be * changed after the call * @param ancestorGen same generation ancestor prohibit on excludedNodes * @return number of available nodes */ - private int getAvailableNodesCount(String scope, String excludedScope, + private int getAvailableNodesCount(String scope, List excludedScopes, Collection mutableExcludedNodes, int ancestorGen) { Preconditions.checkArgument(scope != null); @@ -689,13 +704,15 @@ private int getAvailableNodesCount(String scope, String excludedScope, } // number of nodes to exclude int excludedCount = 0; - if (excludedScope != null) { - Node excludedScopeNode = getNode(excludedScope); - if (excludedScopeNode != null) { - if (excludedScope.startsWith(scope)) { - excludedCount += excludedScopeNode.getNumOfLeaves(); - } else if (scope.startsWith(excludedScope)) { - return 0; + if (excludedScopes != null) { + for (String excludedScope: excludedScopes) { + Node excludedScopeNode = getNode(excludedScope); + if (excludedScopeNode != null) { + if (excludedScope.startsWith(scope)) { + excludedCount += excludedScopeNode.getNumOfLeaves(); + } else if (scope.startsWith(excludedScope)) { + return 0; + } } } } @@ -732,7 +749,7 @@ public String toString() { try { // print the number of leaves int numOfLeaves = clusterTree.getNumOfLeaves(); - tree.append("Expected number of leaves:"); + tree.append("Number of leaves:"); tree.append(numOfLeaves); tree.append("\n"); // print all nodes @@ -753,11 +770,14 @@ private void checkScope(String scope) { } } - private void checkExcludedScope(String excludedScope) { - if (excludedScope != null && - (excludedScope.startsWith(SCOPE_REVERSE_STR))) { - throw new IllegalArgumentException("excludedScope " + excludedScope + - " cannot start with " + SCOPE_REVERSE_STR); + private void checkExcludedScopes(List excludedScopes) { + if (!CollectionUtils.isEmpty(excludedScopes)) { + excludedScopes.stream().forEach(scope -> { + if (scope.startsWith(SCOPE_REVERSE_STR)) { + throw new IllegalArgumentException("excludedScope " + scope + + " cannot start with " + SCOPE_REVERSE_STR); + } + }); } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/Node.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/Node.java index 310b336269830..0007e546770f0 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/Node.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/Node.java @@ -32,11 +32,23 @@ public interface Node { * exclude itself. In another words, its parent's full network location */ String getNetworkLocation(); + /** + * Set this node's network location. + * @param location it's network location + */ + void setNetworkLocation(String location); + /** @return this node's self name in network topology. This should be node's * IP or hostname. * */ String getNetworkName(); + /** + * Set this node's name, can be hostname or Ipaddress. + * @param name it's network name + */ + void setNetworkName(String name); + /** @return this node's full path in network topology. It's the concatenation * of location and name. * */ diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeImpl.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeImpl.java index a9763b971932e..53b05ea294166 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeImpl.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeImpl.java @@ -27,11 +27,11 @@ */ public class NodeImpl implements Node { // host:port# - private final String name; + private String name; // string representation of this node's location, such as /dc1/rack1 - private final String location; + private String location; // location + "/" + name - private final String path; + private String path; // which level of the tree the node resides, start from 1 for root private int level; // node's parent @@ -53,10 +53,7 @@ public NodeImpl(String name, String location, int cost) { } this.name = (name == null) ? ROOT : name; this.location = NetUtils.normalize(location); - this.path = this.location.equals(PATH_SEPARATOR_STR) ? - this.location + this.name : - this.location + PATH_SEPARATOR_STR + this.name; - + this.path = getPath(); this.cost = cost; } @@ -84,6 +81,15 @@ public String getNetworkName() { return name; } + /** + * Set this node's name, can be hostname or Ipaddress. + * @param networkName it's network name + */ + public void setNetworkName(String networkName) { + this.name = networkName; + this.path = getPath(); + } + /** * @return this node's network location */ @@ -91,6 +97,16 @@ public String getNetworkLocation() { return location; } + /** + * Set this node's network location. + * @param networkLocation it's network location + */ + @Override + public void setNetworkLocation(String networkLocation) { + this.location = networkLocation; + this.path = getPath(); + } + /** * @return this node's full path in network topology. It's the concatenation * of location and name. @@ -197,4 +213,10 @@ public int hashCode() { public String toString() { return getNetworkFullPath(); } + + private String getPath() { + return this.location.equals(PATH_SEPARATOR_STR) ? + this.location + this.name : + this.location + PATH_SEPARATOR_STR + this.name; + } } \ No newline at end of file diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaLoader.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaLoader.java index 32d7f16a9969d..8d7abedf2e74a 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaLoader.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaLoader.java @@ -17,6 +17,7 @@ */ package org.apache.hadoop.hdds.scm.net; +import org.apache.commons.io.FilenameUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.w3c.dom.Document; @@ -31,7 +32,9 @@ import javax.xml.parsers.ParserConfigurationException; import java.io.File; import java.io.FileInputStream; +import java.io.FileNotFoundException; import java.io.IOException; +import java.io.InputStream; import java.util.ArrayList; import java.util.HashMap; import java.util.List; @@ -93,44 +96,84 @@ public List getSchemaList() { } /** - * Load user defined network layer schemas from a XML configuration file. + * Load user defined network layer schemas from a XML/YAML configuration file. * @param schemaFilePath path of schema file * @return all valid node schemas defined in schema file */ - public NodeSchemaLoadResult loadSchemaFromXml(String schemaFilePath) - throws IllegalArgumentException { + public NodeSchemaLoadResult loadSchemaFromFile(String schemaFilePath) + throws IllegalArgumentException, FileNotFoundException { try { File schemaFile = new File(schemaFilePath); - if (!schemaFile.exists()) { - String msg = "Network topology layer schema file " + schemaFilePath + - " is not found."; - LOG.warn(msg); - throw new IllegalArgumentException(msg); + + if (schemaFile.exists()) { + LOG.info("Load network topology schema file " + + schemaFile.getAbsolutePath()); + try (FileInputStream inputStream = new FileInputStream(schemaFile)) { + return loadSchemaFromStream(schemaFilePath, inputStream); + } + } else { + // try to load with classloader + ClassLoader classloader = + Thread.currentThread().getContextClassLoader(); + if (classloader == null) { + classloader = NodeSchemaLoader.class.getClassLoader(); + } + if (classloader != null) { + try (InputStream stream = classloader + .getResourceAsStream(schemaFilePath)) { + if (stream != null) { + LOG.info("Loading file from " + classloader + .getResources(schemaFilePath)); + return loadSchemaFromStream(schemaFilePath, stream); + } + } + } + } - return loadSchema(schemaFile); + + String msg = "Network topology layer schema file " + + schemaFilePath + "[" + schemaFile.getAbsolutePath() + + "] is not found."; + LOG.warn(msg); + throw new FileNotFoundException(msg); + + } catch (FileNotFoundException e) { + throw e; } catch (ParserConfigurationException | IOException | SAXException e) { - throw new IllegalArgumentException("Fail to load network topology node" - + " schema file: " + schemaFilePath + " , error:" + e.getMessage()); + throw new IllegalArgumentException("Failed to load network topology node" + + " schema file: " + schemaFilePath + " , error:" + e.getMessage(), + e); + } + } + + private NodeSchemaLoadResult loadSchemaFromStream(String schemaFilePath, + InputStream stream) + throws ParserConfigurationException, SAXException, IOException { + if (FilenameUtils.getExtension(schemaFilePath).toLowerCase() + .compareTo("yaml") == 0) { + return loadSchemaFromYaml(stream); + } else { + return loadSchema(stream); } } /** * Load network topology layer schemas from a XML configuration file. - * @param schemaFile schema file + * @param inputStream schema file as an inputStream * @return all valid node schemas defined in schema file * @throws ParserConfigurationException ParserConfigurationException happen * @throws IOException no such schema file * @throws SAXException xml file has some invalid elements * @throws IllegalArgumentException xml file content is logically invalid */ - private NodeSchemaLoadResult loadSchema(File schemaFile) throws + private NodeSchemaLoadResult loadSchema(InputStream inputStream) throws ParserConfigurationException, SAXException, IOException { - LOG.info("Loading network topology layer schema file " + schemaFile); + LOG.info("Loading network topology layer schema file"); // Read and parse the schema file. DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance(); dbf.setIgnoringComments(true); DocumentBuilder builder = dbf.newDocumentBuilder(); - Document doc = builder.parse(schemaFile); + Document doc = builder.parse(inputStream); Element root = doc.getDocumentElement(); if (!CONFIGURATION_TAG.equals(root.getTagName())) { @@ -167,39 +210,16 @@ private NodeSchemaLoadResult loadSchema(File schemaFile) throws return schemaList; } - /** - * Load user defined network layer schemas from a YAML configuration file. - * @param schemaFilePath path of schema file - * @return all valid node schemas defined in schema file - */ - public NodeSchemaLoadResult loadSchemaFromYaml(String schemaFilePath) - throws IllegalArgumentException { - try { - File schemaFile = new File(schemaFilePath); - if (!schemaFile.exists()) { - String msg = "Network topology layer schema file " + schemaFilePath + - " is not found."; - LOG.warn(msg); - throw new IllegalArgumentException(msg); - } - return loadSchemaFromYaml(schemaFile); - } catch (Exception e) { - throw new IllegalArgumentException("Fail to load network topology node" - + " schema file: " + schemaFilePath + " , error:" - + e.getMessage()); - } - } - /** * Load network topology layer schemas from a YAML configuration file. - * @param schemaFile schema file + * @param schemaFile as inputStream * @return all valid node schemas defined in schema file * @throws ParserConfigurationException ParserConfigurationException happen * @throws IOException no such schema file * @throws SAXException xml file has some invalid elements * @throws IllegalArgumentException xml file content is logically invalid */ - private NodeSchemaLoadResult loadSchemaFromYaml(File schemaFile) { + private NodeSchemaLoadResult loadSchemaFromYaml(InputStream schemaFile) { LOG.info("Loading network topology layer schema file {}", schemaFile); NodeSchemaLoadResult finalSchema; @@ -207,13 +227,12 @@ private NodeSchemaLoadResult loadSchemaFromYaml(File schemaFile) { Yaml yaml = new Yaml(); NodeSchema nodeTree; - try (FileInputStream fileInputStream = new FileInputStream(schemaFile)) { - nodeTree = yaml.loadAs(fileInputStream, NodeSchema.class); - } + nodeTree = yaml.loadAs(schemaFile, NodeSchema.class); + List schemaList = new ArrayList<>(); if (nodeTree.getType() != LayerType.ROOT) { throw new IllegalArgumentException("First layer is not a ROOT node." - + " schema file: " + schemaFile.getAbsolutePath()); + + " schema file."); } schemaList.add(nodeTree); if (nodeTree.getSublayer() != null) { @@ -224,11 +243,11 @@ private NodeSchemaLoadResult loadSchemaFromYaml(File schemaFile) { if (nodeTree.getType() == LayerType.LEAF_NODE && nodeTree.getSublayer() != null) { throw new IllegalArgumentException("Leaf node in the middle of path." - + " schema file: " + schemaFile.getAbsolutePath()); + + " schema file."); } if (nodeTree.getType() == LayerType.ROOT) { throw new IllegalArgumentException("Multiple root nodes are defined." - + " schema file: " + schemaFile.getAbsolutePath()); + + " schema file."); } schemaList.add(nodeTree); if (nodeTree.getSublayer() != null) { @@ -238,12 +257,10 @@ private NodeSchemaLoadResult loadSchemaFromYaml(File schemaFile) { } } finalSchema = new NodeSchemaLoadResult(schemaList, true); - } catch (RuntimeException e) { - throw e; } catch (Exception e) { throw new IllegalArgumentException("Fail to load network topology node" - + " schema file: " + schemaFile.getAbsolutePath() + " , error:" - + e.getMessage()); + + " schema file: " + schemaFile + " , error:" + + e.getMessage(), e); } return finalSchema; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaManager.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaManager.java index 9a598c619242b..c60c2c80aa9e7 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaManager.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaManager.java @@ -19,7 +19,6 @@ import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; -import org.apache.commons.io.FilenameUtils; import org.apache.hadoop.hdds.scm.ScmConfigKeys; import org.apache.hadoop.hdds.scm.net.NodeSchemaLoader.NodeSchemaLoadResult; import org.slf4j.Logger; @@ -63,23 +62,17 @@ public void init(Configuration conf) { String schemaFile = conf.get( ScmConfigKeys.OZONE_SCM_NETWORK_TOPOLOGY_SCHEMA_FILE, ScmConfigKeys.OZONE_SCM_NETWORK_TOPOLOGY_SCHEMA_FILE_DEFAULT); - NodeSchemaLoadResult result; try { - if (FilenameUtils.getExtension(schemaFile).toLowerCase() - .compareTo("yaml") == 0) { - result = NodeSchemaLoader.getInstance().loadSchemaFromYaml(schemaFile); - } else { - result = NodeSchemaLoader.getInstance().loadSchemaFromXml(schemaFile); - } + result = NodeSchemaLoader.getInstance().loadSchemaFromFile(schemaFile); allSchema = result.getSchemaList(); enforcePrefix = result.isEnforePrefix(); maxLevel = allSchema.size(); } catch (Throwable e) { - String msg = "Fail to load schema file:" + schemaFile - + ", error:" + e.getMessage(); - LOG.error(msg); - throw new RuntimeException(msg); + String msg = "Failed to load schema file:" + schemaFile + + ", error: " + e.getMessage(); + LOG.error(msg, e); + throw new RuntimeException(msg, e); } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java index 7f75dd1b000e5..2828f6ea41ca0 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java @@ -25,9 +25,12 @@ import org.apache.hadoop.hdds.protocol.proto.HddsProtos; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationType; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import java.io.IOException; import java.util.ArrayList; +import java.util.Iterator; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -38,12 +41,15 @@ */ public final class Pipeline { + private static final Logger LOG = LoggerFactory.getLogger(Pipeline.class); private final PipelineID id; private final ReplicationType type; private final ReplicationFactor factor; private PipelineState state; private Map nodeStatus; + // nodes with ordered distance to client + private ThreadLocal> nodesInOrder = new ThreadLocal<>(); /** * The immutable properties of pipeline object is used in @@ -112,6 +118,14 @@ public DatanodeDetails getFirstNode() throws IOException { return nodeStatus.keySet().iterator().next(); } + public DatanodeDetails getClosestNode() throws IOException { + if (nodesInOrder.get() == null || nodesInOrder.get().isEmpty()) { + LOG.debug("Nodes in order is empty, delegate to getFirstNode"); + return getFirstNode(); + } + return nodesInOrder.get().get(0); + } + public boolean isClosed() { return state == PipelineState.CLOSED; } @@ -120,6 +134,18 @@ public boolean isOpen() { return state == PipelineState.OPEN; } + public void setNodesInOrder(List nodes) { + nodesInOrder.set(nodes); + } + + public List getNodesInOrder() { + if (nodesInOrder.get() == null || nodesInOrder.get().isEmpty()) { + LOG.debug("Nodes in order is empty, delegate to getNodes"); + return getNodes(); + } + return nodesInOrder.get(); + } + void reportDatanode(DatanodeDetails dn) throws IOException { if (nodeStatus.get(dn) == null) { throw new IOException( @@ -152,6 +178,24 @@ public HddsProtos.Pipeline getProtobufMessage() .addAllMembers(nodeStatus.keySet().stream() .map(DatanodeDetails::getProtoBufMessage) .collect(Collectors.toList())); + // To save the message size on wire, only transfer the node order based on + // network topology + List nodes = nodesInOrder.get(); + if (nodes != null && !nodes.isEmpty()) { + for (int i = 0; i < nodes.size(); i++) { + Iterator it = nodeStatus.keySet().iterator(); + for (int j = 0; j < nodeStatus.keySet().size(); j++) { + if (it.next().equals(nodes.get(i))) { + builder.addMemberOrders(j); + break; + } + } + } + if (LOG.isDebugEnabled()) { + LOG.debug("Serialize pipeline {} with nodesInOrder{ }", id.toString(), + nodes); + } + } return builder.build(); } @@ -164,10 +208,10 @@ public static Pipeline getFromProtobuf(HddsProtos.Pipeline pipeline) .setState(PipelineState.fromProtobuf(pipeline.getState())) .setNodes(pipeline.getMembersList().stream() .map(DatanodeDetails::getFromProtoBuf).collect(Collectors.toList())) + .setNodesInOrder(pipeline.getMemberOrdersList()) .build(); } - @Override public boolean equals(Object o) { if (this == o) { @@ -228,6 +272,8 @@ public static class Builder { private ReplicationFactor factor = null; private PipelineState state = null; private Map nodeStatus = null; + private List nodeOrder = null; + private List nodesInOrder = null; public Builder() {} @@ -237,6 +283,7 @@ public Builder(Pipeline pipeline) { this.factor = pipeline.factor; this.state = pipeline.state; this.nodeStatus = pipeline.nodeStatus; + this.nodesInOrder = pipeline.nodesInOrder.get(); } public Builder setId(PipelineID id1) { @@ -265,13 +312,44 @@ public Builder setNodes(List nodes) { return this; } + public Builder setNodesInOrder(List orders) { + this.nodeOrder = orders; + return this; + } + public Pipeline build() { Preconditions.checkNotNull(id); Preconditions.checkNotNull(type); Preconditions.checkNotNull(factor); Preconditions.checkNotNull(state); Preconditions.checkNotNull(nodeStatus); - return new Pipeline(id, type, factor, state, nodeStatus); + Pipeline pipeline = new Pipeline(id, type, factor, state, nodeStatus); + + if (nodeOrder != null && !nodeOrder.isEmpty()) { + // This branch is for build from ProtoBuf + List nodesWithOrder = new ArrayList<>(); + for(int i = 0; i < nodeOrder.size(); i++) { + int nodeIndex = nodeOrder.get(i); + Iterator it = nodeStatus.keySet().iterator(); + while(it.hasNext() && nodeIndex >= 0) { + DatanodeDetails node = it.next(); + if (nodeIndex == 0) { + nodesWithOrder.add(node); + break; + } + nodeIndex--; + } + } + if (LOG.isDebugEnabled()) { + LOG.debug("Deserialize nodesInOrder {} in pipeline {}", + nodesWithOrder, id.toString()); + } + pipeline.setNodesInOrder(nodesWithOrder); + } else if (nodesInOrder != null){ + // This branch is for pipeline clone + pipeline.setNodesInOrder(nodesInOrder); + } + return pipeline; } } @@ -279,7 +357,7 @@ public Pipeline build() { * Possible Pipeline states in SCM. */ public enum PipelineState { - ALLOCATED, OPEN, CLOSED; + ALLOCATED, OPEN, DORMANT, CLOSED; public static PipelineState fromProtobuf(HddsProtos.PipelineState state) throws UnknownPipelineStateException { @@ -287,6 +365,7 @@ public static PipelineState fromProtobuf(HddsProtos.PipelineState state) switch (state) { case PIPELINE_ALLOCATED: return ALLOCATED; case PIPELINE_OPEN: return OPEN; + case PIPELINE_DORMANT: return DORMANT; case PIPELINE_CLOSED: return CLOSED; default: throw new UnknownPipelineStateException( @@ -300,6 +379,7 @@ public static HddsProtos.PipelineState getProtobuf(PipelineState state) switch (state) { case ALLOCATED: return HddsProtos.PipelineState.PIPELINE_ALLOCATED; case OPEN: return HddsProtos.PipelineState.PIPELINE_OPEN; + case DORMANT: return HddsProtos.PipelineState.PIPELINE_DORMANT; case CLOSED: return HddsProtos.PipelineState.PIPELINE_CLOSED; default: throw new UnknownPipelineStateException( diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java index 12b591210ac38..18045f88cbd27 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java @@ -17,6 +17,7 @@ */ package org.apache.hadoop.hdds.scm.protocol; +import org.apache.hadoop.hdds.protocol.DatanodeDetails; import org.apache.hadoop.hdds.scm.ScmConfigKeys; import org.apache.hadoop.hdds.scm.container.common.helpers.ExcludeList; import org.apache.hadoop.security.KerberosInfo; @@ -74,4 +75,12 @@ List allocateBlock(long size, int numBlocks, * Gets the Clusterid and SCM Id from SCM. */ ScmInfo getScmInfo() throws IOException; + + /** + * Sort datanodes with distance to client. + * @param nodes list of network name of each node. + * @param clientMachine client address, depends, can be hostname or ipaddress. + */ + List sortDatanodes(List nodes, + String clientMachine) throws IOException; } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java index cc220a5dad617..88db8205a408f 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java @@ -146,6 +146,22 @@ Pipeline createReplicationPipeline(HddsProtos.ReplicationType type, */ List listPipelines() throws IOException; + /** + * Activates a dormant pipeline. + * + * @param pipelineID ID of the pipeline to activate. + * @throws IOException in case of any Exception + */ + void activatePipeline(HddsProtos.PipelineID pipelineID) throws IOException; + + /** + * Deactivates an active pipeline. + * + * @param pipelineID ID of the pipeline to deactivate. + * @throws IOException in case of any Exception + */ + void deactivatePipeline(HddsProtos.PipelineID pipelineID) throws IOException; + /** * Closes a pipeline given the pipelineID. * @@ -177,4 +193,22 @@ Pipeline createReplicationPipeline(HddsProtos.ReplicationType type, * @throws IOException */ boolean forceExitSafeMode() throws IOException; + + /** + * Start ReplicationManager. + */ + void startReplicationManager() throws IOException; + + /** + * Stop ReplicationManager. + */ + void stopReplicationManager() throws IOException; + + /** + * Returns ReplicationManager status. + * + * @return True if ReplicationManager is running, false otherwise. + */ + boolean getReplicationManagerStatus() throws IOException; + } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ScmBlockLocationProtocolClientSideTranslatorPB.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ScmBlockLocationProtocolClientSideTranslatorPB.java index 559022f286974..a262bb5bdbdd9 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ScmBlockLocationProtocolClientSideTranslatorPB.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ScmBlockLocationProtocolClientSideTranslatorPB.java @@ -24,19 +24,29 @@ import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.hdds.client.ContainerBlockID; +import org.apache.hadoop.hdds.protocol.DatanodeDetails; import org.apache.hadoop.hdds.protocol.proto.HddsProtos; +import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos.SCMBlockLocationRequest; +import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos.SCMBlockLocationResponse; +import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos.Type; import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos.AllocateBlockResponse; import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos.AllocateScmBlockRequestProto; import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos.AllocateScmBlockResponseProto; import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos.DeleteScmKeyBlocksRequestProto; import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos.DeleteScmKeyBlocksResponseProto; import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos.KeyBlocks; +import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos + .SortDatanodesRequestProto; +import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos + .SortDatanodesResponseProto; import org.apache.hadoop.hdds.scm.ScmInfo; import org.apache.hadoop.hdds.scm.container.common.helpers.AllocatedBlock; import org.apache.hadoop.hdds.scm.container.common.helpers.ExcludeList; +import org.apache.hadoop.hdds.scm.exceptions.SCMException; import org.apache.hadoop.hdds.scm.pipeline.Pipeline; import org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol; import org.apache.hadoop.hdds.tracing.TracingUtil; +import org.apache.hadoop.ipc.ProtobufHelper; import org.apache.hadoop.ipc.ProtocolTranslator; import org.apache.hadoop.ipc.RPC; import org.apache.hadoop.ozone.common.BlockGroup; @@ -46,6 +56,8 @@ import com.google.protobuf.RpcController; import com.google.protobuf.ServiceException; +import static org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos.Status.OK; + /** * This class is the client-side translator to translate the requests made on * the {@link ScmBlockLocationProtocol} interface to the RPC server @@ -72,6 +84,42 @@ public ScmBlockLocationProtocolClientSideTranslatorPB( this.rpcProxy = rpcProxy; } + /** + * Returns a SCMBlockLocationRequest builder with specified type. + * @param cmdType type of the request + */ + private SCMBlockLocationRequest.Builder createSCMBlockRequest(Type cmdType) { + return SCMBlockLocationRequest.newBuilder() + .setCmdType(cmdType) + .setTraceID(TracingUtil.exportCurrentSpan()); + } + + /** + * Submits client request to SCM server. + * @param req client request + * @return response from SCM + * @throws IOException thrown if any Protobuf service exception occurs + */ + private SCMBlockLocationResponse submitRequest( + SCMBlockLocationRequest req) throws IOException { + try { + SCMBlockLocationResponse response = + rpcProxy.send(NULL_RPC_CONTROLLER, req); + return response; + } catch (ServiceException e) { + throw ProtobufHelper.getRemoteException(e); + } + } + + private SCMBlockLocationResponse handleError(SCMBlockLocationResponse resp) + throws SCMException { + if (resp.getStatus() != OK) { + throw new SCMException(resp.getMessage(), + SCMException.ResultCodes.values()[resp.getStatus().ordinal()]); + } + return resp; + } + /** * Asks SCM where a block should be allocated. SCM responds with the * set of datanodes that should be used creating this block. @@ -96,20 +144,18 @@ public List allocateBlock(long size, int num, .setType(type) .setFactor(factor) .setOwner(owner) - .setTraceID(TracingUtil.exportCurrentSpan()) .setExcludeList(excludeList.getProtoBuf()) .build(); - final AllocateScmBlockResponseProto response; - try { - response = rpcProxy.allocateScmBlock(NULL_RPC_CONTROLLER, request); - } catch (ServiceException e) { - throw transformServiceException(e); - } - if (response.getErrorCode() != - AllocateScmBlockResponseProto.Error.success) { - throw new IOException(response.hasErrorMessage() ? - response.getErrorMessage() : "Allocate block failed."); - } + + SCMBlockLocationRequest wrapper = createSCMBlockRequest( + Type.AllocateScmBlock) + .setAllocateScmBlockRequest(request) + .build(); + + final SCMBlockLocationResponse wrappedResponse = + handleError(submitRequest(wrapper)); + final AllocateScmBlockResponseProto response = + wrappedResponse.getAllocateScmBlockResponse(); List blocks = new ArrayList<>(response.getBlocksCount()); for (AllocateBlockResponse resp : response.getBlocksList()) { @@ -141,12 +187,16 @@ public List deleteKeyBlocks( .addAllKeyBlocks(keyBlocksProto) .build(); - final DeleteScmKeyBlocksResponseProto resp; - try { - resp = rpcProxy.deleteScmKeyBlocks(NULL_RPC_CONTROLLER, request); - } catch (ServiceException e) { - throw transformServiceException(e); - } + SCMBlockLocationRequest wrapper = createSCMBlockRequest( + Type.DeleteScmKeyBlocks) + .setDeleteScmKeyBlocksRequest(request) + .build(); + + final SCMBlockLocationResponse wrappedResponse = + handleError(submitRequest(wrapper)); + final DeleteScmKeyBlocksResponseProto resp = + wrappedResponse.getDeleteScmKeyBlocksResponse(); + List results = new ArrayList<>(resp.getResultsCount()); results.addAll(resp.getResultsList().stream().map( @@ -157,30 +207,6 @@ public List deleteKeyBlocks( return results; } - private IOException transformServiceException( - ServiceException se) throws IOException { - //TODO SCM has no perfect way to return with business exceptions. All - //the exceptions will be mapped to ServiceException. - //ServiceException is handled in a special way in hadoop rpc: the message - //contains the whole stack trace which is not required for the business - //exception. As of now I remove the stack trace (use first line only). - //Long term we need a proper way of the exception propagation. - Throwable cause = se.getCause(); - if (cause == null) { - return new IOException( - new ServiceException(useFirstLine(se.getMessage()), se.getCause())); - } - return new IOException(useFirstLine(cause.getMessage()), cause.getCause()); - } - - private String useFirstLine(String message) { - if (message == null) { - return null; - } else { - return message.split("\n")[0]; - } - } - /** * Gets the cluster Id and Scm Id from SCM. * @return ScmInfo @@ -191,17 +217,50 @@ public ScmInfo getScmInfo() throws IOException { HddsProtos.GetScmInfoRequestProto request = HddsProtos.GetScmInfoRequestProto.getDefaultInstance(); HddsProtos.GetScmInfoResponseProto resp; - try { - resp = rpcProxy.getScmInfo(NULL_RPC_CONTROLLER, request); - } catch (ServiceException e) { - throw transformServiceException(e); - } + + SCMBlockLocationRequest wrapper = createSCMBlockRequest( + Type.GetScmInfo) + .setGetScmInfoRequest(request) + .build(); + + final SCMBlockLocationResponse wrappedResponse = + handleError(submitRequest(wrapper)); + resp = wrappedResponse.getGetScmInfoResponse(); ScmInfo.Builder builder = new ScmInfo.Builder() .setClusterId(resp.getClusterId()) .setScmId(resp.getScmId()); return builder.build(); } + /** + * Sort the datanodes based on distance from client. + * @return List + * @throws IOException + */ + @Override + public List sortDatanodes(List nodes, + String clientMachine) throws IOException { + SortDatanodesRequestProto request = SortDatanodesRequestProto + .newBuilder() + .addAllNodeNetworkName(nodes) + .setClient(clientMachine) + .build(); + SCMBlockLocationRequest wrapper = createSCMBlockRequest( + Type.SortDatanodes) + .setSortDatanodesRequest(request) + .build(); + + final SCMBlockLocationResponse wrappedResponse = + handleError(submitRequest(wrapper)); + SortDatanodesResponseProto resp = + wrappedResponse.getSortDatanodesResponse(); + List results = new ArrayList<>(resp.getNodeCount()); + results.addAll(resp.getNodeList().stream() + .map(node -> DatanodeDetails.getFromProtoBuf(node)) + .collect(Collectors.toList())); + return results; + } + @Override public Object getUnderlyingProxyObject() { return rpcProxy; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java index ffb4686b3d378..01db597dfae1a 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java @@ -16,58 +16,57 @@ */ package org.apache.hadoop.hdds.scm.protocolPB; -import com.google.common.base.Preconditions; -import com.google.protobuf.RpcController; -import com.google.protobuf.ServiceException; +import java.io.Closeable; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.function.Consumer; + import org.apache.hadoop.classification.InterfaceAudience; -import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ListPipelineRequestProto; -import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ListPipelineResponseProto; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos.GetScmInfoResponseProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ActivatePipelineRequestProto; import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ClosePipelineRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ContainerRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ContainerResponseProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.DeactivatePipelineRequestProto; import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ForceExitSafeModeRequestProto; import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ForceExitSafeModeResponseProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.GetContainerRequestProto; import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.GetContainerWithPipelineRequestProto; -import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.GetContainerWithPipelineResponseProto; import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.InSafeModeRequestProto; -import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.InSafeModeResponseProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ListPipelineRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ListPipelineResponseProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.NodeQueryRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.NodeQueryResponseProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ObjectStageChangeRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.PipelineRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.PipelineResponseProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ReplicationManagerStatusRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ReplicationManagerStatusResponseProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.SCMDeleteContainerRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.SCMListContainerRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.SCMListContainerResponseProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ScmContainerLocationRequest; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ScmContainerLocationRequest.Builder; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.ScmContainerLocationResponse; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.StartReplicationManagerRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.StopReplicationManagerRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.Type; import org.apache.hadoop.hdds.scm.ScmInfo; -import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerWithPipeline; import org.apache.hadoop.hdds.scm.container.ContainerInfo; +import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerWithPipeline; import org.apache.hadoop.hdds.scm.pipeline.Pipeline; import org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol; -import org.apache.hadoop.hdds.protocol.proto.HddsProtos; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ContainerRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ContainerResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.GetContainerRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.GetContainerResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.NodeQueryRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.NodeQueryResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ObjectStageChangeRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.PipelineRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.PipelineResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.SCMDeleteContainerRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.SCMListContainerRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.SCMListContainerResponseProto; import org.apache.hadoop.hdds.tracing.TracingUtil; import org.apache.hadoop.ipc.ProtobufHelper; import org.apache.hadoop.ipc.ProtocolTranslator; import org.apache.hadoop.ipc.RPC; -import java.io.Closeable; -import java.io.IOException; -import java.util.ArrayList; -import java.util.List; +import com.google.common.base.Preconditions; +import com.google.protobuf.RpcController; +import com.google.protobuf.ServiceException; /** * This class is the client-side translator to translate the requests made on @@ -95,14 +94,35 @@ public StorageContainerLocationProtocolClientSideTranslatorPB( this.rpcProxy = rpcProxy; } + /** + * Helper method to wrap the request and send the message. + */ + private ScmContainerLocationResponse submitRequest( + StorageContainerLocationProtocolProtos.Type type, + Consumer builderConsumer) throws IOException { + final ScmContainerLocationResponse response; + try { + + Builder builder = ScmContainerLocationRequest.newBuilder() + .setCmdType(type) + .setTraceID(TracingUtil.exportCurrentSpan()); + builderConsumer.accept(builder); + ScmContainerLocationRequest wrapper = builder.build(); + + response = rpcProxy.submitRequest(NULL_RPC_CONTROLLER, wrapper); + } catch (ServiceException ex) { + throw ProtobufHelper.getRemoteException(ex); + } + return response; + } + /** * Asks SCM where a container should be allocated. SCM responds with the set * of datanodes that should be used creating this container. Ozone/SCM only * supports replication factor of either 1 or 3. - * @param type - Replication Type + * + * @param type - Replication Type * @param factor - Replication Count - * @return - * @throws IOException */ @Override public ContainerWithPipeline allocateContainer( @@ -116,12 +136,11 @@ public ContainerWithPipeline allocateContainer( .setOwner(owner) .build(); - final ContainerResponseProto response; - try { - response = rpcProxy.allocateContainer(NULL_RPC_CONTROLLER, request); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + ContainerResponseProto response = + submitRequest(Type.AllocateContainer, + builder -> builder.setContainerRequest(request)) + .getContainerResponse(); + //TODO should be migrated to use the top level status structure. if (response.getErrorCode() != ContainerResponseProto.Error.success) { throw new IOException(response.hasErrorMessage() ? response.getErrorMessage() : "Allocate container failed."); @@ -138,13 +157,12 @@ public ContainerInfo getContainer(long containerID) throws IOException { .setContainerID(containerID) .setTraceID(TracingUtil.exportCurrentSpan()) .build(); - try { - GetContainerResponseProto response = - rpcProxy.getContainer(NULL_RPC_CONTROLLER, request); - return ContainerInfo.fromProtobuf(response.getContainerInfo()); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + ScmContainerLocationResponse response = + submitRequest(Type.GetContainer, + (builder) -> builder.setGetContainerRequest(request)); + return ContainerInfo + .fromProtobuf(response.getGetContainerResponse().getContainerInfo()); + } /** @@ -158,14 +176,15 @@ public ContainerWithPipeline getContainerWithPipeline(long containerID) GetContainerWithPipelineRequestProto.newBuilder() .setTraceID(TracingUtil.exportCurrentSpan()) .setContainerID(containerID).build(); - try { - GetContainerWithPipelineResponseProto response = - rpcProxy.getContainerWithPipeline(NULL_RPC_CONTROLLER, request); - return ContainerWithPipeline.fromProtobuf( - response.getContainerWithPipeline()); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + + ScmContainerLocationResponse response = + submitRequest(Type.GetContainerWithPipeline, + (builder) -> builder.setGetContainerWithPipelineRequest(request)); + + return ContainerWithPipeline.fromProtobuf( + response.getGetContainerWithPipelineResponse() + .getContainerWithPipeline()); + } /** @@ -185,26 +204,22 @@ public List listContainer(long startContainerID, int count) builder.setTraceID(TracingUtil.exportCurrentSpan()); SCMListContainerRequestProto request = builder.build(); - try { - SCMListContainerResponseProto response = - rpcProxy.listContainer(NULL_RPC_CONTROLLER, request); - List containerList = new ArrayList<>(); - for (HddsProtos.ContainerInfoProto containerInfoProto : response - .getContainersList()) { - containerList.add(ContainerInfo.fromProtobuf(containerInfoProto)); - } - return containerList; - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); + SCMListContainerResponseProto response = + submitRequest(Type.ListContainer, + builder1 -> builder1.setScmListContainerRequest(request)) + .getScmListContainerResponse(); + List containerList = new ArrayList<>(); + for (HddsProtos.ContainerInfoProto containerInfoProto : response + .getContainersList()) { + containerList.add(ContainerInfo.fromProtobuf(containerInfoProto)); } + return containerList; + } /** * Ask SCM to delete a container by name. SCM will remove * the container mapping in its database. - * - * @param containerID - * @throws IOException */ @Override public void deleteContainer(long containerID) @@ -216,18 +231,13 @@ public void deleteContainer(long containerID) .setTraceID(TracingUtil.exportCurrentSpan()) .setContainerID(containerID) .build(); - try { - rpcProxy.deleteContainer(NULL_RPC_CONTROLLER, request); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + submitRequest(Type.DeleteContainer, + builder -> builder.setScmDeleteContainerRequest(request)); + } /** * Queries a list of Node Statuses. - * - * @param nodeStatuses - * @return List of Datanodes. */ @Override public List queryNode(HddsProtos.NodeState @@ -240,21 +250,18 @@ public List queryNode(HddsProtos.NodeState .setState(nodeStatuses) .setTraceID(TracingUtil.exportCurrentSpan()) .setScope(queryScope).setPoolName(poolName).build(); - try { - NodeQueryResponseProto response = - rpcProxy.queryNode(NULL_RPC_CONTROLLER, request); - return response.getDatanodesList(); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + NodeQueryResponseProto response = submitRequest(Type.QueryNode, + builder -> builder.setNodeQueryRequest(request)).getNodeQueryResponse(); + return response.getDatanodesList(); } /** * Notify from client that creates object on datanodes. - * @param type object type - * @param id object id - * @param op operation type (e.g., create, close, delete) + * + * @param type object type + * @param id object id + * @param op operation type (e.g., create, close, delete) * @param stage object creation stage : begin/complete */ @Override @@ -272,20 +279,17 @@ public void notifyObjectStageChange( .setOp(op) .setStage(stage) .build(); - try { - rpcProxy.notifyObjectStageChange(NULL_RPC_CONTROLLER, request); - } catch(ServiceException e){ - throw ProtobufHelper.getRemoteException(e); - } + submitRequest(Type.NotifyObjectStageChange, + builder -> builder.setObjectStageChangeRequest(request)); + } /** * Creates a replication pipeline of a specified type. * * @param replicationType - replication type - * @param factor - factor 1 or 3 - * @param nodePool - optional machine list to build a pipeline. - * @throws IOException + * @param factor - factor 1 or 3 + * @param nodePool - optional machine list to build a pipeline. */ @Override public Pipeline createReplicationPipeline(HddsProtos.ReplicationType @@ -297,57 +301,82 @@ public Pipeline createReplicationPipeline(HddsProtos.ReplicationType .setReplicationFactor(factor) .setReplicationType(replicationType) .build(); - try { - PipelineResponseProto response = - rpcProxy.allocatePipeline(NULL_RPC_CONTROLLER, request); - if (response.getErrorCode() == - PipelineResponseProto.Error.success) { - Preconditions.checkState(response.hasPipeline(), "With success, " + - "must come a pipeline"); - return Pipeline.getFromProtobuf(response.getPipeline()); - } else { - String errorMessage = String.format("create replication pipeline " + - "failed. code : %s Message: %s", response.getErrorCode(), - response.hasErrorMessage() ? response.getErrorMessage() : ""); - throw new IOException(errorMessage); - } - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); + + PipelineResponseProto response = + submitRequest(Type.AllocatePipeline, + builder -> builder.setPipelineRequest(request)) + .getPipelineResponse(); + if (response.getErrorCode() == + PipelineResponseProto.Error.success) { + Preconditions.checkState(response.hasPipeline(), "With success, " + + "must come a pipeline"); + return Pipeline.getFromProtobuf(response.getPipeline()); + } else { + String errorMessage = String.format("create replication pipeline " + + "failed. code : %s Message: %s", response.getErrorCode(), + response.hasErrorMessage() ? response.getErrorMessage() : ""); + throw new IOException(errorMessage); } + } @Override public List listPipelines() throws IOException { - try { - ListPipelineRequestProto request = ListPipelineRequestProto - .newBuilder().setTraceID(TracingUtil.exportCurrentSpan()) - .build(); - ListPipelineResponseProto response = rpcProxy.listPipelines( - NULL_RPC_CONTROLLER, request); - List list = new ArrayList<>(); - for (HddsProtos.Pipeline pipeline : response.getPipelinesList()) { - Pipeline fromProtobuf = Pipeline.getFromProtobuf(pipeline); - list.add(fromProtobuf); - } - return list; - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); + ListPipelineRequestProto request = ListPipelineRequestProto + .newBuilder().setTraceID(TracingUtil.exportCurrentSpan()) + .build(); + + ListPipelineResponseProto response = submitRequest(Type.ListPipelines, + builder -> builder.setListPipelineRequest(request)) + .getListPipelineResponse(); + + List list = new ArrayList<>(); + for (HddsProtos.Pipeline pipeline : response.getPipelinesList()) { + Pipeline fromProtobuf = Pipeline.getFromProtobuf(pipeline); + list.add(fromProtobuf); } + return list; + + } + + @Override + public void activatePipeline(HddsProtos.PipelineID pipelineID) + throws IOException { + ActivatePipelineRequestProto request = + ActivatePipelineRequestProto.newBuilder() + .setTraceID(TracingUtil.exportCurrentSpan()) + .setPipelineID(pipelineID) + .build(); + submitRequest(Type.ActivatePipeline, + builder -> builder.setActivatePipelineRequest(request)); + + } + + @Override + public void deactivatePipeline(HddsProtos.PipelineID pipelineID) + throws IOException { + + DeactivatePipelineRequestProto request = + DeactivatePipelineRequestProto.newBuilder() + .setTraceID(TracingUtil.exportCurrentSpan()) + .setPipelineID(pipelineID) + .build(); + submitRequest(Type.DeactivatePipeline, + builder -> builder.setDeactivatePipelineRequest(request)); } @Override public void closePipeline(HddsProtos.PipelineID pipelineID) throws IOException { - try { - ClosePipelineRequestProto request = - ClosePipelineRequestProto.newBuilder() - .setTraceID(TracingUtil.exportCurrentSpan()) - .setPipelineID(pipelineID) - .build(); - rpcProxy.closePipeline(NULL_RPC_CONTROLLER, request); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + + ClosePipelineRequestProto request = + ClosePipelineRequestProto.newBuilder() + .setTraceID(TracingUtil.exportCurrentSpan()) + .setPipelineID(pipelineID) + .build(); + submitRequest(Type.ClosePipeline, + builder -> builder.setClosePipelineRequest(request)); + } @Override @@ -356,16 +385,14 @@ public ScmInfo getScmInfo() throws IOException { HddsProtos.GetScmInfoRequestProto.newBuilder() .setTraceID(TracingUtil.exportCurrentSpan()) .build(); - try { - HddsProtos.GetScmInfoResponseProto resp = rpcProxy.getScmInfo( - NULL_RPC_CONTROLLER, request); - ScmInfo.Builder builder = new ScmInfo.Builder() - .setClusterId(resp.getClusterId()) - .setScmId(resp.getScmId()); - return builder.build(); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + + GetScmInfoResponseProto resp = submitRequest(Type.GetScmInfo, + builder -> builder.setGetScmInfoRequest(request)) + .getGetScmInfoResponse(); + ScmInfo.Builder builder = new ScmInfo.Builder() + .setClusterId(resp.getClusterId()) + .setScmId(resp.getScmId()); + return builder.build(); } @@ -373,38 +400,67 @@ public ScmInfo getScmInfo() throws IOException { * Check if SCM is in safe mode. * * @return Returns true if SCM is in safe mode else returns false. - * @throws IOException */ @Override public boolean inSafeMode() throws IOException { InSafeModeRequestProto request = InSafeModeRequestProto.getDefaultInstance(); - try { - InSafeModeResponseProto resp = rpcProxy.inSafeMode( - NULL_RPC_CONTROLLER, request); - return resp.getInSafeMode(); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + + return submitRequest(Type.InSafeMode, + builder -> builder.setInSafeModeRequest(request)) + .getInSafeModeResponse().getInSafeMode(); + } /** * Force SCM out of Safe mode. * * @return returns true if operation is successful. - * @throws IOException */ @Override public boolean forceExitSafeMode() throws IOException { ForceExitSafeModeRequestProto request = ForceExitSafeModeRequestProto.getDefaultInstance(); - try { - ForceExitSafeModeResponseProto resp = rpcProxy - .forceExitSafeMode(NULL_RPC_CONTROLLER, request); - return resp.getExitedSafeMode(); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } + ForceExitSafeModeResponseProto resp = + submitRequest(Type.ForceExitSafeMode, + builder -> builder.setForceExitSafeModeRequest(request)) + .getForceExitSafeModeResponse(); + + return resp.getExitedSafeMode(); + + } + + @Override + public void startReplicationManager() throws IOException { + + StartReplicationManagerRequestProto request = + StartReplicationManagerRequestProto.getDefaultInstance(); + submitRequest(Type.StartReplicationManager, + builder -> builder.setStartReplicationManagerRequest(request)); + + } + + @Override + public void stopReplicationManager() throws IOException { + + StopReplicationManagerRequestProto request = + StopReplicationManagerRequestProto.getDefaultInstance(); + submitRequest(Type.StopReplicationManager, + builder -> builder.setStopReplicationManagerRequest(request)); + + } + + @Override + public boolean getReplicationManagerStatus() throws IOException { + + ReplicationManagerStatusRequestProto request = + ReplicationManagerStatusRequestProto.getDefaultInstance(); + ReplicationManagerStatusResponseProto response = + submitRequest(Type.GetReplicationManagerStatus, + builder -> builder.setSeplicationManagerStatusRequest(request)) + .getReplicationManagerStatusResponse(); + return response.getIsRunning(); + } @Override diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/CheckedBiFunction.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/CheckedBiFunction.java new file mode 100644 index 0000000000000..df84859ab0294 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/CheckedBiFunction.java @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.scm.storage; + + +import java.io.IOException; + +/** + * Defines a functional interface having two inputs which throws IOException. + */ +@FunctionalInterface +public interface CheckedBiFunction { + void apply(LEFT left, RIGHT right) throws THROWABLE; +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/ContainerProtocolCalls.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/ContainerProtocolCalls.java index 5a1a75eb90fd9..d0ba60d9ad7e8 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/ContainerProtocolCalls.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/ContainerProtocolCalls.java @@ -18,7 +18,6 @@ package org.apache.hadoop.hdds.scm.storage; -import org.apache.hadoop.hdds.protocol.DatanodeDetails; import org.apache.hadoop.hdds.scm.XceiverClientReply; import org.apache.hadoop.hdds.scm.container.common.helpers .BlockNotCommittedException; @@ -72,6 +71,7 @@ import org.apache.hadoop.hdds.client.BlockID; import java.io.IOException; +import java.util.ArrayList; import java.util.List; import java.util.concurrent.ExecutionException; @@ -92,12 +92,11 @@ private ContainerProtocolCalls() { * * @param xceiverClient client to perform call * @param datanodeBlockID blockID to identify container - * @param traceID container protocol call args * @return container protocol get block response * @throws IOException if there is an I/O error while performing the call */ public static GetBlockResponseProto getBlock(XceiverClientSpi xceiverClient, - DatanodeBlockID datanodeBlockID, String traceID) throws IOException { + DatanodeBlockID datanodeBlockID) throws IOException { GetBlockRequestProto.Builder readBlockRequest = GetBlockRequestProto .newBuilder() .setBlockID(datanodeBlockID); @@ -107,7 +106,6 @@ public static GetBlockResponseProto getBlock(XceiverClientSpi xceiverClient, .newBuilder() .setCmdType(Type.GetBlock) .setContainerID(datanodeBlockID.getContainerID()) - .setTraceID(traceID) .setDatanodeUuid(id) .setGetBlock(readBlockRequest); String encodedToken = getEncodedBlockToken(getService(datanodeBlockID)); @@ -116,9 +114,8 @@ public static GetBlockResponseProto getBlock(XceiverClientSpi xceiverClient, } ContainerCommandRequestProto request = builder.build(); - ContainerCommandResponseProto response = xceiverClient.sendCommand(request); - validateContainerResponse(response); - + ContainerCommandResponseProto response = + xceiverClient.sendCommand(request, getValidatorList()); return response.getGetBlock(); } @@ -127,13 +124,12 @@ public static GetBlockResponseProto getBlock(XceiverClientSpi xceiverClient, * * @param xceiverClient client to perform call * @param blockID blockId for the Block - * @param traceID container protocol call args * @return container protocol getLastCommittedBlockLength response * @throws IOException if there is an I/O error while performing the call */ public static ContainerProtos.GetCommittedBlockLengthResponseProto getCommittedBlockLength( - XceiverClientSpi xceiverClient, BlockID blockID, String traceID) + XceiverClientSpi xceiverClient, BlockID blockID) throws IOException { ContainerProtos.GetCommittedBlockLengthRequestProto.Builder getBlockLengthRequestBuilder = @@ -144,7 +140,6 @@ public static GetBlockResponseProto getBlock(XceiverClientSpi xceiverClient, ContainerCommandRequestProto.newBuilder() .setCmdType(Type.GetCommittedBlockLength) .setContainerID(blockID.getContainerID()) - .setTraceID(traceID) .setDatanodeUuid(id) .setGetCommittedBlockLength(getBlockLengthRequestBuilder); String encodedToken = getEncodedBlockToken(new Text(blockID. @@ -153,8 +148,8 @@ public static GetBlockResponseProto getBlock(XceiverClientSpi xceiverClient, builder.setEncodedToken(encodedToken); } ContainerCommandRequestProto request = builder.build(); - ContainerCommandResponseProto response = xceiverClient.sendCommand(request); - validateContainerResponse(response); + ContainerCommandResponseProto response = + xceiverClient.sendCommand(request, getValidatorList()); return response.getGetCommittedBlockLength(); } @@ -163,20 +158,19 @@ public static GetBlockResponseProto getBlock(XceiverClientSpi xceiverClient, * * @param xceiverClient client to perform call * @param containerBlockData block data to identify container - * @param traceID container protocol call args * @return putBlockResponse * @throws IOException if there is an I/O error while performing the call */ public static ContainerProtos.PutBlockResponseProto putBlock( - XceiverClientSpi xceiverClient, BlockData containerBlockData, - String traceID) throws IOException { + XceiverClientSpi xceiverClient, BlockData containerBlockData) + throws IOException { PutBlockRequestProto.Builder createBlockRequest = PutBlockRequestProto.newBuilder().setBlockData(containerBlockData); String id = xceiverClient.getPipeline().getFirstNode().getUuidString(); ContainerCommandRequestProto.Builder builder = ContainerCommandRequestProto.newBuilder().setCmdType(Type.PutBlock) .setContainerID(containerBlockData.getBlockID().getContainerID()) - .setTraceID(traceID).setDatanodeUuid(id) + .setDatanodeUuid(id) .setPutBlock(createBlockRequest); String encodedToken = getEncodedBlockToken(getService(containerBlockData.getBlockID())); @@ -184,8 +178,8 @@ public static ContainerProtos.PutBlockResponseProto putBlock( builder.setEncodedToken(encodedToken); } ContainerCommandRequestProto request = builder.build(); - ContainerCommandResponseProto response = xceiverClient.sendCommand(request); - validateContainerResponse(response); + ContainerCommandResponseProto response = + xceiverClient.sendCommand(request, getValidatorList()); return response.getPutBlock(); } @@ -194,15 +188,13 @@ public static ContainerProtos.PutBlockResponseProto putBlock( * * @param xceiverClient client to perform call * @param containerBlockData block data to identify container - * @param traceID container protocol call args * @return putBlockResponse * @throws IOException if there is an error while performing the call * @throws InterruptedException * @throws ExecutionException */ public static XceiverClientReply putBlockAsync( - XceiverClientSpi xceiverClient, BlockData containerBlockData, - String traceID) + XceiverClientSpi xceiverClient, BlockData containerBlockData) throws IOException, InterruptedException, ExecutionException { PutBlockRequestProto.Builder createBlockRequest = PutBlockRequestProto.newBuilder().setBlockData(containerBlockData); @@ -210,7 +202,7 @@ public static XceiverClientReply putBlockAsync( ContainerCommandRequestProto.Builder builder = ContainerCommandRequestProto.newBuilder().setCmdType(Type.PutBlock) .setContainerID(containerBlockData.getBlockID().getContainerID()) - .setTraceID(traceID).setDatanodeUuid(id) + .setDatanodeUuid(id) .setPutBlock(createBlockRequest); String encodedToken = getEncodedBlockToken(getService(containerBlockData.getBlockID())); @@ -227,36 +219,31 @@ public static XceiverClientReply putBlockAsync( * @param xceiverClient client to perform call * @param chunk information about chunk to read * @param blockID ID of the block - * @param traceID container protocol call args - * @param excludeDns datamode to exclude while executing the command + * @param validators functions to validate the response * @return container protocol read chunk response * @throws IOException if there is an I/O error while performing the call */ - public static XceiverClientReply readChunk(XceiverClientSpi xceiverClient, - ChunkInfo chunk, BlockID blockID, String traceID, - List excludeDns) - throws IOException { - ReadChunkRequestProto.Builder readChunkRequest = ReadChunkRequestProto - .newBuilder() - .setBlockID(blockID.getDatanodeBlockIDProtobuf()) - .setChunkData(chunk); - String id = xceiverClient.getPipeline().getFirstNode().getUuidString(); - ContainerCommandRequestProto.Builder builder = ContainerCommandRequestProto - .newBuilder() - .setCmdType(Type.ReadChunk) - .setContainerID(blockID.getContainerID()) - .setTraceID(traceID) - .setDatanodeUuid(id) - .setReadChunk(readChunkRequest); + public static ContainerProtos.ReadChunkResponseProto readChunk( + XceiverClientSpi xceiverClient, ChunkInfo chunk, BlockID blockID, + List validators) throws IOException { + ReadChunkRequestProto.Builder readChunkRequest = + ReadChunkRequestProto.newBuilder() + .setBlockID(blockID.getDatanodeBlockIDProtobuf()) + .setChunkData(chunk); + String id = xceiverClient.getPipeline().getClosestNode().getUuidString(); + ContainerCommandRequestProto.Builder builder = + ContainerCommandRequestProto.newBuilder().setCmdType(Type.ReadChunk) + .setContainerID(blockID.getContainerID()) + .setDatanodeUuid(id).setReadChunk(readChunkRequest); String encodedToken = getEncodedBlockToken(new Text(blockID. getContainerBlockID().toString())); if (encodedToken != null) { builder.setEncodedToken(encodedToken); } ContainerCommandRequestProto request = builder.build(); - XceiverClientReply reply = - xceiverClient.sendCommand(request, excludeDns); - return reply; + ContainerCommandResponseProto reply = + xceiverClient.sendCommand(request, validators); + return reply.getReadChunk(); } /** @@ -266,11 +253,10 @@ public static XceiverClientReply readChunk(XceiverClientSpi xceiverClient, * @param chunk information about chunk to write * @param blockID ID of the block * @param data the data of the chunk to write - * @param traceID container protocol call args * @throws IOException if there is an error while performing the call */ public static void writeChunk(XceiverClientSpi xceiverClient, ChunkInfo chunk, - BlockID blockID, ByteString data, String traceID) + BlockID blockID, ByteString data) throws IOException { WriteChunkRequestProto.Builder writeChunkRequest = WriteChunkRequestProto .newBuilder() @@ -282,7 +268,6 @@ public static void writeChunk(XceiverClientSpi xceiverClient, ChunkInfo chunk, .newBuilder() .setCmdType(Type.WriteChunk) .setContainerID(blockID.getContainerID()) - .setTraceID(traceID) .setDatanodeUuid(id) .setWriteChunk(writeChunkRequest); String encodedToken = getEncodedBlockToken(new Text(blockID. @@ -291,8 +276,7 @@ public static void writeChunk(XceiverClientSpi xceiverClient, ChunkInfo chunk, builder.setEncodedToken(encodedToken); } ContainerCommandRequestProto request = builder.build(); - ContainerCommandResponseProto response = xceiverClient.sendCommand(request); - validateContainerResponse(response); + xceiverClient.sendCommand(request, getValidatorList()); } /** @@ -302,12 +286,11 @@ public static void writeChunk(XceiverClientSpi xceiverClient, ChunkInfo chunk, * @param chunk information about chunk to write * @param blockID ID of the block * @param data the data of the chunk to write - * @param traceID container protocol call args * @throws IOException if there is an I/O error while performing the call */ public static XceiverClientReply writeChunkAsync( XceiverClientSpi xceiverClient, ChunkInfo chunk, BlockID blockID, - ByteString data, String traceID) + ByteString data) throws IOException, ExecutionException, InterruptedException { WriteChunkRequestProto.Builder writeChunkRequest = WriteChunkRequestProto.newBuilder() @@ -316,7 +299,7 @@ public static XceiverClientReply writeChunkAsync( String id = xceiverClient.getPipeline().getFirstNode().getUuidString(); ContainerCommandRequestProto.Builder builder = ContainerCommandRequestProto.newBuilder().setCmdType(Type.WriteChunk) - .setContainerID(blockID.getContainerID()).setTraceID(traceID) + .setContainerID(blockID.getContainerID()) .setDatanodeUuid(id).setWriteChunk(writeChunkRequest); String encodedToken = getEncodedBlockToken(new Text(blockID. getContainerBlockID().toString())); @@ -336,13 +319,12 @@ public static XceiverClientReply writeChunkAsync( * @param client - client that communicates with the container. * @param blockID - ID of the block * @param data - Data to be written into the container. - * @param traceID - Trace ID for logging purpose. * @return container protocol writeSmallFile response * @throws IOException */ public static PutSmallFileResponseProto writeSmallFile( - XceiverClientSpi client, BlockID blockID, byte[] data, - String traceID) throws IOException { + XceiverClientSpi client, BlockID blockID, byte[] data) + throws IOException { BlockData containerBlockData = BlockData.newBuilder().setBlockID(blockID.getDatanodeBlockIDProtobuf()) @@ -375,7 +357,6 @@ public static PutSmallFileResponseProto writeSmallFile( ContainerCommandRequestProto.newBuilder() .setCmdType(Type.PutSmallFile) .setContainerID(blockID.getContainerID()) - .setTraceID(traceID) .setDatanodeUuid(id) .setPutSmallFile(putSmallFileRequest); String encodedToken = getEncodedBlockToken(new Text(blockID. @@ -384,8 +365,8 @@ public static PutSmallFileResponseProto writeSmallFile( builder.setEncodedToken(encodedToken); } ContainerCommandRequestProto request = builder.build(); - ContainerCommandResponseProto response = client.sendCommand(request); - validateContainerResponse(response); + ContainerCommandResponseProto response = + client.sendCommand(request, getValidatorList()); return response.getPutSmallFile(); } @@ -393,12 +374,11 @@ public static PutSmallFileResponseProto writeSmallFile( * createContainer call that creates a container on the datanode. * @param client - client * @param containerID - ID of container - * @param traceID - traceID * @param encodedToken - encodedToken if security is enabled * @throws IOException */ public static void createContainer(XceiverClientSpi client, long containerID, - String traceID, String encodedToken) throws IOException { + String encodedToken) throws IOException { ContainerProtos.CreateContainerRequestProto.Builder createRequest = ContainerProtos.CreateContainerRequestProto .newBuilder(); @@ -415,10 +395,7 @@ public static void createContainer(XceiverClientSpi client, long containerID, request.setContainerID(containerID); request.setCreateContainer(createRequest.build()); request.setDatanodeUuid(id); - request.setTraceID(traceID); - ContainerCommandResponseProto response = client.sendCommand( - request.build()); - validateContainerResponse(response); + client.sendCommand(request.build(), getValidatorList()); } /** @@ -426,12 +403,11 @@ public static void createContainer(XceiverClientSpi client, long containerID, * * @param client * @param force whether or not to forcibly delete the container. - * @param traceID * @param encodedToken - encodedToken if security is enabled * @throws IOException */ public static void deleteContainer(XceiverClientSpi client, long containerID, - boolean force, String traceID, String encodedToken) throws IOException { + boolean force, String encodedToken) throws IOException { ContainerProtos.DeleteContainerRequestProto.Builder deleteRequest = ContainerProtos.DeleteContainerRequestProto.newBuilder(); deleteRequest.setForceDelete(force); @@ -442,14 +418,11 @@ public static void deleteContainer(XceiverClientSpi client, long containerID, request.setCmdType(ContainerProtos.Type.DeleteContainer); request.setContainerID(containerID); request.setDeleteContainer(deleteRequest); - request.setTraceID(traceID); request.setDatanodeUuid(id); - if(encodedToken != null) { + if (encodedToken != null) { request.setEncodedToken(encodedToken); } - ContainerCommandResponseProto response = - client.sendCommand(request.build()); - validateContainerResponse(response); + client.sendCommand(request.build(), getValidatorList()); } /** @@ -457,12 +430,11 @@ public static void deleteContainer(XceiverClientSpi client, long containerID, * * @param client * @param containerID - * @param traceID * @param encodedToken - encodedToken if security is enabled * @throws IOException */ public static void closeContainer(XceiverClientSpi client, - long containerID, String traceID, String encodedToken) + long containerID, String encodedToken) throws IOException { String id = client.getPipeline().getFirstNode().getUuidString(); @@ -471,27 +443,23 @@ public static void closeContainer(XceiverClientSpi client, request.setCmdType(Type.CloseContainer); request.setContainerID(containerID); request.setCloseContainer(CloseContainerRequestProto.getDefaultInstance()); - request.setTraceID(traceID); request.setDatanodeUuid(id); if(encodedToken != null) { request.setEncodedToken(encodedToken); } - ContainerCommandResponseProto response = - client.sendCommand(request.build()); - validateContainerResponse(response); + client.sendCommand(request.build(), getValidatorList()); } /** * readContainer call that gets meta data from an existing container. * * @param client - client - * @param traceID - trace ID * @param encodedToken - encodedToken if security is enabled * @throws IOException */ public static ReadContainerResponseProto readContainer( - XceiverClientSpi client, long containerID, - String traceID, String encodedToken) throws IOException { + XceiverClientSpi client, long containerID, String encodedToken) + throws IOException { String id = client.getPipeline().getFirstNode().getUuidString(); ContainerCommandRequestProto.Builder request = @@ -500,13 +468,11 @@ public static ReadContainerResponseProto readContainer( request.setContainerID(containerID); request.setReadContainer(ReadContainerRequestProto.getDefaultInstance()); request.setDatanodeUuid(id); - request.setTraceID(traceID); if(encodedToken != null) { request.setEncodedToken(encodedToken); } ContainerCommandResponseProto response = - client.sendCommand(request.build()); - validateContainerResponse(response); + client.sendCommand(request.build(), getValidatorList()); return response.getReadContainer(); } @@ -516,12 +482,11 @@ public static ReadContainerResponseProto readContainer( * * @param client * @param blockID - ID of the block - * @param traceID - trace ID * @return GetSmallFileResponseProto * @throws IOException */ public static GetSmallFileResponseProto readSmallFile(XceiverClientSpi client, - BlockID blockID, String traceID) throws IOException { + BlockID blockID) throws IOException { GetBlockRequestProto.Builder getBlock = GetBlockRequestProto .newBuilder() .setBlockID(blockID.getDatanodeBlockIDProtobuf()); @@ -529,13 +494,12 @@ public static GetSmallFileResponseProto readSmallFile(XceiverClientSpi client, GetSmallFileRequestProto .newBuilder().setBlock(getBlock) .build(); - String id = client.getPipeline().getFirstNode().getUuidString(); + String id = client.getPipeline().getClosestNode().getUuidString(); ContainerCommandRequestProto.Builder builder = ContainerCommandRequestProto .newBuilder() .setCmdType(Type.GetSmallFile) .setContainerID(blockID.getContainerID()) - .setTraceID(traceID) .setDatanodeUuid(id) .setGetSmallFile(getSmallFileRequest); String encodedToken = getEncodedBlockToken(new Text(blockID. @@ -544,9 +508,8 @@ public static GetSmallFileResponseProto readSmallFile(XceiverClientSpi client, builder.setEncodedToken(encodedToken); } ContainerCommandRequestProto request = builder.build(); - ContainerCommandResponseProto response = client.sendCommand(request); - validateContainerResponse(response); - + ContainerCommandResponseProto response = + client.sendCommand(request, getValidatorList()); return response.getGetSmallFile(); } @@ -598,4 +561,13 @@ private static Text getService(DatanodeBlockID blockId) { .append(blockId.getLocalID()) .toString()); } + + public static List getValidatorList() { + List validators = new ArrayList<>(1); + CheckedBiFunction + validator = (request, response) -> validateContainerResponse(response); + validators.add(validator); + return validators; + } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/BlockTokenVerifier.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/BlockTokenVerifier.java index 2742acec5c30e..e94808ac9d7d1 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/BlockTokenVerifier.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/BlockTokenVerifier.java @@ -68,7 +68,9 @@ public UserGroupInformation verify(String user, String tokenStr) OzoneBlockTokenIdentifier tokenId = new OzoneBlockTokenIdentifier(); try { token.decodeFromUrlString(tokenStr); - LOGGER.debug("Verifying token:{} for user:{} ", token, user); + if (LOGGER.isDebugEnabled()) { + LOGGER.debug("Verifying token:{} for user:{} ", token, user); + } ByteArrayInputStream buf = new ByteArrayInputStream( token.getIdentifier()); DataInputStream in = new DataInputStream(buf); diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/OzoneBlockTokenSelector.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/OzoneBlockTokenSelector.java index 83797c3c96072..9acc75ae17078 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/OzoneBlockTokenSelector.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/OzoneBlockTokenSelector.java @@ -47,7 +47,9 @@ public Token selectToken(Text service, for (Token token : tokens) { if (OzoneBlockTokenIdentifier.KIND_NAME.equals(token.getKind()) && token.getService().equals(service)) { - LOG.trace("Getting token for service:{}", service); + if (LOG.isTraceEnabled()) { + LOG.trace("Getting token for service:{}", service); + } return (Token) token; } } @@ -66,7 +68,9 @@ public static Token selectBlockToken(Text service, for (Token token : tokens) { if (OzoneBlockTokenIdentifier.KIND_NAME.equals(token.getKind()) && token.getService().equals(service)) { - LOG.trace("Getting token for service:{}", service); + if (LOG.isTraceEnabled()) { + LOG.trace("Getting token for service:{}", service); + } return (Token) token; } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java index 0e4204f9c44a1..8aaba5df999cc 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java @@ -27,7 +27,6 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import java.io.File; import java.nio.file.Path; import java.nio.file.Paths; import java.security.Provider; @@ -46,14 +45,6 @@ import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_GRPC_TLS_PROVIDER_DEFAULT; import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_GRPC_TLS_TEST_CERT; import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_GRPC_TLS_TEST_CERT_DEFAULT; -import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_GRPC_MUTUAL_TLS_REQUIRED; -import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_GRPC_MUTUAL_TLS_REQUIRED_DEFAULT; -import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_TRUST_STORE_FILE_NAME; -import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_TRUST_STORE_FILE_NAME_DEFAULT; -import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_CLIENT_CERTIFICATE_CHAIN_FILE_NAME; -import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_CLIENT_CERTIFICATE_CHAIN_FILE_NAME_DEFAULT; -import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_SERVER_CERTIFICATE_CHAIN_FILE_NAME; -import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_SERVER_CERTIFICATE_CHAIN_FILE_NAME_DEFAULT; import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_KEY_ALGORITHM; import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_KEY_DIR_NAME; @@ -105,12 +96,8 @@ public class SecurityConfig { private final String certificateFileName; private final boolean grpcTlsEnabled; private boolean grpcTlsUseTestCert; - private String trustStoreFileName; - private String serverCertChainFileName; - private String clientCertChainFileName; private final Duration defaultCertDuration; private final boolean isSecurityEnabled; - private boolean grpcMutualTlsRequired; /** * Constructs a SecurityConfig. @@ -157,20 +144,6 @@ public SecurityConfig(Configuration configuration) { HDDS_GRPC_TLS_ENABLED_DEFAULT); if (grpcTlsEnabled) { - this.grpcMutualTlsRequired = configuration.getBoolean( - HDDS_GRPC_MUTUAL_TLS_REQUIRED, HDDS_GRPC_MUTUAL_TLS_REQUIRED_DEFAULT); - - this.trustStoreFileName = this.configuration.get( - HDDS_TRUST_STORE_FILE_NAME, HDDS_TRUST_STORE_FILE_NAME_DEFAULT); - - this.clientCertChainFileName = this.configuration.get( - HDDS_CLIENT_CERTIFICATE_CHAIN_FILE_NAME, - HDDS_CLIENT_CERTIFICATE_CHAIN_FILE_NAME_DEFAULT); - - this.serverCertChainFileName = this.configuration.get( - HDDS_SERVER_CERTIFICATE_CHAIN_FILE_NAME, - HDDS_SERVER_CERTIFICATE_CHAIN_FILE_NAME_DEFAULT); - this.grpcTlsUseTestCert = this.configuration.getBoolean( HDDS_GRPC_TLS_TEST_CERT, HDDS_GRPC_TLS_TEST_CERT_DEFAULT); } @@ -246,23 +219,12 @@ public String getPrivateKeyFileName() { return privateKeyFileName; } - /** - * Returns the File path to where keys are stored. - * - * @return path Key location. - */ - public Path getKeyLocation() { - Preconditions.checkNotNull(this.metadatDir, "Metadata directory can't be" - + " null. Please check configs."); - return Paths.get(metadatDir, keyDir); - } - /** * Returns the File path to where keys are stored with an additional component * name inserted in between. * * @param component - Component Name - String. - * @return Path location. + * @return Path Key location. */ public Path getKeyLocation(String component) { Preconditions.checkNotNull(this.metadatDir, "Metadata directory can't be" @@ -271,18 +233,8 @@ public Path getKeyLocation(String component) { } /** - * Returns the File path to where keys are stored. - * - * @return path Key location. - */ - public Path getCertificateLocation() { - Preconditions.checkNotNull(this.metadatDir, "Metadata directory can't be" - + " null. Please check configs."); - return Paths.get(metadatDir, certificateDir); - } - - /** - * Returns the File path to where keys are stored with an addition component + * Returns the File path to where certificates are stored with an addition + * component * name inserted in between. * * @param component - Component Name - String. @@ -371,61 +323,6 @@ public boolean isGrpcTlsEnabled() { return this.grpcTlsEnabled; } - /** - * Returns true if TLS mutual authentication is enabled for gRPC services. - * @return true if TLS is enabled for gRPC services. - */ - public boolean isGrpcMutualTlsRequired() { - return this.grpcMutualTlsRequired; - } - - /** - * Returns the TLS-enabled gRPC client private key file(Only needed for mutual - * authentication). - * @return the TLS-enabled gRPC client private key file. - */ - public File getClientPrivateKeyFile() { - return Paths.get(getKeyLocation().toString(), - "client." + privateKeyFileName).toFile(); - } - - /** - * Returns the TLS-enabled gRPC server private key file. - * @return the TLS-enabled gRPC server private key file. - */ - public File getServerPrivateKeyFile() { - return Paths.get(getKeyLocation().toString(), - "server." + privateKeyFileName).toFile(); - } - - /** - * Get the trusted CA certificate file. (CA certificate) - * @return the trusted CA certificate. - */ - public File getTrustStoreFile() { - return Paths.get(getKeyLocation().toString(), trustStoreFileName). - toFile(); - } - - /** - * Get the TLS-enabled gRPC Client certificate chain file (only needed for - * mutual authentication). - * @return the TLS-enabled gRPC Server certificate chain file. - */ - public File getClientCertChainFile() { - return Paths.get(getKeyLocation().toString(), clientCertChainFileName). - toFile(); - } - - /** - * Get the TLS-enabled gRPC Server certificate chain file. - * @return the TLS-enabled gRPC Server certificate chain file. - */ - public File getServerCertChainFile() { - return Paths.get(getKeyLocation().toString(), serverCertChainFileName). - toFile(); - } - /** * Get the gRPC TLS provider. * @return the gRPC TLS Provider. @@ -437,7 +334,7 @@ public SslProvider getGrpcSslProvider() { /** * Return true if using test certificates with authority as localhost. - * This should be used only for unit test where certifiates are generated + * This should be used only for unit test where certificates are generated * by openssl with localhost as DN and should never use for production as it * will bypass the hostname/ip matching verification. * @return true if using test certificates. @@ -464,7 +361,7 @@ private Provider initSecurityProvider(String providerName) { /** * Returns max date for which S3 tokens will be valid. - * */ + */ public long getS3TokenMaxDate() { return getConfiguration().getTimeDuration( OzoneConfigKeys.OZONE_S3_TOKEN_MAX_LIFETIME_KEY, diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/PKIProfiles/DefaultProfile.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/PKIProfiles/DefaultProfile.java index 13319d16572bc..5fdb6f7d96696 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/PKIProfiles/DefaultProfile.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/PKIProfiles/DefaultProfile.java @@ -236,7 +236,9 @@ public boolean validateGeneralName(int type, String value) { try { final InetAddress byAddress = InetAddress.getByAddress( Hex.decodeHex(value.substring(1))); - LOG.debug("Host Name/IP Address : {}", byAddress.toString()); + if (LOG.isDebugEnabled()) { + LOG.debug("Host Name/IP Address : {}", byAddress.toString()); + } return true; } catch (UnknownHostException | DecoderException e) { return false; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/CertificateClient.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/CertificateClient.java index 480758b9ee1b6..34b4930fa7d37 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/CertificateClient.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/CertificateClient.java @@ -69,6 +69,12 @@ X509Certificate getCertificate(String certSerialId) */ X509Certificate getCertificate(); + /** + * Return the latest CA certificate known to the client. + * @return latest ca certificate known to the client. + */ + X509Certificate getCACertificate(); + /** * Verifies if this certificate is part of a trusted chain. * @param certificate - certificate. @@ -141,6 +147,19 @@ boolean verifySignature(byte[] data, byte[] signature, void storeCertificate(String pemEncodedCert, boolean force) throws CertificateException; + /** + * Stores the Certificate for this client. Don't use this api to add + * trusted certificates of others. + * + * @param pemEncodedCert - pem encoded X509 Certificate + * @param force - override any existing file + * @param caCert - Is CA certificate. + * @throws CertificateException - on Error. + * + */ + void storeCertificate(String pemEncodedCert, boolean force, boolean caCert) + throws CertificateException; + /** * Stores the trusted chain of certificates. * diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DNCertificateClient.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DNCertificateClient.java index 7790d0429197b..76986586d344c 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DNCertificateClient.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DNCertificateClient.java @@ -25,6 +25,7 @@ import org.slf4j.LoggerFactory; import org.apache.hadoop.hdds.security.x509.SecurityConfig; + /** * Certificate client for DataNodes. */ @@ -32,13 +33,16 @@ public class DNCertificateClient extends DefaultCertificateClient { private static final Logger LOG = LoggerFactory.getLogger(DNCertificateClient.class); + + public static final String COMPONENT_NAME = "dn"; + public DNCertificateClient(SecurityConfig securityConfig, String certSerialId) { - super(securityConfig, LOG, certSerialId); + super(securityConfig, LOG, certSerialId, COMPONENT_NAME); } public DNCertificateClient(SecurityConfig securityConfig) { - super(securityConfig, LOG, null); + super(securityConfig, LOG, null, COMPONENT_NAME); } /** diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DefaultCertificateClient.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DefaultCertificateClient.java index 26be970043414..ff99e080c49eb 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DefaultCertificateClient.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DefaultCertificateClient.java @@ -20,7 +20,9 @@ package org.apache.hadoop.hdds.security.x509.certificate.client; import com.google.common.base.Preconditions; +import org.apache.commons.io.FilenameUtils; import org.apache.commons.lang3.RandomStringUtils; +import org.apache.commons.lang3.math.NumberUtils; import org.apache.commons.validator.routines.DomainValidator; import org.apache.hadoop.hdds.security.x509.SecurityConfig; import org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec; @@ -80,6 +82,8 @@ public abstract class DefaultCertificateClient implements CertificateClient { private static final String CERT_FILE_NAME_FORMAT = "%s.crt"; + private static final String CA_CERT_PREFIX = "CA-"; + private static final int CA_CERT_PREFIX_LEN = 3; private final Logger logger; private final SecurityConfig securityConfig; private final KeyCodec keyCodec; @@ -88,16 +92,18 @@ public abstract class DefaultCertificateClient implements CertificateClient { private X509Certificate x509Certificate; private Map certificateMap; private String certSerialId; - + private String caCertId; + private String component; DefaultCertificateClient(SecurityConfig securityConfig, Logger log, - String certSerialId) { + String certSerialId, String component) { Objects.requireNonNull(securityConfig); this.securityConfig = securityConfig; - keyCodec = new KeyCodec(securityConfig); + keyCodec = new KeyCodec(securityConfig, component); this.logger = log; this.certificateMap = new ConcurrentHashMap<>(); this.certSerialId = certSerialId; + this.component = component; loadAllCertificates(); } @@ -107,7 +113,7 @@ public abstract class DefaultCertificateClient implements CertificateClient { * */ private void loadAllCertificates() { // See if certs directory exists in file system. - Path certPath = securityConfig.getCertificateLocation(); + Path certPath = securityConfig.getCertificateLocation(component); if (Files.exists(certPath) && Files.isDirectory(certPath)) { getLogger().info("Loading certificate from location:{}.", certPath); @@ -115,7 +121,8 @@ private void loadAllCertificates() { if (certFiles != null) { CertificateCodec certificateCodec = - new CertificateCodec(securityConfig); + new CertificateCodec(securityConfig, component); + long latestCaCertSerailId = -1L; for (File file : certFiles) { if (file.isFile()) { try { @@ -129,6 +136,15 @@ private void loadAllCertificates() { } certificateMap.putIfAbsent(cert.getSerialNumber().toString(), cert); + if (file.getName().startsWith(CA_CERT_PREFIX)) { + String certFileName = FilenameUtils.getBaseName( + file.getName()); + long tmpCaCertSerailId = NumberUtils.toLong( + certFileName.substring(CA_CERT_PREFIX_LEN)); + if (tmpCaCertSerailId > latestCaCertSerailId) { + latestCaCertSerailId = tmpCaCertSerailId; + } + } getLogger().info("Added certificate from file:{}.", file.getAbsolutePath()); } else { @@ -141,6 +157,9 @@ private void loadAllCertificates() { } } } + if (latestCaCertSerailId != -1) { + caCertId = Long.toString(latestCaCertSerailId); + } } } } @@ -157,7 +176,7 @@ public PrivateKey getPrivateKey() { return privateKey; } - Path keyPath = securityConfig.getKeyLocation(); + Path keyPath = securityConfig.getKeyLocation(component); if (OzoneSecurityUtil.checkIfFileExist(keyPath, securityConfig.getPrivateKeyFileName())) { try { @@ -181,7 +200,7 @@ public PublicKey getPublicKey() { return publicKey; } - Path keyPath = securityConfig.getKeyLocation(); + Path keyPath = securityConfig.getKeyLocation(component); if (OzoneSecurityUtil.checkIfFileExist(keyPath, securityConfig.getPublicKeyFileName())) { try { @@ -218,6 +237,18 @@ public X509Certificate getCertificate() { return x509Certificate; } + /** + * Return the latest CA certificate known to the client. + * @return latest ca certificate known to the client. + */ + @Override + public X509Certificate getCACertificate() { + if (caCertId != null) { + return certificateMap.get(caCertId); + } + return null; + } + /** * Returns the certificate with the specified certificate serial id if it * exists else try to get it from SCM. @@ -452,23 +483,45 @@ public X509Certificate queryCertificate(String query) { * Stores the Certificate for this client. Don't use this api to add trusted * certificates of others. * - * @param pemEncodedCert - pem encoded X509 Certificate - * @param force - override any existing file + * @param pemEncodedCert - pem encoded X509 Certificate + * @param force - override any existing file * @throws CertificateException - on Error. * */ @Override public void storeCertificate(String pemEncodedCert, boolean force) throws CertificateException { - CertificateCodec certificateCodec = new CertificateCodec(securityConfig); + this.storeCertificate(pemEncodedCert, force, false); + } + + /** + * Stores the Certificate for this client. Don't use this api to add trusted + * certificates of others. + * + * @param pemEncodedCert - pem encoded X509 Certificate + * @param force - override any existing file + * @param caCert - Is CA certificate. + * @throws CertificateException - on Error. + * + */ + @Override + public void storeCertificate(String pemEncodedCert, boolean force, + boolean caCert) throws CertificateException { + CertificateCodec certificateCodec = new CertificateCodec(securityConfig, + component); try { - Path basePath = securityConfig.getCertificateLocation(); + Path basePath = securityConfig.getCertificateLocation(component); X509Certificate cert = CertificateCodec.getX509Certificate(pemEncodedCert); String certName = String.format(CERT_FILE_NAME_FORMAT, cert.getSerialNumber().toString()); + if(caCert) { + certName = CA_CERT_PREFIX + certName; + caCertId = cert.getSerialNumber().toString(); + } + certificateCodec.writeCertificate(basePath, certName, pemEncodedCert, force); certificateMap.putIfAbsent(cert.getSerialNumber().toString(), cert); @@ -717,7 +770,7 @@ protected boolean validateKeyPair(PublicKey pubKey) * location. * */ protected void bootstrapClientKeys() throws CertificateException { - Path keyPath = securityConfig.getKeyLocation(); + Path keyPath = securityConfig.getKeyLocation(component); if (Files.notExists(keyPath)) { try { Files.createDirectories(keyPath); diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/OMCertificateClient.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/OMCertificateClient.java index b1f7504026648..cb3ce7536e1ef 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/OMCertificateClient.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/OMCertificateClient.java @@ -39,13 +39,15 @@ public class OMCertificateClient extends DefaultCertificateClient { private static final Logger LOG = LoggerFactory.getLogger(OMCertificateClient.class); + public static final String COMPONENT_NAME = "om"; + public OMCertificateClient(SecurityConfig securityConfig, String certSerialId) { - super(securityConfig, LOG, certSerialId); + super(securityConfig, LOG, certSerialId, COMPONENT_NAME); } public OMCertificateClient(SecurityConfig securityConfig) { - super(securityConfig, LOG, null); + super(securityConfig, LOG, null, COMPONENT_NAME); } protected InitResponse handleCase(InitCase init) throws diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/utils/CertificateCodec.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/utils/CertificateCodec.java index 90d5325fb5f93..2c8721b199bd7 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/utils/CertificateCodec.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/utils/CertificateCodec.java @@ -19,9 +19,7 @@ package org.apache.hadoop.hdds.security.x509.certificate.utils; -import com.google.common.base.Preconditions; import org.apache.commons.io.IOUtils; -import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdds.security.exception.SCMSecurityException; import org.apache.hadoop.hdds.security.x509.SecurityConfig; import org.bouncycastle.cert.X509CertificateHolder; @@ -70,7 +68,7 @@ public class CertificateCodec { Stream.of(OWNER_READ, OWNER_WRITE, OWNER_EXECUTE) .collect(Collectors.toSet()); /** - * Creates an CertificateCodec. + * Creates a CertificateCodec with component name. * * @param config - Security Config. * @param component - Component String. @@ -80,27 +78,6 @@ public CertificateCodec(SecurityConfig config, String component) { this.location = securityConfig.getCertificateLocation(component); } - /** - * Creates an CertificateCodec. - * - * @param config - Security Config. - */ - public CertificateCodec(SecurityConfig config) { - this.securityConfig = config; - this.location = securityConfig.getCertificateLocation(); - } - - /** - * Creates an CertificateCodec. - * - * @param configuration - Configuration - */ - public CertificateCodec(Configuration configuration) { - Preconditions.checkNotNull(configuration, "Config cannot be null"); - this.securityConfig = new SecurityConfig(configuration); - this.location = securityConfig.getCertificateLocation(); - } - /** * Returns a X509 Certificate from the Certificate Holder. * diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyGenerator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyGenerator.java index ded50f9f653bf..640f5ca0b9462 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyGenerator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyGenerator.java @@ -108,8 +108,10 @@ public KeyPair generateKey(int size) throws */ public KeyPair generateKey(int size, String algorithm, String provider) throws NoSuchProviderException, NoSuchAlgorithmException { - LOG.debug("Generating key pair using size:{}, Algorithm:{}, Provider:{}", - size, algorithm, provider); + if (LOG.isDebugEnabled()) { + LOG.debug("Generating key pair using size:{}, Algorithm:{}, Provider:{}", + size, algorithm, provider); + } KeyPairGenerator generator = KeyPairGenerator .getInstance(algorithm, provider); generator.initialize(size); diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/KeyCodec.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/KeyCodec.java index a5ebdaefffc05..82873b06c7146 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/KeyCodec.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/KeyCodec.java @@ -22,7 +22,6 @@ import com.google.common.base.Preconditions; import org.apache.commons.io.FileUtils; import org.apache.commons.io.output.FileWriterWithEncoding; -import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdds.security.x509.SecurityConfig; import org.bouncycastle.util.io.pem.PemObject; import org.bouncycastle.util.io.pem.PemReader; @@ -76,7 +75,7 @@ public class KeyCodec { private Supplier isPosixFileSystem; /** - * Creates an KeyCodec. + * Creates a KeyCodec with component name. * * @param config - Security Config. * @param component - Component String. @@ -87,29 +86,6 @@ public KeyCodec(SecurityConfig config, String component) { this.location = securityConfig.getKeyLocation(component); } - /** - * Creates an KeyCodec. - * - * @param config - Security Config. - */ - public KeyCodec(SecurityConfig config) { - this.securityConfig = config; - isPosixFileSystem = KeyCodec::isPosix; - this.location = securityConfig.getKeyLocation(); - } - - /** - * Creates an HDDS Key Writer. - * - * @param configuration - Configuration - */ - public KeyCodec(Configuration configuration) { - Preconditions.checkNotNull(configuration, "Config cannot be null"); - this.securityConfig = new SecurityConfig(configuration); - isPosixFileSystem = KeyCodec::isPosix; - this.location = securityConfig.getKeyLocation(); - } - /** * Checks if File System supports posix style security permissions. * diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/tracing/StringCodec.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/tracing/StringCodec.java index 03365cf54ba28..56d59ea6f1a32 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/tracing/StringCodec.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/tracing/StringCodec.java @@ -45,7 +45,9 @@ public JaegerSpanContext extract(StringBuilder s) { if (value != null && !value.equals("")) { String[] parts = value.split(":"); if (parts.length != 4) { - LOG.trace("MalformedTracerStateString: {}", value); + if (LOG.isDebugEnabled()) { + LOG.debug("MalformedTracerStateString: {}", value); + } throw new MalformedTracerStateStringException(value); } else { String traceId = parts[0]; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundService.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java similarity index 95% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundService.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java index 5718008b41510..ca8d87053f7fc 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundService.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java @@ -15,7 +15,7 @@ * the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import com.google.common.annotations.VisibleForTesting; import com.google.common.collect.Lists; @@ -102,15 +102,18 @@ public void start() { public class PeriodicalTask implements Runnable { @Override public synchronized void run() { - LOG.debug("Running background service : {}", serviceName); + if (LOG.isDebugEnabled()) { + LOG.debug("Running background service : {}", serviceName); + } BackgroundTaskQueue tasks = getTasks(); if (tasks.isEmpty()) { // No task found, or some problems to init tasks // return and retry in next interval. return; } - - LOG.debug("Number of background tasks to execute : {}", tasks.size()); + if (LOG.isDebugEnabled()) { + LOG.debug("Number of background tasks to execute : {}", tasks.size()); + } CompletionService taskCompletionService = new ExecutorCompletionService<>(exec); diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundTask.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundTask.java similarity index 96% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundTask.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundTask.java index 47e8ebc98fd7e..d5ad2a394dd3a 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundTask.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundTask.java @@ -15,7 +15,7 @@ * the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import java.util.concurrent.Callable; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundTaskQueue.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundTaskQueue.java similarity index 97% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundTaskQueue.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundTaskQueue.java index b56ef0c804bad..005d14b8e3cc6 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundTaskQueue.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundTaskQueue.java @@ -15,7 +15,7 @@ * the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import java.util.PriorityQueue; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundTaskResult.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundTaskResult.java similarity index 97% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundTaskResult.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundTaskResult.java index 198300fb5f504..be8032b06a3e2 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundTaskResult.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundTaskResult.java @@ -15,7 +15,7 @@ * the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; /** * Result of a {@link BackgroundTask}. diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BatchOperation.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BatchOperation.java new file mode 100644 index 0000000000000..377c7f6a1a868 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BatchOperation.java @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.utils; + +import com.google.common.collect.Lists; + +import java.util.List; + +/** + * An utility class to store a batch of DB write operations. + */ +public class BatchOperation { + + /** + * Enum for write operations. + */ + public enum Operation { + DELETE, PUT + } + + private List operations = + Lists.newArrayList(); + + /** + * Add a PUT operation into the batch. + */ + public void put(byte[] key, byte[] value) { + operations.add(new SingleOperation(Operation.PUT, key, value)); + } + + /** + * Add a DELETE operation into the batch. + */ + public void delete(byte[] key) { + operations.add(new SingleOperation(Operation.DELETE, key, null)); + + } + + public List getOperations() { + return operations; + } + + /** + * A SingleOperation represents a PUT or DELETE operation + * and the data the operation needs to manipulates. + */ + public static class SingleOperation { + + private Operation opt; + private byte[] key; + private byte[] value; + + public SingleOperation(Operation opt, byte[] key, byte[] value) { + this.opt = opt; + if (key == null) { + throw new IllegalArgumentException("key cannot be null"); + } + this.key = key.clone(); + this.value = value == null ? null : value.clone(); + } + + public Operation getOpt() { + return opt; + } + + public byte[] getKey() { + return key.clone(); + } + + public byte[] getValue() { + return value == null ? null : value.clone(); + } + } +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/EntryConsumer.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/EntryConsumer.java similarity index 97% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/EntryConsumer.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/EntryConsumer.java index c40739861183a..dc08c2bd63c8a 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/EntryConsumer.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/EntryConsumer.java @@ -16,7 +16,7 @@ * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import java.io.IOException; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/HddsVersionInfo.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/HddsVersionInfo.java similarity index 91% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/HddsVersionInfo.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/HddsVersionInfo.java index 13c62b71a2fa8..6a372d123712c 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/HddsVersionInfo.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/HddsVersionInfo.java @@ -16,7 +16,7 @@ * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; @@ -50,7 +50,9 @@ public static void main(String[] args) { "Compiled with protoc " + HDDS_VERSION_INFO.getProtocVersion()); System.out.println( "From source with checksum " + HDDS_VERSION_INFO.getSrcChecksum()); - LOG.debug("This command was run using " + - ClassUtil.findContainingJar(HddsVersionInfo.class)); + if (LOG.isDebugEnabled()) { + LOG.debug("This command was run using " + + ClassUtil.findContainingJar(HddsVersionInfo.class)); + } } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStore.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStore.java similarity index 93% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStore.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStore.java index ed116a381c32f..0598987f9b50b 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStore.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStore.java @@ -16,10 +16,10 @@ * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import org.apache.commons.lang3.tuple.ImmutablePair; -import org.apache.hadoop.utils.MetadataKeyFilters.MetadataKeyFilter; +import org.apache.hadoop.hdds.utils.MetadataKeyFilters.MetadataKeyFilter; import org.fusesource.leveldbjni.JniDBFactory; import org.iq80.leveldb.DB; import org.iq80.leveldb.DBIterator; @@ -77,7 +77,9 @@ public LevelDBStore(File dbPath, Options options) private void openDB(File dbPath, Options options) throws IOException { if (dbPath.getParentFile().mkdirs()) { - LOG.debug("Db path {} created.", dbPath.getParentFile()); + if (LOG.isDebugEnabled()) { + LOG.debug("Db path {} created.", dbPath.getParentFile()); + } } db = JniDBFactory.factory.open(dbPath, options); if (LOG.isDebugEnabled()) { @@ -240,6 +242,12 @@ public void compactDB() throws IOException { } } + @Override + public void flushDB(boolean sync) { + // TODO: Implement flush for level db + // do nothing + } + @Override public void writeBatch(BatchOperation operation) throws IOException { List operations = @@ -364,17 +372,21 @@ private List> getRangeKVs(byte[] startKey, int scanned = filter.getKeysScannedNum(); int hinted = filter.getKeysHintedNum(); if (scanned > 0 || hinted > 0) { - LOG.debug( - "getRangeKVs ({}) numOfKeysScanned={}, numOfKeysHinted={}", - filter.getClass().getSimpleName(), filter.getKeysScannedNum(), - filter.getKeysHintedNum()); + if (LOG.isDebugEnabled()) { + LOG.debug( + "getRangeKVs ({}) numOfKeysScanned={}, numOfKeysHinted={}", + filter.getClass().getSimpleName(), + filter.getKeysScannedNum(), filter.getKeysHintedNum()); + } } } } long end = System.currentTimeMillis(); long timeConsumed = end - start; - LOG.debug("Time consumed for getRangeKVs() is {}ms," - + " result length is {}.", timeConsumed, result.size()); + if (LOG.isDebugEnabled()) { + LOG.debug("Time consumed for getRangeKVs() is {}ms," + + " result length is {}.", timeConsumed, result.size()); + } } } return result; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStoreIterator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStoreIterator.java similarity index 85% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStoreIterator.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStoreIterator.java index cd07b645eaac4..f5b6769b70de3 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStoreIterator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStoreIterator.java @@ -16,19 +16,17 @@ * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import org.iq80.leveldb.DBIterator; import java.util.Map; import java.util.NoSuchElementException; -import org.apache.hadoop.utils.MetadataStore.KeyValue; - - /** * LevelDB store iterator. */ -public class LevelDBStoreIterator implements MetaStoreIterator { +public class LevelDBStoreIterator + implements MetaStoreIterator { private DBIterator levelDBIterator; @@ -44,10 +42,10 @@ public boolean hasNext() { } @Override - public KeyValue next() { + public MetadataStore.KeyValue next() { if(levelDBIterator.hasNext()) { Map.Entry entry = levelDBIterator.next(); - return KeyValue.create(entry.getKey(), entry.getValue()); + return MetadataStore.KeyValue.create(entry.getKey(), entry.getValue()); } throw new NoSuchElementException("LevelDB Store has no more elements"); } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetaStoreIterator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetaStoreIterator.java similarity index 96% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetaStoreIterator.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetaStoreIterator.java index 52d0a3efd3bf5..2a33de712ea09 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetaStoreIterator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetaStoreIterator.java @@ -16,7 +16,7 @@ * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import java.util.Iterator; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetadataKeyFilters.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetadataKeyFilters.java similarity index 99% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetadataKeyFilters.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetadataKeyFilters.java index 04c87aee2586a..a88ce475bab59 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetadataKeyFilters.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetadataKeyFilters.java @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import com.google.common.base.Preconditions; import com.google.common.base.Strings; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetadataStore.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetadataStore.java similarity index 96% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetadataStore.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetadataStore.java index 7d3bc6ba9a1a4..f05e6d2d275aa 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetadataStore.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetadataStore.java @@ -16,11 +16,11 @@ * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import org.apache.commons.lang3.tuple.ImmutablePair; import org.apache.hadoop.classification.InterfaceStability; -import org.apache.hadoop.utils.MetadataKeyFilters.MetadataKeyFilter; +import org.apache.hadoop.hdds.utils.MetadataKeyFilters.MetadataKeyFilter; import java.io.Closeable; import java.io.IOException; @@ -131,6 +131,12 @@ List> getSequentialRangeKVs(byte[] startKey, */ void compactDB() throws IOException; + /** + * Flush the outstanding I/O operations of the DB. + * @param sync if true will sync the outstanding I/Os to the disk. + */ + void flushDB(boolean sync) throws IOException; + /** * Destroy the content of the specified database, * a destroyed database will not be able to load again. diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetadataStoreBuilder.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetadataStoreBuilder.java similarity index 99% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetadataStoreBuilder.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetadataStoreBuilder.java index 9df4206bd245c..85bb6aa4bffc5 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetadataStoreBuilder.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/MetadataStoreBuilder.java @@ -16,7 +16,7 @@ * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import java.io.File; import java.io.IOException; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RetriableTask.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RetriableTask.java similarity index 98% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RetriableTask.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RetriableTask.java index a847ae0f081c9..a3ee1fd51bd85 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RetriableTask.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RetriableTask.java @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import org.apache.hadoop.io.retry.RetryPolicy; import org.apache.hadoop.util.ThreadUtil; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStore.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java similarity index 96% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStore.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java index 0ca99b68d394f..7dd1bde1b7795 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStore.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java @@ -16,7 +16,7 @@ * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import com.google.common.base.Preconditions; import org.apache.commons.io.FileUtils; @@ -75,7 +75,8 @@ public RocksDBStore(File dbFile, Options options) jmxProperties.put("dbName", dbFile.getName()); statMBeanName = HddsUtils.registerWithJmxProperties( "Ozone", "RocksDbStore", jmxProperties, - new RocksDBStoreMBean(dbOptions.statistics())); + RocksDBStoreMBean.create(dbOptions.statistics(), + dbFile.getName())); if (statMBeanName == null) { LOG.warn("jmx registration failed during RocksDB init, db path :{}", dbFile.getAbsolutePath()); @@ -278,6 +279,19 @@ public void compactDB() throws IOException { } } + @Override + public void flushDB(boolean sync) throws IOException { + if (db != null) { + try { + // for RocksDB it is sufficient to flush the WAL as entire db can + // be reconstructed using it. + db.flushWal(sync); + } catch (RocksDBException e) { + throw toIOException("Failed to flush db", e); + } + } + } + private void deleteQuietly(File fileOrDir) { if (fileOrDir != null && fileOrDir.exists()) { try { diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreIterator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStoreIterator.java similarity index 83% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreIterator.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStoreIterator.java index 6e9b6958da179..e39ec5774580b 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreIterator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStoreIterator.java @@ -17,18 +17,17 @@ */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import org.rocksdb.RocksIterator; import java.util.NoSuchElementException; -import org.apache.hadoop.utils.MetadataStore.KeyValue; - /** * RocksDB store iterator. */ -public class RocksDBStoreIterator implements MetaStoreIterator { +public class RocksDBStoreIterator + implements MetaStoreIterator { private RocksIterator rocksDBIterator; @@ -43,10 +42,11 @@ public boolean hasNext() { } @Override - public KeyValue next() { + public MetadataStore.KeyValue next() { if (rocksDBIterator.isValid()) { - KeyValue value = KeyValue.create(rocksDBIterator.key(), rocksDBIterator - .value()); + MetadataStore.KeyValue value = + MetadataStore.KeyValue.create(rocksDBIterator.key(), rocksDBIterator + .value()); rocksDBIterator.next(); return value; } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStoreMBean.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStoreMBean.java new file mode 100644 index 0000000000000..60d4db880c4ab --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStoreMBean.java @@ -0,0 +1,219 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.utils; + +import org.apache.hadoop.metrics2.MetricsCollector; +import org.apache.hadoop.metrics2.MetricsRecordBuilder; +import org.apache.hadoop.metrics2.MetricsSource; +import org.apache.hadoop.metrics2.MetricsSystem; +import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; +import org.apache.hadoop.metrics2.lib.Interns; +import org.rocksdb.HistogramData; +import org.rocksdb.HistogramType; +import org.rocksdb.Statistics; +import org.rocksdb.TickerType; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.management.Attribute; +import javax.management.AttributeList; +import javax.management.AttributeNotFoundException; +import javax.management.DynamicMBean; +import javax.management.InvalidAttributeValueException; +import javax.management.MBeanAttributeInfo; +import javax.management.MBeanException; +import javax.management.MBeanInfo; +import javax.management.ReflectionException; +import java.lang.reflect.Method; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +/** + * Adapter JMX bean to publish all the Rocksdb metrics. + */ +public class RocksDBStoreMBean implements DynamicMBean, MetricsSource { + + private Statistics statistics; + + private Set histogramAttributes = new HashSet<>(); + + private String contextName; + + private static final Logger LOG = + LoggerFactory.getLogger(RocksDBStoreMBean.class); + + public final static String ROCKSDB_CONTEXT_PREFIX = "Rocksdb_"; + + public RocksDBStoreMBean(Statistics statistics, String dbName) { + this.contextName = ROCKSDB_CONTEXT_PREFIX + dbName; + this.statistics = statistics; + histogramAttributes.add("Average"); + histogramAttributes.add("Median"); + histogramAttributes.add("Percentile95"); + histogramAttributes.add("Percentile99"); + histogramAttributes.add("StandardDeviation"); + } + + public static RocksDBStoreMBean create(Statistics statistics, + String contextName) { + + RocksDBStoreMBean rocksDBStoreMBean = new RocksDBStoreMBean( + statistics, contextName); + MetricsSystem ms = DefaultMetricsSystem.instance(); + MetricsSource metricsSource = ms.getSource(rocksDBStoreMBean.contextName); + if (metricsSource != null) { + return (RocksDBStoreMBean)metricsSource; + } else { + return ms.register(rocksDBStoreMBean.contextName, + "RocksDB Metrics", + rocksDBStoreMBean); + } + } + + @Override + public Object getAttribute(String attribute) + throws AttributeNotFoundException, MBeanException, ReflectionException { + for (String histogramAttribute : histogramAttributes) { + if (attribute.endsWith("_" + histogramAttribute.toUpperCase())) { + String keyName = attribute + .substring(0, attribute.length() - histogramAttribute.length() - 1); + try { + HistogramData histogram = + statistics.getHistogramData(HistogramType.valueOf(keyName)); + try { + Method method = + HistogramData.class.getMethod("get" + histogramAttribute); + return method.invoke(histogram); + } catch (Exception e) { + throw new ReflectionException(e, + "Can't read attribute " + attribute); + } + } catch (IllegalArgumentException exception) { + throw new AttributeNotFoundException( + "No such attribute in RocksDB stats: " + attribute); + } + } + } + try { + return statistics.getTickerCount(TickerType.valueOf(attribute)); + } catch (IllegalArgumentException ex) { + throw new AttributeNotFoundException( + "No such attribute in RocksDB stats: " + attribute); + } + } + + @Override + public void setAttribute(Attribute attribute) + throws AttributeNotFoundException, InvalidAttributeValueException, + MBeanException, ReflectionException { + + } + + @Override + public AttributeList getAttributes(String[] attributes) { + AttributeList result = new AttributeList(); + for (String attributeName : attributes) { + try { + Object value = getAttribute(attributeName); + result.add(value); + } catch (Exception e) { + //TODO + } + } + return result; + } + + @Override + public AttributeList setAttributes(AttributeList attributes) { + return null; + } + + @Override + public Object invoke(String actionName, Object[] params, String[] signature) + throws MBeanException, ReflectionException { + return null; + } + + @Override + public MBeanInfo getMBeanInfo() { + + List attributes = new ArrayList<>(); + for (TickerType tickerType : TickerType.values()) { + attributes.add(new MBeanAttributeInfo(tickerType.name(), "long", + "RocksDBStat: " + tickerType.name(), true, false, false)); + } + for (HistogramType histogramType : HistogramType.values()) { + for (String histogramAttribute : histogramAttributes) { + attributes.add(new MBeanAttributeInfo( + histogramType.name() + "_" + histogramAttribute.toUpperCase(), + "long", "RocksDBStat: " + histogramType.name(), true, false, + false)); + } + } + + return new MBeanInfo("", "RocksDBStat", + attributes.toArray(new MBeanAttributeInfo[0]), null, null, null); + + } + + @Override + public void getMetrics(MetricsCollector metricsCollector, boolean b) { + MetricsRecordBuilder rb = metricsCollector.addRecord(contextName); + getHistogramData(rb); + getTickerTypeData(rb); + } + + /** + * Collect all histogram metrics from RocksDB statistics. + * @param rb Metrics Record Builder. + */ + private void getHistogramData(MetricsRecordBuilder rb) { + for (HistogramType histogramType : HistogramType.values()) { + HistogramData histogram = + statistics.getHistogramData( + HistogramType.valueOf(histogramType.name())); + for (String histogramAttribute : histogramAttributes) { + try { + Method method = + HistogramData.class.getMethod("get" + histogramAttribute); + double metricValue = (double) method.invoke(histogram); + rb.addGauge(Interns.info(histogramType.name() + "_" + + histogramAttribute.toUpperCase(), "RocksDBStat"), + metricValue); + } catch (Exception e) { + LOG.error("Error reading histogram data {} ", e); + } + } + } + } + + /** + * Collect all Counter metrics from RocksDB statistics. + * @param rb Metrics Record Builder. + */ + private void getTickerTypeData(MetricsRecordBuilder rb) { + for (TickerType tickerType : TickerType.values()) { + rb.addCounter(Interns.info(tickerType.name(), "RocksDBStat"), + statistics.getTickerCount(tickerType)); + } + } + +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/Scheduler.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/Scheduler.java similarity index 98% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/Scheduler.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/Scheduler.java index 71eaf3397fc27..9edc104481016 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/Scheduler.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/Scheduler.java @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import org.apache.ratis.util.function.CheckedRunnable; import org.slf4j.Logger; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/UniqueId.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/UniqueId.java new file mode 100644 index 0000000000000..091453612577a --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/UniqueId.java @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.hdds.utils; + +import org.apache.hadoop.hdds.HddsUtils; + +/** + * This class uses system current time milliseconds to generate unique id. + */ +public final class UniqueId { + /* + * When we represent time in milliseconds using 'long' data type, + * the LSB bits are used. Currently we are only using 44 bits (LSB), + * 20 bits (MSB) are not used. + * We will exhaust this 44 bits only when we are in year 2525, + * until then we can safely use this 20 bits (MSB) for offset to generate + * unique id within millisecond. + * + * Year : Mon Dec 31 18:49:04 IST 2525 + * TimeInMillis: 17545641544247 + * Binary Representation: + * MSB (20 bits): 0000 0000 0000 0000 0000 + * LSB (44 bits): 1111 1111 0101 0010 1001 1011 1011 0100 1010 0011 0111 + * + * We have 20 bits to run counter, we should exclude the first bit (MSB) + * as we don't want to deal with negative values. + * To be on safer side we will use 'short' data type which is of length + * 16 bits and will give us 65,536 values for offset. + * + */ + + private static volatile short offset = 0; + + /** + * Private constructor so that no one can instantiate this class. + */ + private UniqueId() {} + + /** + * Calculate and returns next unique id based on System#currentTimeMillis. + * + * @return unique long value + */ + public static synchronized long next() { + long utcTime = HddsUtils.getUtcTime(); + if ((utcTime & 0xFFFF000000000000L) == 0) { + return utcTime << Short.SIZE | (offset++ & 0x0000FFFF); + } + throw new RuntimeException("Got invalid UTC time," + + " cannot generate unique Id. UTC Time: " + utcTime); + } +} \ No newline at end of file diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/VersionInfo.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/VersionInfo.java similarity index 98% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/VersionInfo.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/VersionInfo.java index 9b86934eadee7..ca9f859ccb403 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/VersionInfo.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/VersionInfo.java @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/BatchOperation.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/BatchOperation.java new file mode 100644 index 0000000000000..8ca5d188ebcd7 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/BatchOperation.java @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.apache.hadoop.hdds.utils.db; + +/** + * Class represents a batch operation, collects multiple db operation. + */ +public interface BatchOperation extends AutoCloseable { + + void close(); +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/ByteArrayKeyValue.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/ByteArrayKeyValue.java similarity index 94% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/ByteArrayKeyValue.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/ByteArrayKeyValue.java index ca5583c50cafe..7c602911fe490 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/ByteArrayKeyValue.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/ByteArrayKeyValue.java @@ -16,9 +16,9 @@ * limitations under the License. * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; -import org.apache.hadoop.utils.db.Table.KeyValue; +import org.apache.hadoop.hdds.utils.db.Table.KeyValue; /** * Key value for raw Table implementations. diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Codec.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/Codec.java similarity index 97% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Codec.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/Codec.java index 15720ea6614eb..36ece3ea774cc 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Codec.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/Codec.java @@ -16,7 +16,7 @@ * limitations under the License. * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.IOException; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/CodecRegistry.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/CodecRegistry.java similarity index 98% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/CodecRegistry.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/CodecRegistry.java index 104fd4b618e54..f92189aef5bac 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/CodecRegistry.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/CodecRegistry.java @@ -16,7 +16,7 @@ * limitations under the License. * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.IOException; import java.util.HashMap; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBCheckpoint.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBCheckpoint.java new file mode 100644 index 0000000000000..6a45298cb6a3c --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBCheckpoint.java @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.hdds.utils.db; + +import java.io.IOException; +import java.nio.file.Path; + +/** + * Generic DB Checkpoint interface. + */ +public interface DBCheckpoint { + + /** + * Get Snapshot location. + */ + Path getCheckpointLocation(); + + /** + * Get Snapshot creation timestamp. + */ + long getCheckpointTimestamp(); + + /** + * Get last sequence number of Snapshot. + */ + long getLatestSequenceNumber(); + + /** + * Time taken in milliseconds for the checkpoint to be created. + */ + long checkpointCreationTimeTaken(); + + /** + * Destroy the contents of the specified checkpoint to ensure + * proper cleanup of the footprint on disk. + * + * @throws IOException if I/O error happens + */ + void cleanupCheckpoint() throws IOException; + + /** + * Set the OM Ratis snapshot index corresponding to the OM DB checkpoint. + * The snapshot index is the latest snapshot index saved by ratis + * snapshots. It is not guaranteed to be the last ratis index applied to + * the OM DB state. + * @param omRatisSnapshotIndex the saved ratis snapshot index + */ + void setRatisSnapshotIndex(long omRatisSnapshotIndex); + + /** + * Get the OM Ratis snapshot index corresponding to the OM DB checkpoint. + * The ratis snapshot index indicates upto which index is definitely + * included in the DB checkpoint. It is not guaranteed to be the last ratis + * log index applied to the DB checkpoint. + */ + long getRatisSnapshotIndex(); +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBConfigFromFile.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBConfigFromFile.java similarity index 99% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBConfigFromFile.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBConfigFromFile.java index 94370b1cd5b18..43754255eabfd 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBConfigFromFile.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBConfigFromFile.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import com.google.common.base.Preconditions; import org.eclipse.jetty.util.StringUtil; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBProfile.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBProfile.java similarity index 98% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBProfile.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBProfile.java index 4d3d6bc7700b0..57516fd89a40b 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBProfile.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBProfile.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import org.apache.hadoop.conf.StorageUnit; import org.rocksdb.BlockBasedTableConfig; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBStore.java similarity index 81% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBStore.java index 9e0c4a4b42c26..b3f58384203fc 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBStore.java @@ -17,13 +17,15 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.File; import java.io.IOException; import java.util.ArrayList; +import java.util.Map; import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.hdds.utils.db.cache.TableCacheImpl; /** * The DBStore interface provides the ability to create Tables, which store @@ -46,7 +48,9 @@ public interface DBStore extends AutoCloseable { /** - * Gets an existing TableStore with implicit key/value conversion. + * Gets an existing TableStore with implicit key/value conversion and + * with default cleanup policy for cache. Default cache clean up policy is + * manual. * * @param name - Name of the TableStore to get * @param keyType @@ -57,6 +61,15 @@ public interface DBStore extends AutoCloseable { Table getTable(String name, Class keyType, Class valueType) throws IOException; + /** + * Gets an existing TableStore with implicit key/value conversion and + * with specified cleanup policy for cache. + * @throws IOException + */ + Table getTable(String name, + Class keyType, Class valueType, + TableCacheImpl.CacheCleanupPolicy cleanupPolicy) throws IOException; + /** * Lists the Known list of Tables in a DB. * @@ -158,4 +171,26 @@ void move(KEY sourceKey, KEY destKey, VALUE value, * @return DB file location. */ File getDbLocation(); + + /** + * Get List of Index to Table Names. + * (For decoding table from column family index) + * @return Map of Index -> TableName + */ + Map getTableNames(); + + /** + * Get Codec registry. + * @return codec registry. + */ + CodecRegistry getCodecRegistry(); + + /** + * Get data written to DB since a specific sequence number. + * @param sequenceNumber + * @return + * @throws SequenceNumberNotFoundException + */ + DBUpdatesWrapper getUpdatesSince(long sequenceNumber) + throws SequenceNumberNotFoundException; } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBStoreBuilder.java similarity index 76% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBStoreBuilder.java index 34bdc5dbc351c..263864fede890 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBStoreBuilder.java @@ -17,17 +17,21 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import com.google.common.base.Preconditions; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdfs.DFSUtil; import org.eclipse.jetty.util.StringUtil; import org.rocksdb.ColumnFamilyDescriptor; import org.rocksdb.ColumnFamilyOptions; import org.rocksdb.DBOptions; +import org.rocksdb.InfoLogLevel; import org.rocksdb.RocksDB; +import org.rocksdb.Statistics; +import org.rocksdb.StatsLevel; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -42,6 +46,9 @@ import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_DB_PROFILE; import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_DEFAULT_DB_PROFILE; +import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS; +import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS_DEFAULT; +import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS_OFF; /** * DBStore Builder. @@ -49,6 +56,8 @@ public final class DBStoreBuilder { private static final Logger LOG = LoggerFactory.getLogger(DBStoreBuilder.class); + public static final Logger ROCKS_DB_LOGGER = + LoggerFactory.getLogger(RocksDB.class); private Set tables; private DBProfile dbProfile; private DBOptions rocksDBOption; @@ -57,16 +66,22 @@ public final class DBStoreBuilder { private List tableNames; private Configuration configuration; private CodecRegistry registry; - private boolean readOnly = false; + private String rocksDbStat; + private RocksDBConfiguration rocksDBConfiguration; - private DBStoreBuilder(Configuration configuration) { + private DBStoreBuilder(OzoneConfiguration configuration) { tables = new HashSet<>(); tableNames = new LinkedList<>(); this.configuration = configuration; this.registry = new CodecRegistry(); + this.rocksDbStat = configuration.getTrimmed( + OZONE_METADATA_STORE_ROCKSDB_STATISTICS, + OZONE_METADATA_STORE_ROCKSDB_STATISTICS_DEFAULT); + this.rocksDBConfiguration = + configuration.getObject(RocksDBConfiguration.class); } - public static DBStoreBuilder newBuilder(Configuration configuration) { + public static DBStoreBuilder newBuilder(OzoneConfiguration configuration) { return new DBStoreBuilder(configuration); } @@ -114,11 +129,6 @@ public DBStoreBuilder setPath(Path path) { return this; } - public DBStoreBuilder setReadOnly(boolean rdOnly) { - readOnly = rdOnly; - return this; - } - /** * Builds a DBStore instance and returns that. * @@ -137,7 +147,7 @@ public DBStore build() throws IOException { if (!dbFile.getParentFile().exists()) { throw new IOException("The DB destination directory should exist."); } - return new RDBStore(dbFile, options, tables, registry, readOnly); + return new RDBStore(dbFile, options, tables, registry); } /** @@ -193,7 +203,26 @@ private DBOptions getDbProfile() { if (option == null) { LOG.info("Using default options. {}", dbProfile.toString()); - return dbProfile.getDBOptions(); + option = dbProfile.getDBOptions(); + } + + if (rocksDBConfiguration.isRocksdbLoggingEnabled()) { + org.rocksdb.Logger logger = new org.rocksdb.Logger(option) { + @Override + protected void log(InfoLogLevel infoLogLevel, String s) { + ROCKS_DB_LOGGER.info(s); + } + }; + InfoLogLevel level = InfoLogLevel.valueOf(rocksDBConfiguration + .getRocksdbLogLevel() + "_LEVEL"); + logger.setInfoLogLevel(level); + option.setLogger(logger); + } + + if (!rocksDbStat.equals(OZONE_METADATA_STORE_ROCKSDB_STATISTICS_OFF)) { + Statistics statistics = new Statistics(); + statistics.setStatsLevel(StatsLevel.valueOf(rocksDbStat)); + option = option.setStatistics(statistics); } return option; } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBUpdatesWrapper.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBUpdatesWrapper.java new file mode 100644 index 0000000000000..aa48c5e83b0d8 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DBUpdatesWrapper.java @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.hdds.utils.db; + +import java.util.ArrayList; +import java.util.List; + +/** + * Wrapper class to hold DB data read from the RocksDB log file. + */ +public class DBUpdatesWrapper { + + private List dataList = new ArrayList<>(); + private long currentSequenceNumber = -1; + + public void addWriteBatch(byte[] data, long sequenceNumber) { + dataList.add(data); + if (currentSequenceNumber < sequenceNumber) { + currentSequenceNumber = sequenceNumber; + } + } + + public List getData() { + return dataList; + } + + public void setCurrentSequenceNumber(long sequenceNumber) { + this.currentSequenceNumber = sequenceNumber; + } + + public long getCurrentSequenceNumber() { + return currentSequenceNumber; + } +} + diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/IntegerCodec.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/IntegerCodec.java similarity index 96% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/IntegerCodec.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/IntegerCodec.java index fc80ff9883e8e..e95e0f1b757c9 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/IntegerCodec.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/IntegerCodec.java @@ -16,7 +16,7 @@ * limitations under the License. */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.IOException; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/LongCodec.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/LongCodec.java similarity index 96% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/LongCodec.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/LongCodec.java index c7a249e37a6aa..6c95246ebea72 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/LongCodec.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/LongCodec.java @@ -16,7 +16,7 @@ * limitations under the License. * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import com.google.common.primitives.Longs; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBBatchOperation.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBBatchOperation.java similarity index 93% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBBatchOperation.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBBatchOperation.java index a8b78ed8a92f5..42843b080d738 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBBatchOperation.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBBatchOperation.java @@ -16,7 +16,7 @@ * limitations under the License. * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.IOException; @@ -37,6 +37,10 @@ public RDBBatchOperation() { writeBatch = new WriteBatch(); } + public RDBBatchOperation(WriteBatch writeBatch) { + this.writeBatch = writeBatch; + } + public void commit(RocksDB db, WriteOptions writeOptions) throws IOException { try { db.write(writeOptions, writeBatch); diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBCheckpointManager.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBCheckpointManager.java similarity index 96% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBCheckpointManager.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBCheckpointManager.java index 68d196f97a816..42b9b77d2d874 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBCheckpointManager.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBCheckpointManager.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.nio.file.Path; import java.nio.file.Paths; @@ -83,7 +83,7 @@ public RocksDBCheckpoint createCheckpoint(String parentDir) { Instant end = Instant.now(); long duration = Duration.between(start, end).toMillis(); - LOG.debug("Created checkpoint at " + checkpointPath.toString() + " in " + LOG.info("Created checkpoint at " + checkpointPath.toString() + " in " + duration + " milliseconds"); return new RocksDBCheckpoint( diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java new file mode 100644 index 0000000000000..53bd424642afa --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java @@ -0,0 +1,381 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.hdds.utils.db; + +import static org.apache.hadoop.ozone.OzoneConsts.OM_DB_CHECKPOINTS_DIR_NAME; + +import javax.management.ObjectName; +import java.io.File; +import java.io.IOException; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Hashtable; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import org.apache.hadoop.hdds.HddsUtils; +import org.apache.hadoop.hdds.utils.RocksDBStoreMBean; +import org.apache.hadoop.hdfs.DFSUtil; +import org.apache.hadoop.metrics2.util.MBeans; + +import com.google.common.base.Preconditions; +import org.apache.hadoop.hdds.utils.db.cache.TableCacheImpl; +import org.apache.ratis.thirdparty.com.google.common.annotations.VisibleForTesting; +import org.rocksdb.ColumnFamilyDescriptor; +import org.rocksdb.ColumnFamilyHandle; +import org.rocksdb.DBOptions; +import org.rocksdb.FlushOptions; +import org.rocksdb.RocksDB; +import org.rocksdb.RocksDBException; +import org.rocksdb.TransactionLogIterator; +import org.rocksdb.WriteOptions; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * RocksDB Store that supports creating Tables in DB. + */ +public class RDBStore implements DBStore { + private static final Logger LOG = + LoggerFactory.getLogger(RDBStore.class); + private RocksDB db; + private File dbLocation; + private final WriteOptions writeOptions; + private final DBOptions dbOptions; + private final CodecRegistry codecRegistry; + private final Hashtable handleTable; + private ObjectName statMBeanName; + private RDBCheckpointManager checkPointManager; + private String checkpointsParentDir; + private List columnFamilyHandles; + + @VisibleForTesting + public RDBStore(File dbFile, DBOptions options, + Set families) throws IOException { + this(dbFile, options, families, new CodecRegistry()); + } + + public RDBStore(File dbFile, DBOptions options, Set families, + CodecRegistry registry) + throws IOException { + Preconditions.checkNotNull(dbFile, "DB file location cannot be null"); + Preconditions.checkNotNull(families); + Preconditions.checkArgument(families.size() > 0); + handleTable = new Hashtable<>(); + codecRegistry = registry; + final List columnFamilyDescriptors = + new ArrayList<>(); + columnFamilyHandles = new ArrayList<>(); + + for (TableConfig family : families) { + columnFamilyDescriptors.add(family.getDescriptor()); + } + + dbOptions = options; + dbLocation = dbFile; + // TODO: Read from the next Config. + writeOptions = new WriteOptions(); + + try { + db = RocksDB.open(dbOptions, dbLocation.getAbsolutePath(), + columnFamilyDescriptors, columnFamilyHandles); + + for (int x = 0; x < columnFamilyHandles.size(); x++) { + handleTable.put( + DFSUtil.bytes2String(columnFamilyHandles.get(x).getName()), + columnFamilyHandles.get(x)); + } + + if (dbOptions.statistics() != null) { + Map jmxProperties = new HashMap<>(); + jmxProperties.put("dbName", dbFile.getName()); + statMBeanName = HddsUtils.registerWithJmxProperties( + "Ozone", "RocksDbStore", jmxProperties, + RocksDBStoreMBean.create(dbOptions.statistics(), + dbFile.getName())); + if (statMBeanName == null) { + LOG.warn("jmx registration failed during RocksDB init, db path :{}", + dbFile.getAbsolutePath()); + } + } + + //create checkpoints directory if not exists. + checkpointsParentDir = Paths.get(dbLocation.getParent(), + OM_DB_CHECKPOINTS_DIR_NAME).toString(); + File checkpointsDir = new File(checkpointsParentDir); + if (!checkpointsDir.exists()) { + boolean success = checkpointsDir.mkdir(); + if (!success) { + LOG.warn("Unable to create RocksDB checkpoint directory"); + } + } + + //Initialize checkpoint manager + checkPointManager = new RDBCheckpointManager(db, "om"); + + } catch (RocksDBException e) { + throw toIOException( + "Failed init RocksDB, db path : " + dbFile.getAbsolutePath(), e); + } + + if (LOG.isDebugEnabled()) { + LOG.debug("RocksDB successfully opened."); + LOG.debug("[Option] dbLocation= {}", dbLocation.getAbsolutePath()); + LOG.debug("[Option] createIfMissing = {}", options.createIfMissing()); + LOG.debug("[Option] maxOpenFiles= {}", options.maxOpenFiles()); + } + } + + public static IOException toIOException(String msg, RocksDBException e) { + String statusCode = e.getStatus() == null ? "N/A" : + e.getStatus().getCodeString(); + String errMessage = e.getMessage() == null ? "Unknown error" : + e.getMessage(); + String output = msg + "; status : " + statusCode + + "; message : " + errMessage; + return new IOException(output, e); + } + + @Override + public void compactDB() throws IOException { + if (db != null) { + try { + db.compactRange(); + } catch (RocksDBException e) { + throw toIOException("Failed to compact db", e); + } + } + } + + @Override + public void close() throws IOException { + + for (final ColumnFamilyHandle handle : handleTable.values()) { + handle.close(); + } + + if (statMBeanName != null) { + MBeans.unregister(statMBeanName); + statMBeanName = null; + } + + if (db != null) { + db.close(); + } + + if (dbOptions != null) { + dbOptions.close(); + } + + if (writeOptions != null) { + writeOptions.close(); + } + } + + @Override + public void move(KEY key, Table source, + Table dest) throws IOException { + try (BatchOperation batchOperation = initBatchOperation()) { + + VALUE value = source.get(key); + dest.putWithBatch(batchOperation, key, value); + source.deleteWithBatch(batchOperation, key); + commitBatchOperation(batchOperation); + } + } + + @Override + public void move(KEY key, VALUE value, Table source, + Table dest) throws IOException { + move(key, key, value, source, dest); + } + + @Override + public void move(KEY sourceKey, KEY destKey, VALUE value, + Table source, + Table dest) throws IOException { + try (BatchOperation batchOperation = initBatchOperation()) { + dest.putWithBatch(batchOperation, destKey, value); + source.deleteWithBatch(batchOperation, sourceKey); + commitBatchOperation(batchOperation); + } + } + + @Override + public long getEstimatedKeyCount() throws IOException { + try { + return db.getLongProperty("rocksdb.estimate-num-keys"); + } catch (RocksDBException e) { + throw toIOException("Unable to get the estimated count.", e); + } + } + + @Override + public BatchOperation initBatchOperation() { + return new RDBBatchOperation(); + } + + @Override + public void commitBatchOperation(BatchOperation operation) + throws IOException { + ((RDBBatchOperation) operation).commit(db, writeOptions); + } + + + @VisibleForTesting + protected ObjectName getStatMBeanName() { + return statMBeanName; + } + + @Override + public Table getTable(String name) throws IOException { + ColumnFamilyHandle handle = handleTable.get(name); + if (handle == null) { + throw new IOException("No such table in this DB. TableName : " + name); + } + return new RDBTable(this.db, handle, this.writeOptions); + } + + @Override + public Table getTable(String name, + Class keyType, Class valueType) throws IOException { + return new TypedTable(getTable(name), codecRegistry, keyType, + valueType); + } + + @Override + public Table getTable(String name, + Class keyType, Class valueType, + TableCacheImpl.CacheCleanupPolicy cleanupPolicy) throws IOException { + return new TypedTable(getTable(name), codecRegistry, keyType, + valueType, cleanupPolicy); + } + + @Override + public ArrayList listTables() throws IOException { + ArrayList
    returnList = new ArrayList<>(); + for (ColumnFamilyHandle handle : handleTable.values()) { + returnList.add(new RDBTable(db, handle, writeOptions)); + } + return returnList; + } + + @Override + public void flush() throws IOException { + final FlushOptions flushOptions = new FlushOptions().setWaitForFlush(true); + try { + db.flush(flushOptions); + } catch (RocksDBException e) { + LOG.error("Unable to Flush RocksDB data", e); + throw toIOException("Unable to Flush RocksDB data", e); + } + } + + @Override + public DBCheckpoint getCheckpoint(boolean flush) { + final FlushOptions flushOptions = new FlushOptions().setWaitForFlush(flush); + try { + db.flush(flushOptions); + } catch (RocksDBException e) { + LOG.error("Unable to Flush RocksDB data before creating snapshot", e); + } + return checkPointManager.createCheckpoint(checkpointsParentDir); + } + + @Override + public File getDbLocation() { + return dbLocation; + } + + @Override + public Map getTableNames() { + Map tableNames = new HashMap<>(); + StringCodec stringCodec = new StringCodec(); + + for (ColumnFamilyHandle columnFamilyHandle : columnFamilyHandles) { + try { + tableNames.put(columnFamilyHandle.getID(), stringCodec + .fromPersistedFormat(columnFamilyHandle.getName())); + } catch (RocksDBException | IOException e) { + LOG.error("Unexpected exception while reading column family handle " + + "name", e); + } + } + return tableNames; + } + + @Override + public CodecRegistry getCodecRegistry() { + return codecRegistry; + } + + @Override + public DBUpdatesWrapper getUpdatesSince(long sequenceNumber) + throws SequenceNumberNotFoundException { + + DBUpdatesWrapper dbUpdatesWrapper = new DBUpdatesWrapper(); + try { + TransactionLogIterator transactionLogIterator = + db.getUpdatesSince(sequenceNumber); + + // Only the first record needs to be checked if its seq number < + // ( 1 + passed_in_sequence_number). For example, if seqNumber passed + // in is 100, then we can read from the WAL ONLY if the first sequence + // number is <= 101. If it is 102, then 101 may already be flushed to + // SST. If it 99, we can skip 99 and 100, and then read from 101. + + boolean checkValidStartingSeqNumber = true; + + while (transactionLogIterator.isValid()) { + TransactionLogIterator.BatchResult result = + transactionLogIterator.getBatch(); + long currSequenceNumber = result.sequenceNumber(); + if (checkValidStartingSeqNumber && + currSequenceNumber > 1 + sequenceNumber) { + throw new SequenceNumberNotFoundException("Unable to read data from" + + " RocksDB wal to get delta updates. It may have already been" + + "flushed to SSTs."); + } + // If the above condition was not satisfied, then it is OK to reset + // the flag. + checkValidStartingSeqNumber = false; + if (currSequenceNumber <= sequenceNumber) { + transactionLogIterator.next(); + continue; + } + dbUpdatesWrapper.addWriteBatch(result.writeBatch().data(), + result.sequenceNumber()); + transactionLogIterator.next(); + } + } catch (RocksDBException e) { + LOG.error("Unable to get delta updates since sequenceNumber {} ", + sequenceNumber, e); + } + return dbUpdatesWrapper; + } + + @VisibleForTesting + public RocksDB getDb() { + return db; + } + +} \ No newline at end of file diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStoreIterator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreIterator.java similarity index 98% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStoreIterator.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreIterator.java index ecc065bc3760e..784738b0cec28 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStoreIterator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreIterator.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.IOException; import java.util.NoSuchElementException; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBTable.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBTable.java similarity index 86% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBTable.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBTable.java index 7bbe9d91b171b..49ccc020922fa 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBTable.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBTable.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.IOException; import java.nio.charset.StandardCharsets; @@ -119,6 +119,20 @@ public boolean isEmpty() throws IOException { } } + @Override + public boolean isExist(byte[] key) throws IOException { + try { + // RocksDB#keyMayExist + // If the key definitely does not exist in the database, then this + // method returns false, else true. + return db.keyMayExist(handle, key, new StringBuilder()) + && db.get(handle, key) != null; + } catch (RocksDBException e) { + throw toIOException( + "Error in accessing DB. ", e); + } + } + @Override public byte[] get(byte[] key) throws IOException { try { @@ -152,6 +166,7 @@ public void deleteWithBatch(BatchOperation batch, byte[] key) @Override public TableIterator iterator() { ReadOptions readOptions = new ReadOptions(); + readOptions.setFillCache(false); return new RDBStoreIterator(db.newIterator(handle, readOptions)); } @@ -168,4 +183,14 @@ public String getName() throws IOException { public void close() throws Exception { // Nothing do for a Column Family. } + + @Override + public long getEstimatedKeyCount() throws IOException { + try { + return db.getLongProperty(handle, "rocksdb.estimate-num-keys"); + } catch (RocksDBException e) { + throw toIOException( + "Failed to get estimated key count of table " + getName(), e); + } + } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RocksDBCheckpoint.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDBCheckpoint.java similarity index 86% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RocksDBCheckpoint.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDBCheckpoint.java index 88b3f75b1aa78..149743816c203 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RocksDBCheckpoint.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDBCheckpoint.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.IOException; import java.nio.file.Path; @@ -38,6 +38,7 @@ public class RocksDBCheckpoint implements DBCheckpoint { private long checkpointTimestamp = System.currentTimeMillis(); private long latestSequenceNumber = -1; private long checkpointCreationTimeTaken = 0L; + private long ratisSnapshotIndex = 0L; public RocksDBCheckpoint(Path checkpointLocation) { this.checkpointLocation = checkpointLocation; @@ -75,7 +76,18 @@ public long checkpointCreationTimeTaken() { @Override public void cleanupCheckpoint() throws IOException { - LOG.debug("Cleaning up checkpoint at " + checkpointLocation.toString()); + LOG.info("Cleaning up RocksDB checkpoint at " + + checkpointLocation.toString()); FileUtils.deleteDirectory(checkpointLocation.toFile()); } + + @Override + public void setRatisSnapshotIndex(long omRatisSnapshotIndex) { + this.ratisSnapshotIndex = omRatisSnapshotIndex; + } + + @Override + public long getRatisSnapshotIndex() { + return ratisSnapshotIndex; + } } \ No newline at end of file diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDBConfiguration.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDBConfiguration.java new file mode 100644 index 0000000000000..1a8c846a3eb21 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDBConfiguration.java @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.utils.db; + +import org.apache.hadoop.hdds.conf.Config; +import org.apache.hadoop.hdds.conf.ConfigGroup; +import org.apache.hadoop.hdds.conf.ConfigTag; +import org.apache.hadoop.hdds.conf.ConfigType; + +/** + * Holds configuration items for OM RocksDB. + */ +@ConfigGroup(prefix = "hadoop.hdds.db") +public class RocksDBConfiguration { + + private boolean rocksdbLogEnabled; + + @Config(key = "rocksdb.logging.enabled", + type = ConfigType.BOOLEAN, + defaultValue = "false", + tags = {ConfigTag.OM}, + description = "Enable/Disable RocksDB logging for OM.") + public void setRocksdbLoggingEnabled(boolean enabled) { + this.rocksdbLogEnabled = enabled; + } + + public boolean isRocksdbLoggingEnabled() { + return rocksdbLogEnabled; + } + + private String rocksdbLogLevel; + + @Config(key = "rocksdb.logging.level", + type = ConfigType.STRING, + defaultValue = "INFO", + tags = {ConfigTag.OM}, + description = "OM RocksDB logging level (INFO/DEBUG/WARN/ERROR/FATAL)") + public void setRocksdbLogLevel(String level) { + this.rocksdbLogLevel = level; + } + + public String getRocksdbLogLevel() { + return rocksdbLogLevel; + } + +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/SequenceNumberNotFoundException.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/SequenceNumberNotFoundException.java new file mode 100644 index 0000000000000..e9b4fa391ecf2 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/SequenceNumberNotFoundException.java @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.hdds.utils.db; + +import java.io.IOException; + +/** + * Thrown if RocksDB is unable to find requested data from WAL file. + */ +public class SequenceNumberNotFoundException extends IOException { + + public SequenceNumberNotFoundException() { + super(); + } + + public SequenceNumberNotFoundException(String message) { + super(message); + } + +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/StringCodec.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/StringCodec.java similarity index 97% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/StringCodec.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/StringCodec.java index 5f38d68fb88c3..f8237367c7245 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/StringCodec.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/StringCodec.java @@ -16,7 +16,7 @@ * limitations under the License. * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.IOException; import org.apache.hadoop.hdfs.DFSUtil; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Table.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/Table.java similarity index 75% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Table.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/Table.java index 905a68b064624..0502541e9c5c7 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Table.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/Table.java @@ -17,14 +17,16 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.IOException; +import java.util.Iterator; +import java.util.Map; import org.apache.commons.lang3.NotImplementedException; import org.apache.hadoop.classification.InterfaceStability; -import org.apache.hadoop.utils.db.cache.CacheKey; -import org.apache.hadoop.utils.db.cache.CacheValue; +import org.apache.hadoop.hdds.utils.db.cache.CacheKey; +import org.apache.hadoop.hdds.utils.db.cache.CacheValue; /** * Interface for key-value store that stores ozone metadata. Ozone metadata is * stored as key value pairs, both key and value are arbitrary byte arrays. Each @@ -58,6 +60,16 @@ void putWithBatch(BatchOperation batch, KEY key, VALUE value) */ boolean isEmpty() throws IOException; + /** + * Check if a given key exists in Metadata store. + * (Optimization to save on data deserialization) + * A lock on the key / bucket needs to be acquired before invoking this API. + * @param key metadata key + * @return true if the metadata store contains a key. + * @throws IOException on Failure + */ + boolean isExist(KEY key) throws IOException; + /** * Returns the value mapped to the given key in byte array or returns null * if the key is not found. @@ -99,6 +111,13 @@ void putWithBatch(BatchOperation batch, KEY key, VALUE value) */ String getName() throws IOException; + /** + * Returns the key count of this Table. Note the result can be inaccurate. + * @return Estimated key count of this Table + * @throws IOException on failure + */ + long getEstimatedKeyCount() throws IOException; + /** * Add entry to the table cache. * @@ -111,6 +130,14 @@ default void addCacheEntry(CacheKey cacheKey, throw new NotImplementedException("addCacheEntry is not implemented"); } + /** + * Get the cache value from table cache. + * @param cacheKey + */ + default CacheValue getCacheValue(CacheKey cacheKey) { + throw new NotImplementedException("getCacheValue is not implemented"); + } + /** * Removes all the entries from the table cache which are having epoch value * less @@ -121,6 +148,14 @@ default void cleanupCache(long epoch) { throw new NotImplementedException("cleanupCache is not implemented"); } + /** + * Return cache iterator maintained for this table. + */ + default Iterator, CacheValue>> + cacheIterator() { + throw new NotImplementedException("cacheIterator is not implemented"); + } + /** * Class used to represent the key and value pair of a db entry. */ diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TableConfig.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/TableConfig.java similarity index 98% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TableConfig.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/TableConfig.java index 897028a821fa5..d8eb401659edb 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TableConfig.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/TableConfig.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import org.apache.commons.lang3.builder.EqualsBuilder; import org.apache.commons.lang3.builder.HashCodeBuilder; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TableIterator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/TableIterator.java similarity index 97% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TableIterator.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/TableIterator.java index 2e8e7edf9ca57..a684157a43b11 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TableIterator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/TableIterator.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.Closeable; import java.io.IOException; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/TypedTable.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/TypedTable.java new file mode 100644 index 0000000000000..597eff1f658fe --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/TypedTable.java @@ -0,0 +1,361 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.apache.hadoop.hdds.utils.db; + +import java.io.IOException; +import java.util.Iterator; +import java.util.Map; + +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Optional; +import org.apache.hadoop.hdds.utils.db.cache.CacheKey; +import org.apache.hadoop.hdds.utils.db.cache.CacheResult; +import org.apache.hadoop.hdds.utils.db.cache.CacheValue; +import org.apache.hadoop.hdds.utils.db.cache.TableCacheImpl; +import org.apache.hadoop.hdds.utils.db.cache.TableCache; +import org.apache.hadoop.hdds.utils.db.cache.TableCacheImpl.CacheCleanupPolicy; + +import static org.apache.hadoop.hdds.utils.db.cache.CacheResult.CacheStatus.EXISTS; +import static org.apache.hadoop.hdds.utils.db.cache.CacheResult.CacheStatus.NOT_EXIST; +/** + * Strongly typed table implementation. + *

    + * Automatically converts values and keys using a raw byte[] based table + * implementation and registered converters. + * + * @param type of the keys in the store. + * @param type of the values in the store. + */ +public class TypedTable implements Table { + + private final Table rawTable; + + private final CodecRegistry codecRegistry; + + private final Class keyType; + + private final Class valueType; + + private final TableCache, CacheValue> cache; + + private final static long EPOCH_DEFAULT = -1L; + + /** + * Create an TypedTable from the raw table. + * Default cleanup policy used for the table is + * {@link CacheCleanupPolicy#MANUAL}. + * @param rawTable + * @param codecRegistry + * @param keyType + * @param valueType + */ + public TypedTable( + Table rawTable, + CodecRegistry codecRegistry, Class keyType, + Class valueType) throws IOException { + this(rawTable, codecRegistry, keyType, valueType, + CacheCleanupPolicy.MANUAL); + } + + /** + * Create an TypedTable from the raw table with specified cleanup policy + * for table cache. + * @param rawTable + * @param codecRegistry + * @param keyType + * @param valueType + * @param cleanupPolicy + */ + public TypedTable( + Table rawTable, + CodecRegistry codecRegistry, Class keyType, + Class valueType, + TableCacheImpl.CacheCleanupPolicy cleanupPolicy) throws IOException { + this.rawTable = rawTable; + this.codecRegistry = codecRegistry; + this.keyType = keyType; + this.valueType = valueType; + cache = new TableCacheImpl<>(cleanupPolicy); + + if (cleanupPolicy == CacheCleanupPolicy.NEVER) { + //fill cache + try(TableIterator> tableIterator = + iterator()) { + + while (tableIterator.hasNext()) { + KeyValue< KEY, VALUE > kv = tableIterator.next(); + + // We should build cache after OM restart when clean up policy is + // NEVER. Setting epoch value -1, so that when it is marked for + // delete, this will be considered for cleanup. + cache.loadInitial(new CacheKey<>(kv.getKey()), + new CacheValue<>(Optional.of(kv.getValue()), EPOCH_DEFAULT)); + } + } + } + } + + @Override + public void put(KEY key, VALUE value) throws IOException { + byte[] keyData = codecRegistry.asRawData(key); + byte[] valueData = codecRegistry.asRawData(value); + rawTable.put(keyData, valueData); + } + + @Override + public void putWithBatch(BatchOperation batch, KEY key, VALUE value) + throws IOException { + byte[] keyData = codecRegistry.asRawData(key); + byte[] valueData = codecRegistry.asRawData(value); + rawTable.putWithBatch(batch, keyData, valueData); + } + + @Override + public boolean isEmpty() throws IOException { + return rawTable.isEmpty(); + } + + @Override + public boolean isExist(KEY key) throws IOException { + + CacheResult> cacheResult = + cache.lookup(new CacheKey<>(key)); + + if (cacheResult.getCacheStatus() == EXISTS) { + return true; + } else if (cacheResult.getCacheStatus() == NOT_EXIST) { + return false; + } else { + return rawTable.isExist(codecRegistry.asRawData(key)); + } + } + + /** + * Returns the value mapped to the given key in byte array or returns null + * if the key is not found. + * + * Caller's of this method should use synchronization mechanism, when + * accessing. First it will check from cache, if it has entry return the + * value, otherwise get from the RocksDB table. + * + * @param key metadata key + * @return VALUE + * @throws IOException + */ + @Override + public VALUE get(KEY key) throws IOException { + // Here the metadata lock will guarantee that cache is not updated for same + // key during get key. + + CacheResult> cacheResult = + cache.lookup(new CacheKey<>(key)); + + if (cacheResult.getCacheStatus() == EXISTS) { + return cacheResult.getValue().getCacheValue(); + } else if (cacheResult.getCacheStatus() == NOT_EXIST) { + return null; + } else { + return getFromTable(key); + } + } + + private VALUE getFromTable(KEY key) throws IOException { + byte[] keyBytes = codecRegistry.asRawData(key); + byte[] valueBytes = rawTable.get(keyBytes); + return codecRegistry.asObject(valueBytes, valueType); + } + + @Override + public void delete(KEY key) throws IOException { + rawTable.delete(codecRegistry.asRawData(key)); + } + + @Override + public void deleteWithBatch(BatchOperation batch, KEY key) + throws IOException { + rawTable.deleteWithBatch(batch, codecRegistry.asRawData(key)); + + } + + @Override + public TableIterator iterator() { + TableIterator> iterator = + rawTable.iterator(); + return new TypedTableIterator(iterator, keyType, valueType); + } + + @Override + public String getName() throws IOException { + return rawTable.getName(); + } + + @Override + public long getEstimatedKeyCount() throws IOException { + return rawTable.getEstimatedKeyCount(); + } + + @Override + public void close() throws Exception { + rawTable.close(); + + } + + @Override + public void addCacheEntry(CacheKey cacheKey, + CacheValue cacheValue) { + // This will override the entry if there is already entry for this key. + cache.put(cacheKey, cacheValue); + } + + @Override + public CacheValue getCacheValue(CacheKey cacheKey) { + return cache.get(cacheKey); + } + + public Iterator, CacheValue>> cacheIterator() { + return cache.iterator(); + } + + @Override + public void cleanupCache(long epoch) { + cache.cleanup(epoch); + } + + @VisibleForTesting + TableCache, CacheValue> getCache() { + return cache; + } + + public Table getRawTable() { + return rawTable; + } + + public CodecRegistry getCodecRegistry() { + return codecRegistry; + } + + public Class getKeyType() { + return keyType; + } + + public Class getValueType() { + return valueType; + } + + /** + * Key value implementation for strongly typed tables. + */ + public class TypedKeyValue implements KeyValue { + + private KeyValue rawKeyValue; + + public TypedKeyValue(KeyValue rawKeyValue) { + this.rawKeyValue = rawKeyValue; + } + + public TypedKeyValue(KeyValue rawKeyValue, + Class keyType, Class valueType) { + this.rawKeyValue = rawKeyValue; + } + + @Override + public KEY getKey() throws IOException { + return codecRegistry.asObject(rawKeyValue.getKey(), keyType); + } + + @Override + public VALUE getValue() throws IOException { + return codecRegistry.asObject(rawKeyValue.getValue(), valueType); + } + } + + /** + * Table Iterator implementation for strongly typed tables. + */ + public class TypedTableIterator implements TableIterator { + + private TableIterator> + rawIterator; + private final Class keyClass; + private final Class valueClass; + + public TypedTableIterator( + TableIterator> rawIterator, + Class keyType, + Class valueType) { + this.rawIterator = rawIterator; + keyClass = keyType; + valueClass = valueType; + } + + @Override + public void seekToFirst() { + rawIterator.seekToFirst(); + } + + @Override + public void seekToLast() { + rawIterator.seekToLast(); + } + + @Override + public TypedKeyValue seek(KEY key) throws IOException { + byte[] keyBytes = codecRegistry.asRawData(key); + KeyValue result = rawIterator.seek(keyBytes); + if (result == null) { + return null; + } + return new TypedKeyValue(result); + } + + @Override + public KEY key() throws IOException { + byte[] result = rawIterator.key(); + if (result == null) { + return null; + } + return codecRegistry.asObject(result, keyClass); + } + + @Override + public TypedKeyValue value() { + KeyValue keyValue = rawIterator.value(); + if(keyValue != null) { + return new TypedKeyValue(keyValue, keyClass, valueClass); + } + return null; + } + + @Override + public void close() throws IOException { + rawIterator.close(); + } + + @Override + public boolean hasNext() { + return rawIterator.hasNext(); + } + + @Override + public TypedKeyValue next() { + return new TypedKeyValue(rawIterator.next(), keyType, + valueType); + } + } +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/CacheKey.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheKey.java similarity index 80% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/CacheKey.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheKey.java index f928e4775a546..7be2921b6a117 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/CacheKey.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheKey.java @@ -16,7 +16,7 @@ * limitations under the License. */ -package org.apache.hadoop.utils.db.cache; +package org.apache.hadoop.hdds.utils.db.cache; import java.util.Objects; @@ -24,7 +24,7 @@ * CacheKey for the RocksDB table. * @param */ -public class CacheKey { +public class CacheKey implements Comparable { private final KEY key; @@ -33,7 +33,7 @@ public CacheKey(KEY key) { this.key = key; } - public KEY getKey() { + public KEY getCacheKey() { return key; } @@ -53,4 +53,13 @@ public boolean equals(Object o) { public int hashCode() { return Objects.hash(key); } + + @Override + public int compareTo(Object o) { + if(Objects.equals(key, ((CacheKey)o).key)) { + return 0; + } else { + return key.toString().compareTo((((CacheKey) o).key).toString()); + } + } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheResult.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheResult.java new file mode 100644 index 0000000000000..8c5a68ba0721b --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheResult.java @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

    + * http://www.apache.org/licenses/LICENSE-2.0 + *

    + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.utils.db.cache; + +import java.util.Objects; + +/** + * CacheResult which is returned as response for Key exist in cache or not. + * @param + */ +public class CacheResult { + + private CacheStatus cacheStatus; + private CACHEVALUE cachevalue; + + public CacheResult(CacheStatus cacheStatus, CACHEVALUE cachevalue) { + this.cacheStatus = cacheStatus; + this.cachevalue = cachevalue; + } + + public CacheStatus getCacheStatus() { + return cacheStatus; + } + + public CACHEVALUE getValue() { + return cachevalue; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + CacheResult< ? > that = (CacheResult< ? >) o; + return cacheStatus == that.cacheStatus && + Objects.equals(cachevalue, that.cachevalue); + } + + @Override + public int hashCode() { + return Objects.hash(cacheStatus, cachevalue); + } + + /** + * Status which tells whether key exists in cache or not. + */ + public enum CacheStatus { + EXISTS, // When key exists in cache. + + NOT_EXIST, // We guarantee that it does not exist. This will be returned + // when the key does not exist in cache, when cache clean up policy is + // NEVER. + MAY_EXIST // This will be returned when the key does not exist in + // cache, when cache clean up policy is MANUAL. So caller need to check + // if it might exist in it's rocksdb table. + } +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/CacheValue.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheValue.java similarity index 94% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/CacheValue.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheValue.java index 34f77ae175295..de9fe0d95f3d4 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/CacheValue.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheValue.java @@ -16,7 +16,7 @@ * limitations under the License. */ -package org.apache.hadoop.utils.db.cache; +package org.apache.hadoop.hdds.utils.db.cache; import com.google.common.base.Optional; @@ -36,7 +36,7 @@ public CacheValue(Optional value, long epoch) { this.epoch = epoch; } - public VALUE getValue() { + public VALUE getCacheValue() { return value.orNull(); } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/EpochEntry.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/EpochEntry.java similarity index 97% rename from hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/EpochEntry.java rename to hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/EpochEntry.java index 6966b3d92d61c..7235202b9a4ca 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/EpochEntry.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/EpochEntry.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db.cache; +package org.apache.hadoop.hdds.utils.db.cache; import java.util.Objects; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/TableCache.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/TableCache.java new file mode 100644 index 0000000000000..de5a07978f51e --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/TableCache.java @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.hdds.utils.db.cache; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Evolving; + +import java.util.Iterator; +import java.util.Map; + +/** + * Cache used for RocksDB tables. + * @param + * @param + */ + +@Private +@Evolving +public interface TableCache { + + /** + * Return the value for the key if it is present, otherwise return null. + * @param cacheKey + * @return CACHEVALUE + */ + CACHEVALUE get(CACHEKEY cacheKey); + + /** + * This method should be called for tables with cache cleanup policy + * {@link TableCacheImpl.CacheCleanupPolicy#NEVER} after system restart to + * fill up the cache. + * @param cacheKey + * @param cacheValue + */ + void loadInitial(CACHEKEY cacheKey, CACHEVALUE cacheValue); + + /** + * Add an entry to the cache, if the key already exists it overrides. + * @param cacheKey + * @param value + */ + void put(CACHEKEY cacheKey, CACHEVALUE value); + + /** + * Removes all the entries from the cache which are having epoch value less + * than or equal to specified epoch value. + * + * If clean up policy is NEVER, this is a do nothing operation. + * If clean up policy is MANUAL, it is caller responsibility to cleanup the + * cache before calling cleanup. + * @param epoch + */ + void cleanup(long epoch); + + /** + * Return the size of the cache. + * @return size + */ + int size(); + + /** + * Return an iterator for the cache. + * @return iterator of the underlying cache for the table. + */ + Iterator> iterator(); + + /** + * Check key exist in cache or not. + * + * If it exists return CacheResult with value and status as + * {@link CacheResult.CacheStatus#EXISTS} + * + * If it does not exist: + * If cache clean up policy is + * {@link TableCacheImpl.CacheCleanupPolicy#NEVER} it means table cache is + * full cache. It return's {@link CacheResult} with null + * and status as {@link CacheResult.CacheStatus#NOT_EXIST}. + * + * If cache clean up policy is + * {@link TableCacheImpl.CacheCleanupPolicy#MANUAL} it means + * table cache is partial cache. It return's {@link CacheResult} with + * null and status as MAY_EXIST. + * + * @param cachekey + */ + CacheResult lookup(CACHEKEY cachekey); + +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/TableCacheImpl.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/TableCacheImpl.java new file mode 100644 index 0000000000000..3e6999a49cfaa --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/TableCacheImpl.java @@ -0,0 +1,169 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.hdds.utils.db.cache; + +import java.util.Iterator; +import java.util.Map; +import java.util.NavigableSet; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentSkipListMap; +import java.util.concurrent.ConcurrentSkipListSet; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.ThreadFactory; + +import com.google.common.util.concurrent.ThreadFactoryBuilder; +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Evolving; + +/** + * Cache implementation for the table. Depending on the cache clean up policy + * this cache will be full cache or partial cache. + * + * If cache cleanup policy is set as {@link CacheCleanupPolicy#MANUAL}, + * this will be a partial cache. + * + * If cache cleanup policy is set as {@link CacheCleanupPolicy#NEVER}, + * this will be a full cache. + */ +@Private +@Evolving +public class TableCacheImpl implements TableCache { + + private final Map cache; + private final NavigableSet> epochEntries; + private ExecutorService executorService; + private CacheCleanupPolicy cleanupPolicy; + + + + public TableCacheImpl(CacheCleanupPolicy cleanupPolicy) { + + // As for full table cache only we need elements to be inserted in sorted + // manner, so that list will be easy. For other we can go with Hash map. + if (cleanupPolicy == CacheCleanupPolicy.NEVER) { + cache = new ConcurrentSkipListMap<>(); + } else { + cache = new ConcurrentHashMap<>(); + } + epochEntries = new ConcurrentSkipListSet<>(); + // Created a singleThreadExecutor, so one cleanup will be running at a + // time. + ThreadFactory build = new ThreadFactoryBuilder().setDaemon(true) + .setNameFormat("PartialTableCache Cleanup Thread - %d").build(); + executorService = Executors.newSingleThreadExecutor(build); + this.cleanupPolicy = cleanupPolicy; + } + + @Override + public CACHEVALUE get(CACHEKEY cachekey) { + return cache.get(cachekey); + } + + @Override + public void loadInitial(CACHEKEY cacheKey, CACHEVALUE cacheValue) { + // No need to add entry to epochEntries. Adding to cache is required during + // normal put operation. + cache.put(cacheKey, cacheValue); + } + + @Override + public void put(CACHEKEY cacheKey, CACHEVALUE value) { + cache.put(cacheKey, value); + epochEntries.add(new EpochEntry<>(value.getEpoch(), cacheKey)); + } + + @Override + public void cleanup(long epoch) { + executorService.submit(() -> evictCache(epoch, cleanupPolicy)); + } + + @Override + public int size() { + return cache.size(); + } + + @Override + public Iterator> iterator() { + return cache.entrySet().iterator(); + } + + private void evictCache(long epoch, CacheCleanupPolicy cacheCleanupPolicy) { + EpochEntry currentEntry = null; + for (Iterator> iterator = epochEntries.iterator(); + iterator.hasNext();) { + currentEntry = iterator.next(); + CACHEKEY cachekey = currentEntry.getCachekey(); + CacheValue cacheValue = cache.computeIfPresent(cachekey, ((k, v) -> { + if (cleanupPolicy == CacheCleanupPolicy.MANUAL) { + if (v.getEpoch() <= epoch) { + iterator.remove(); + return null; + } + } else if (cleanupPolicy == CacheCleanupPolicy.NEVER) { + // Remove only entries which are marked for delete. + if (v.getEpoch() <= epoch && v.getCacheValue() == null) { + iterator.remove(); + return null; + } + } + return v; + })); + // If currentEntry epoch is greater than epoch, we have deleted all + // entries less than specified epoch. So, we can break. + if (cacheValue != null && cacheValue.getEpoch() >= epoch) { + break; + } + } + } + + public CacheResult lookup(CACHEKEY cachekey) { + + CACHEVALUE cachevalue = cache.get(cachekey); + if (cachevalue == null) { + if (cleanupPolicy == CacheCleanupPolicy.NEVER) { + return new CacheResult<>(CacheResult.CacheStatus.NOT_EXIST, null); + } else { + return new CacheResult<>(CacheResult.CacheStatus.MAY_EXIST, + null); + } + } else { + if (cachevalue.getCacheValue() != null) { + return new CacheResult<>(CacheResult.CacheStatus.EXISTS, cachevalue); + } else { + // When entity is marked for delete, cacheValue will be set to null. + // In that case we can return NOT_EXIST irrespective of cache cleanup + // policy. + return new CacheResult<>(CacheResult.CacheStatus.NOT_EXIST, null); + } + } + } + + /** + * Cleanup policies for table cache. + */ + public enum CacheCleanupPolicy { + NEVER, // Cache will not be cleaned up. This mean's the table maintains + // full cache. + MANUAL // Cache will be cleaned up, once after flushing to DB. It is + // caller's responsibility to flush to DB, before calling cleanup cache. + } +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/package-info.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/package-info.java new file mode 100644 index 0000000000000..eb9c5b9da8f32 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/package-info.java @@ -0,0 +1,18 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.utils.db.cache; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/package-info.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/package-info.java new file mode 100644 index 0000000000000..8b56bffa777d8 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/package-info.java @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +/** + * Database interfaces for Ozone. + */ +package org.apache.hadoop.hdds.utils.db; \ No newline at end of file diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/package-info.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/package-info.java new file mode 100644 index 0000000000000..4576dc82a8ea2 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/package-info.java @@ -0,0 +1,18 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.utils; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneAcl.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneAcl.java deleted file mode 100644 index 1827b23bf15f1..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneAcl.java +++ /dev/null @@ -1,233 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - - -package org.apache.hadoop.ozone; - -import java.util.Objects; - -/** - * OzoneACL classes define bucket ACLs used in OZONE. - * - * ACLs in Ozone follow this pattern. - *

      - *
    • user:name:rw - *
    • group:name:rw - *
    • world::rw - *
    - */ -public class OzoneAcl { - private OzoneACLType type; - private String name; - private OzoneACLRights rights; - - /** - * Constructor for OzoneAcl. - */ - public OzoneAcl() { - } - - /** - * Constructor for OzoneAcl. - * - * @param type - Type - * @param name - Name of user - * @param rights - Rights - */ - public OzoneAcl(OzoneACLType type, String name, OzoneACLRights rights) { - this.name = name; - this.rights = rights; - this.type = type; - if (type == OzoneACLType.WORLD && name.length() != 0) { - throw new IllegalArgumentException("Unexpected name part in world type"); - } - if (((type == OzoneACLType.USER) || (type == OzoneACLType.GROUP)) - && (name.length() == 0)) { - throw new IllegalArgumentException("User or group name is required"); - } - } - - /** - * Parses an ACL string and returns the ACL object. - * - * @param acl - Acl String , Ex. user:anu:rw - * - * @return - Ozone ACLs - */ - public static OzoneAcl parseAcl(String acl) throws IllegalArgumentException { - if ((acl == null) || acl.isEmpty()) { - throw new IllegalArgumentException("ACLs cannot be null or empty"); - } - String[] parts = acl.trim().split(":"); - if (parts.length < 3) { - throw new IllegalArgumentException("ACLs are not in expected format"); - } - - OzoneACLType aclType = OzoneACLType.valueOf(parts[0].toUpperCase()); - OzoneACLRights rights = OzoneACLRights.getACLRight(parts[2].toLowerCase()); - - // TODO : Support sanitation of these user names by calling into - // userAuth Interface. - return new OzoneAcl(aclType, parts[1], rights); - } - - @Override - public String toString() { - return type + ":" + name + ":" + OzoneACLRights.getACLRightsString(rights); - } - - /** - * Returns a hash code value for the object. This method is - * supported for the benefit of hash tables. - * - * @return a hash code value for this object. - * - * @see Object#equals(Object) - * @see System#identityHashCode - */ - @Override - public int hashCode() { - return Objects.hash(this.getName(), this.getRights().toString(), - this.getType().toString()); - } - - /** - * Returns name. - * - * @return name - */ - public String getName() { - return name; - } - - /** - * Returns Rights. - * - * @return - Rights - */ - public OzoneACLRights getRights() { - return rights; - } - - /** - * Returns Type. - * - * @return type - */ - public OzoneACLType getType() { - return type; - } - - /** - * Indicates whether some other object is "equal to" this one. - * - * @param obj the reference object with which to compare. - * - * @return {@code true} if this object is the same as the obj - * argument; {@code false} otherwise. - */ - @Override - public boolean equals(Object obj) { - if (obj == null) { - return false; - } - if (getClass() != obj.getClass()) { - return false; - } - OzoneAcl otherAcl = (OzoneAcl) obj; - return otherAcl.getName().equals(this.getName()) && - otherAcl.getRights() == this.getRights() && - otherAcl.getType() == this.getType(); - } - - /** - * ACL types. - */ - public enum OzoneACLType { - USER(OzoneConsts.OZONE_ACL_USER_TYPE), - GROUP(OzoneConsts.OZONE_ACL_GROUP_TYPE), - WORLD(OzoneConsts.OZONE_ACL_WORLD_TYPE); - - /** - * String value for this Enum. - */ - private final String value; - - /** - * Init OzoneACLtypes enum. - * - * @param val String type for this enum. - */ - OzoneACLType(String val) { - value = val; - } - } - - /** - * ACL rights. - */ - public enum OzoneACLRights { - READ, WRITE, READ_WRITE; - - /** - * Returns the ACL rights based on passed in String. - * - * @param type ACL right string - * - * @return OzoneACLRights - */ - public static OzoneACLRights getACLRight(String type) { - if (type == null || type.isEmpty()) { - throw new IllegalArgumentException("ACL right cannot be empty"); - } - - switch (type) { - case OzoneConsts.OZONE_ACL_READ: - return OzoneACLRights.READ; - case OzoneConsts.OZONE_ACL_WRITE: - return OzoneACLRights.WRITE; - case OzoneConsts.OZONE_ACL_READ_WRITE: - case OzoneConsts.OZONE_ACL_WRITE_READ: - return OzoneACLRights.READ_WRITE; - default: - throw new IllegalArgumentException("ACL right is not recognized"); - } - - } - - /** - * Returns String representation of ACL rights. - * @param acl OzoneACLRights - * @return String representation of acl - */ - public static String getACLRightsString(OzoneACLRights acl) { - switch(acl) { - case READ: - return OzoneConsts.OZONE_ACL_READ; - case WRITE: - return OzoneConsts.OZONE_ACL_WRITE; - case READ_WRITE: - return OzoneConsts.OZONE_ACL_READ_WRITE; - default: - throw new IllegalArgumentException("ACL right is not recognized"); - } - } - - } - -} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java index 1463c43e830f3..3f7d0b915d5d5 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java @@ -1,4 +1,4 @@ - /** +/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information @@ -29,7 +29,7 @@ import java.util.concurrent.TimeUnit; - /** +/** * This class contains constants for configuration keys used in Ozone. */ @InterfaceAudience.Public @@ -41,6 +41,8 @@ public final class OzoneConfigKeys { "dfs.container.ipc"; public static final int DFS_CONTAINER_IPC_PORT_DEFAULT = 9859; + public static final String OZONE_METADATA_DIRS = "ozone.metadata.dirs"; + /** * * When set to true, allocate a random free port for ozone container, @@ -118,9 +120,10 @@ public final class OzoneConfigKeys { * */ public static final String OZONE_ADMINISTRATORS = "ozone.administrators"; - - public static final String OZONE_CLIENT_PROTOCOL = - "ozone.client.protocol"; + /** + * Used only for testing purpose. Results in making every user an admin. + * */ + public static final String OZONE_ADMINISTRATORS_WILDCARD = "*"; public static final String OZONE_CLIENT_STREAM_BUFFER_FLUSH_SIZE = "ozone.client.stream.buffer.flush.size"; @@ -322,6 +325,15 @@ public final class OzoneConfigKeys { public static final String DFS_CONTAINER_RATIS_LOG_APPENDER_QUEUE_BYTE_LIMIT_DEFAULT = ScmConfigKeys.DFS_CONTAINER_RATIS_LOG_APPENDER_QUEUE_BYTE_LIMIT_DEFAULT; + public static final String DFS_CONTAINER_RATIS_LOG_PURGE_GAP = + ScmConfigKeys.DFS_CONTAINER_RATIS_LOG_PURGE_GAP; + public static final int DFS_CONTAINER_RATIS_LOG_PURGE_GAP_DEFAULT = + ScmConfigKeys.DFS_CONTAINER_RATIS_LOG_PURGE_GAP_DEFAULT; + public static final String DFS_CONTAINER_RATIS_LEADER_NUM_PENDING_REQUESTS = + ScmConfigKeys.DFS_CONTAINER_RATIS_LEADER_NUM_PENDING_REQUESTS; + public static final int + DFS_CONTAINER_RATIS_LEADER_NUM_PENDING_REQUESTS_DEFAULT = + ScmConfigKeys.DFS_CONTAINER_RATIS_LEADER_NUM_PENDING_REQUESTS_DEFAULT; public static final String DFS_RATIS_SERVER_REQUEST_TIMEOUT_DURATION_KEY = ScmConfigKeys.DFS_RATIS_SERVER_REQUEST_TIMEOUT_DURATION_KEY; public static final TimeDuration @@ -351,12 +363,12 @@ public final class OzoneConfigKeys { HDDS_DATANODE_STORAGE_UTILIZATION_WARNING_THRESHOLD = "hdds.datanode.storage.utilization.warning.threshold"; public static final double - HDDS_DATANODE_STORAGE_UTILIZATION_WARNING_THRESHOLD_DEFAULT = 0.95; + HDDS_DATANODE_STORAGE_UTILIZATION_WARNING_THRESHOLD_DEFAULT = 0.75; public static final String HDDS_DATANODE_STORAGE_UTILIZATION_CRITICAL_THRESHOLD = "hdds.datanode.storage.utilization.critical.threshold"; public static final double - HDDS_DATANODE_STORAGE_UTILIZATION_CRITICAL_THRESHOLD_DEFAULT = 0.75; + HDDS_DATANODE_STORAGE_UTILIZATION_CRITICAL_THRESHOLD_DEFAULT = 0.95; public static final String OZONE_SECURITY_ENABLED_KEY = "ozone.security.enabled"; @@ -370,7 +382,7 @@ public final class OzoneConfigKeys { */ public static final String OZONE_CLIENT_CHECKSUM_TYPE = "ozone.client.checksum.type"; - public static final String OZONE_CLIENT_CHECKSUM_TYPE_DEFAULT = "SHA256"; + public static final String OZONE_CLIENT_CHECKSUM_TYPE_DEFAULT = "CRC32"; public static final String OZONE_CLIENT_BYTES_PER_CHECKSUM = "ozone.client.bytes.per.checksum"; public static final String OZONE_CLIENT_BYTES_PER_CHECKSUM_DEFAULT = "1MB"; @@ -384,6 +396,8 @@ public final class OzoneConfigKeys { "ozone.acl.authorizer.class"; public static final String OZONE_ACL_AUTHORIZER_CLASS_DEFAULT = "org.apache.hadoop.ozone.security.acl.OzoneAccessAuthorizer"; + public static final String OZONE_ACL_AUTHORIZER_CLASS_NATIVE = + "org.apache.hadoop.ozone.security.acl.OzoneNativeAuthorizer"; public static final String OZONE_ACL_ENABLED = "ozone.acl.enabled"; public static final boolean OZONE_ACL_ENABLED_DEFAULT = @@ -435,6 +449,13 @@ public final class OzoneConfigKeys { OZONE_FREON_HTTP_KERBEROS_KEYTAB_FILE_KEY = "ozone.freon.http.kerberos.keytab"; + public static final String OZONE_NETWORK_TOPOLOGY_AWARE_READ_KEY = + "ozone.network.topology.aware.read"; + public static final boolean OZONE_NETWORK_TOPOLOGY_AWARE_READ_DEFAULT = false; + + public static final String OZONE_MANAGER_FAIR_LOCK = "ozone.om.lock.fair"; + public static final boolean OZONE_MANAGER_FAIR_LOCK_DEFAULT = false; + /** * There is no need to instantiate this class. */ diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java index ce1e97e34d918..9817d877eb55a 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java @@ -52,12 +52,11 @@ public final class OzoneConsts { public static final String OZONE_ACL_USER_TYPE = "user"; public static final String OZONE_ACL_GROUP_TYPE = "group"; public static final String OZONE_ACL_WORLD_TYPE = "world"; + public static final String OZONE_ACL_ANONYMOUS_TYPE = "anonymous"; public static final String OZONE_ACL_IP_TYPE = "ip"; public static final String OZONE_ACL_READ = "r"; public static final String OZONE_ACL_WRITE = "w"; - public static final String OZONE_ACL_READ_WRITE = "rw"; - public static final String OZONE_ACL_WRITE_READ = "wr"; public static final String OZONE_ACL_DELETE = "d"; public static final String OZONE_ACL_LIST = "l"; public static final String OZONE_ACL_ALL = "a"; @@ -77,6 +76,12 @@ public final class OzoneConsts { public static final String OZONE_USER = "user"; public static final String OZONE_REQUEST = "request"; + // OM Http server endpoints + public static final String OZONE_OM_SERVICE_LIST_HTTP_ENDPOINT = + "/serviceList"; + public static final String OZONE_OM_DB_CHECKPOINT_HTTP_ENDPOINT = + "/dbCheckpoint"; + // Ozone File System scheme public static final String OZONE_URI_SCHEME = "o3fs"; @@ -114,6 +119,7 @@ public final class OzoneConsts { public static final String DN_CONTAINER_DB = "-dn-"+ CONTAINER_DB_SUFFIX; public static final String DELETED_BLOCK_DB = "deletedBlock.db"; public static final String OM_DB_NAME = "om.db"; + public static final String OM_DB_BACKUP_PREFIX = "om.db.backup."; public static final String OM_DB_CHECKPOINTS_DIR_NAME = "om.db.checkpoints"; public static final String OZONE_MANAGER_TOKEN_DB_NAME = "om-token.db"; public static final String SCM_DB_NAME = "scm.db"; @@ -165,6 +171,8 @@ public static Versioning getVersioning(boolean versioning) { public static final String OM_USER_PREFIX = "$"; public static final String OM_S3_PREFIX ="S3:"; public static final String OM_S3_VOLUME_PREFIX = "s3"; + public static final String OM_S3_SECRET = "S3Secret:"; + public static final String OM_PREFIX = "Prefix:"; /** * Max chunk size limit. @@ -231,6 +239,8 @@ private OzoneConsts() { public static final String KEY = "key"; public static final String QUOTA = "quota"; public static final String QUOTA_IN_BYTES = "quotaInBytes"; + public static final String OBJECT_ID = "objectID"; + public static final String UPDATE_ID = "updateID"; public static final String CLIENT_ID = "clientID"; public static final String OWNER = "owner"; public static final String ADMIN = "admin"; @@ -240,6 +250,7 @@ private OzoneConsts() { public static final String MAX_KEYS = "maxKeys"; public static final String PREFIX = "prefix"; public static final String KEY_PREFIX = "keyPrefix"; + public static final String ACL = "acl"; public static final String ACLS = "acls"; public static final String USER_ACL = "userAcl"; public static final String ADD_ACLS = "addAcls"; @@ -247,6 +258,7 @@ private OzoneConsts() { public static final String MAX_NUM_OF_BUCKETS = "maxNumOfBuckets"; public static final String TO_KEY_NAME = "toKeyName"; public static final String STORAGE_TYPE = "storageType"; + public static final String RESOURCE_TYPE = "resourceType"; public static final String IS_VERSION_ENABLED = "isVersionEnabled"; public static final String CREATION_TIME = "creationTime"; public static final String DATA_SIZE = "dataSize"; @@ -257,6 +269,8 @@ private OzoneConsts() { public static final String UPLOAD_ID = "uploadID"; public static final String PART_NUMBER_MARKER = "partNumberMarker"; public static final String MAX_PARTS = "maxParts"; + public static final String S3_BUCKET = "s3Bucket"; + public static final String S3_GETSECRET_USER = "S3GetSecretUser"; @@ -287,4 +301,27 @@ private OzoneConsts() { // OM Ratis snapshot file to store the last applied index public static final String OM_RATIS_SNAPSHOT_INDEX = "ratisSnapshotIndex"; + + // OM Http request parameter to be used while downloading DB checkpoint + // from OM leader to follower + public static final String OM_RATIS_SNAPSHOT_BEFORE_DB_CHECKPOINT = + "snapshotBeforeCheckpoint"; + + public static final String JAVA_TMP_DIR = "java.io.tmpdir"; + public static final String LOCALHOST = "localhost"; + + + public static final int S3_BUCKET_MIN_LENGTH = 3; + public static final int S3_BUCKET_MAX_LENGTH = 64; + + //GDPR + public static final String GDPR_FLAG = "gdprEnabled"; + public static final String GDPR_ALGORITHM_NAME = "AES"; + public static final int GDPR_DEFAULT_RANDOM_SECRET_LENGTH = 16; + public static final String GDPR_CHARSET = "UTF-8"; + public static final String GDPR_LENGTH = "length"; + public static final String GDPR_SECRET = "secret"; + public static final String GDPR_ALGORITHM = "algorithm"; + + } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/AuditLogger.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/AuditLogger.java index 9357774313bcd..ee6f45dadb4c4 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/AuditLogger.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/AuditLogger.java @@ -73,4 +73,14 @@ public void logReadFailure(AuditMessage msg) { msg.getThrowable()); } + public void logWrite(AuditMessage auditMessage) { + if (auditMessage.getThrowable() == null) { + this.logger.logIfEnabled(FQCN, Level.INFO, WRITE_MARKER, auditMessage, + auditMessage.getThrowable()); + } else { + this.logger.logIfEnabled(FQCN, Level.ERROR, WRITE_MARKER, auditMessage, + auditMessage.getThrowable()); + } + } + } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/SCMAction.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/SCMAction.java index f6629551e17ea..d03ad157220af 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/SCMAction.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/SCMAction.java @@ -33,9 +33,15 @@ public enum SCMAction implements AuditAction { LIST_CONTAINER, LIST_PIPELINE, CLOSE_PIPELINE, + ACTIVATE_PIPELINE, + DEACTIVATE_PIPELINE, DELETE_CONTAINER, IN_SAFE_MODE, - FORCE_EXIT_SAFE_MODE; + FORCE_EXIT_SAFE_MODE, + SORT_DATANODE, + START_REPLICATION_MANAGER, + STOP_REPLICATION_MANAGER, + GET_REPLICATION_MANAGER_STATUS; @Override public String getAction() { diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Checksum.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Checksum.java index 1a359fe5c4d39..0e70515a492df 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Checksum.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Checksum.java @@ -225,15 +225,17 @@ private byte[] computeMD5Checksum(byte[] data, int offset, int len) { /** * Computes the ChecksumData for the input data and verifies that it - * matches with that of the input checksumData. + * matches with that of the input checksumData, starting from index + * startIndex. * @param byteString input data * @param checksumData checksumData to match with + * @param startIndex index of first checksum in checksumData to match with + * data's computed checksum. * @throws OzoneChecksumException is thrown if checksums do not match */ - public static boolean verifyChecksum( - ByteString byteString, ChecksumData checksumData) - throws OzoneChecksumException { - return verifyChecksum(byteString.toByteArray(), checksumData); + public static boolean verifyChecksum(ByteString byteString, + ChecksumData checksumData, int startIndex) throws OzoneChecksumException { + return verifyChecksum(byteString.toByteArray(), checksumData, startIndex); } /** @@ -245,6 +247,20 @@ public static boolean verifyChecksum( */ public static boolean verifyChecksum(byte[] data, ChecksumData checksumData) throws OzoneChecksumException { + return verifyChecksum(data, checksumData, 0); + } + + /** + * Computes the ChecksumData for the input data and verifies that it + * matches with that of the input checksumData. + * @param data input data + * @param checksumData checksumData to match with + * @param startIndex index of first checksum in checksumData to match with + * data's computed checksum. + * @throws OzoneChecksumException is thrown if checksums do not match + */ + public static boolean verifyChecksum(byte[] data, ChecksumData checksumData, + int startIndex) throws OzoneChecksumException { ChecksumType checksumType = checksumData.getChecksumType(); if (checksumType == ChecksumType.NONE) { // Checksum is set to NONE. No further verification is required. @@ -256,7 +272,8 @@ public static boolean verifyChecksum(byte[] data, ChecksumData checksumData) ChecksumData computedChecksumData = checksum.computeChecksum(data, 0, data.length); - return checksumData.verifyChecksumDataMatches(computedChecksumData); + return checksumData.verifyChecksumDataMatches(computedChecksumData, + startIndex); } /** diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumByteBuffer.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumByteBuffer.java new file mode 100644 index 0000000000000..7ce643db4711c --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumByteBuffer.java @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + * Some portions of this file Copyright (c) 2004-2006 Intel Corportation + * and licensed under the BSD license. + */ +package org.apache.hadoop.ozone.common; + +import org.apache.ratis.util.Preconditions; + +import java.nio.ByteBuffer; +import java.util.zip.Checksum; + +/** + * A sub-interface of {@link Checksum} + * with a method to update checksum from a {@link ByteBuffer}. + */ +public interface ChecksumByteBuffer extends Checksum { + /** + * Updates the current checksum with the specified bytes in the buffer. + * Upon return, the buffer's position will be equal to its limit. + * + * @param buffer the bytes to update the checksum with + */ + void update(ByteBuffer buffer); + + @Override + default void update(byte[] b, int off, int len) { + update(ByteBuffer.wrap(b, off, len).asReadOnlyBuffer()); + } + + /** + * An abstract class implementing {@link ChecksumByteBuffer} + * with a 32-bit checksum and a lookup table. + */ + @SuppressWarnings("innerassignment") + abstract class CrcIntTable implements ChecksumByteBuffer { + /** Current CRC value with bit-flipped. */ + private int crc; + + CrcIntTable() { + reset(); + Preconditions.assertTrue(getTable().length == 8 * (1 << 8)); + } + + abstract int[] getTable(); + + @Override + public final long getValue() { + return (~crc) & 0xffffffffL; + } + + @Override + public final void reset() { + crc = 0xffffffff; + } + + @Override + public final void update(int b) { + crc = (crc >>> 8) ^ getTable()[(((crc ^ b) << 24) >>> 24)]; + } + + @Override + public final void update(ByteBuffer b) { + crc = update(crc, b, getTable()); + } + + private static int update(int crc, ByteBuffer b, int[] table) { + for(; b.remaining() > 7;) { + final int c0 = (b.get() ^ crc) & 0xff; + final int c1 = (b.get() ^ (crc >>>= 8)) & 0xff; + final int c2 = (b.get() ^ (crc >>>= 8)) & 0xff; + final int c3 = (b.get() ^ (crc >>> 8)) & 0xff; + crc = (table[0x700 + c0] ^ table[0x600 + c1]) + ^ (table[0x500 + c2] ^ table[0x400 + c3]); + + final int c4 = b.get() & 0xff; + final int c5 = b.get() & 0xff; + final int c6 = b.get() & 0xff; + final int c7 = b.get() & 0xff; + + crc ^= (table[0x300 + c4] ^ table[0x200 + c5]) + ^ (table[0x100 + c6] ^ table[c7]); + } + + // loop unroll - duff's device style + switch (b.remaining()) { + case 7: + crc = (crc >>> 8) ^ table[((crc ^ b.get()) & 0xff)]; + case 6: + crc = (crc >>> 8) ^ table[((crc ^ b.get()) & 0xff)]; + case 5: + crc = (crc >>> 8) ^ table[((crc ^ b.get()) & 0xff)]; + case 4: + crc = (crc >>> 8) ^ table[((crc ^ b.get()) & 0xff)]; + case 3: + crc = (crc >>> 8) ^ table[((crc ^ b.get()) & 0xff)]; + case 2: + crc = (crc >>> 8) ^ table[((crc ^ b.get()) & 0xff)]; + case 1: + crc = (crc >>> 8) ^ table[((crc ^ b.get()) & 0xff)]; + default: // noop + } + + return crc; + } + } +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumData.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumData.java index dafa0e32a25b3..4a927fbae6cd9 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumData.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumData.java @@ -40,9 +40,14 @@ public class ChecksumData { private List checksums; public ChecksumData(ChecksumType checksumType, int bytesPerChecksum) { + this(checksumType, bytesPerChecksum, Lists.newArrayList()); + } + + public ChecksumData(ChecksumType checksumType, int bytesPerChecksum, + List checksums) { this.type = checksumType; this.bytesPerChecksum = bytesPerChecksum; - this.checksums = Lists.newArrayList(); + this.checksums = checksums; } /** @@ -111,13 +116,20 @@ public static ChecksumData getFromProtoBuf( } /** - * Verify that this ChecksumData matches with the input ChecksumData. + * Verify that this ChecksumData from startIndex to endIndex matches with the + * provided ChecksumData. + * The checksum at startIndex of this ChecksumData will be matched with the + * checksum at index 0 of the provided ChecksumData, and checksum at + * (startIndex + 1) of this ChecksumData with checksum at index 1 of + * provided ChecksumData and so on. * @param that the ChecksumData to match with + * @param startIndex index of the first checksum from this ChecksumData + * which will be used to compare checksums * @return true if checksums match * @throws OzoneChecksumException */ - public boolean verifyChecksumDataMatches(ChecksumData that) throws - OzoneChecksumException { + public boolean verifyChecksumDataMatches(ChecksumData that, int startIndex) + throws OzoneChecksumException { // pre checks if (this.checksums.size() == 0) { @@ -130,18 +142,22 @@ public boolean verifyChecksumDataMatches(ChecksumData that) throws "checksums"); } - if (this.checksums.size() != that.checksums.size()) { - throw new OzoneChecksumException("Original and Computed checksumData's " + - "has different number of checksums"); - } + int numChecksums = that.checksums.size(); - // Verify that checksum matches at each index - for (int index = 0; index < this.checksums.size(); index++) { - if (!matchChecksumAtIndex(this.checksums.get(index), - that.checksums.get(index))) { - // checksum mismatch. throw exception. - throw new OzoneChecksumException(index); + try { + // Verify that checksum matches at each index + for (int index = 0; index < numChecksums; index++) { + if (!matchChecksumAtIndex(this.checksums.get(startIndex + index), + that.checksums.get(index))) { + // checksum mismatch. throw exception. + throw new OzoneChecksumException(index); + } } + } catch (ArrayIndexOutOfBoundsException e) { + throw new OzoneChecksumException("Computed checksum has " + + numChecksums + " number of checksums. Original checksum has " + + (this.checksums.size() - startIndex) + " number of checksums " + + "starting from index " + startIndex); } return true; } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/PureJavaCrc32ByteBuffer.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/PureJavaCrc32ByteBuffer.java new file mode 100644 index 0000000000000..0d1f6307501a3 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/PureJavaCrc32ByteBuffer.java @@ -0,0 +1,556 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ozone.common; + +/** + * Similar to {@link org.apache.hadoop.util.PureJavaCrc32} + * except that this class implement {@link ChecksumByteBuffer}. + */ +final class PureJavaCrc32ByteBuffer extends ChecksumByteBuffer.CrcIntTable { + @Override + int[] getTable() { + return T; + } + + /** + * CRC-32 lookup table generated by the polynomial 0xEDB88320. + * See also org.apache.hadoop.util.TestPureJavaCrc32.Table. + */ + private static final int[] T = { + /* T8_0 */ + 0x00000000, 0x77073096, 0xEE0E612C, 0x990951BA, + 0x076DC419, 0x706AF48F, 0xE963A535, 0x9E6495A3, + 0x0EDB8832, 0x79DCB8A4, 0xE0D5E91E, 0x97D2D988, + 0x09B64C2B, 0x7EB17CBD, 0xE7B82D07, 0x90BF1D91, + 0x1DB71064, 0x6AB020F2, 0xF3B97148, 0x84BE41DE, + 0x1ADAD47D, 0x6DDDE4EB, 0xF4D4B551, 0x83D385C7, + 0x136C9856, 0x646BA8C0, 0xFD62F97A, 0x8A65C9EC, + 0x14015C4F, 0x63066CD9, 0xFA0F3D63, 0x8D080DF5, + 0x3B6E20C8, 0x4C69105E, 0xD56041E4, 0xA2677172, + 0x3C03E4D1, 0x4B04D447, 0xD20D85FD, 0xA50AB56B, + 0x35B5A8FA, 0x42B2986C, 0xDBBBC9D6, 0xACBCF940, + 0x32D86CE3, 0x45DF5C75, 0xDCD60DCF, 0xABD13D59, + 0x26D930AC, 0x51DE003A, 0xC8D75180, 0xBFD06116, + 0x21B4F4B5, 0x56B3C423, 0xCFBA9599, 0xB8BDA50F, + 0x2802B89E, 0x5F058808, 0xC60CD9B2, 0xB10BE924, + 0x2F6F7C87, 0x58684C11, 0xC1611DAB, 0xB6662D3D, + 0x76DC4190, 0x01DB7106, 0x98D220BC, 0xEFD5102A, + 0x71B18589, 0x06B6B51F, 0x9FBFE4A5, 0xE8B8D433, + 0x7807C9A2, 0x0F00F934, 0x9609A88E, 0xE10E9818, + 0x7F6A0DBB, 0x086D3D2D, 0x91646C97, 0xE6635C01, + 0x6B6B51F4, 0x1C6C6162, 0x856530D8, 0xF262004E, + 0x6C0695ED, 0x1B01A57B, 0x8208F4C1, 0xF50FC457, + 0x65B0D9C6, 0x12B7E950, 0x8BBEB8EA, 0xFCB9887C, + 0x62DD1DDF, 0x15DA2D49, 0x8CD37CF3, 0xFBD44C65, + 0x4DB26158, 0x3AB551CE, 0xA3BC0074, 0xD4BB30E2, + 0x4ADFA541, 0x3DD895D7, 0xA4D1C46D, 0xD3D6F4FB, + 0x4369E96A, 0x346ED9FC, 0xAD678846, 0xDA60B8D0, + 0x44042D73, 0x33031DE5, 0xAA0A4C5F, 0xDD0D7CC9, + 0x5005713C, 0x270241AA, 0xBE0B1010, 0xC90C2086, + 0x5768B525, 0x206F85B3, 0xB966D409, 0xCE61E49F, + 0x5EDEF90E, 0x29D9C998, 0xB0D09822, 0xC7D7A8B4, + 0x59B33D17, 0x2EB40D81, 0xB7BD5C3B, 0xC0BA6CAD, + 0xEDB88320, 0x9ABFB3B6, 0x03B6E20C, 0x74B1D29A, + 0xEAD54739, 0x9DD277AF, 0x04DB2615, 0x73DC1683, + 0xE3630B12, 0x94643B84, 0x0D6D6A3E, 0x7A6A5AA8, + 0xE40ECF0B, 0x9309FF9D, 0x0A00AE27, 0x7D079EB1, + 0xF00F9344, 0x8708A3D2, 0x1E01F268, 0x6906C2FE, + 0xF762575D, 0x806567CB, 0x196C3671, 0x6E6B06E7, + 0xFED41B76, 0x89D32BE0, 0x10DA7A5A, 0x67DD4ACC, + 0xF9B9DF6F, 0x8EBEEFF9, 0x17B7BE43, 0x60B08ED5, + 0xD6D6A3E8, 0xA1D1937E, 0x38D8C2C4, 0x4FDFF252, + 0xD1BB67F1, 0xA6BC5767, 0x3FB506DD, 0x48B2364B, + 0xD80D2BDA, 0xAF0A1B4C, 0x36034AF6, 0x41047A60, + 0xDF60EFC3, 0xA867DF55, 0x316E8EEF, 0x4669BE79, + 0xCB61B38C, 0xBC66831A, 0x256FD2A0, 0x5268E236, + 0xCC0C7795, 0xBB0B4703, 0x220216B9, 0x5505262F, + 0xC5BA3BBE, 0xB2BD0B28, 0x2BB45A92, 0x5CB36A04, + 0xC2D7FFA7, 0xB5D0CF31, 0x2CD99E8B, 0x5BDEAE1D, + 0x9B64C2B0, 0xEC63F226, 0x756AA39C, 0x026D930A, + 0x9C0906A9, 0xEB0E363F, 0x72076785, 0x05005713, + 0x95BF4A82, 0xE2B87A14, 0x7BB12BAE, 0x0CB61B38, + 0x92D28E9B, 0xE5D5BE0D, 0x7CDCEFB7, 0x0BDBDF21, + 0x86D3D2D4, 0xF1D4E242, 0x68DDB3F8, 0x1FDA836E, + 0x81BE16CD, 0xF6B9265B, 0x6FB077E1, 0x18B74777, + 0x88085AE6, 0xFF0F6A70, 0x66063BCA, 0x11010B5C, + 0x8F659EFF, 0xF862AE69, 0x616BFFD3, 0x166CCF45, + 0xA00AE278, 0xD70DD2EE, 0x4E048354, 0x3903B3C2, + 0xA7672661, 0xD06016F7, 0x4969474D, 0x3E6E77DB, + 0xAED16A4A, 0xD9D65ADC, 0x40DF0B66, 0x37D83BF0, + 0xA9BCAE53, 0xDEBB9EC5, 0x47B2CF7F, 0x30B5FFE9, + 0xBDBDF21C, 0xCABAC28A, 0x53B39330, 0x24B4A3A6, + 0xBAD03605, 0xCDD70693, 0x54DE5729, 0x23D967BF, + 0xB3667A2E, 0xC4614AB8, 0x5D681B02, 0x2A6F2B94, + 0xB40BBE37, 0xC30C8EA1, 0x5A05DF1B, 0x2D02EF8D, + /* T8_1 */ + 0x00000000, 0x191B3141, 0x32366282, 0x2B2D53C3, + 0x646CC504, 0x7D77F445, 0x565AA786, 0x4F4196C7, + 0xC8D98A08, 0xD1C2BB49, 0xFAEFE88A, 0xE3F4D9CB, + 0xACB54F0C, 0xB5AE7E4D, 0x9E832D8E, 0x87981CCF, + 0x4AC21251, 0x53D92310, 0x78F470D3, 0x61EF4192, + 0x2EAED755, 0x37B5E614, 0x1C98B5D7, 0x05838496, + 0x821B9859, 0x9B00A918, 0xB02DFADB, 0xA936CB9A, + 0xE6775D5D, 0xFF6C6C1C, 0xD4413FDF, 0xCD5A0E9E, + 0x958424A2, 0x8C9F15E3, 0xA7B24620, 0xBEA97761, + 0xF1E8E1A6, 0xE8F3D0E7, 0xC3DE8324, 0xDAC5B265, + 0x5D5DAEAA, 0x44469FEB, 0x6F6BCC28, 0x7670FD69, + 0x39316BAE, 0x202A5AEF, 0x0B07092C, 0x121C386D, + 0xDF4636F3, 0xC65D07B2, 0xED705471, 0xF46B6530, + 0xBB2AF3F7, 0xA231C2B6, 0x891C9175, 0x9007A034, + 0x179FBCFB, 0x0E848DBA, 0x25A9DE79, 0x3CB2EF38, + 0x73F379FF, 0x6AE848BE, 0x41C51B7D, 0x58DE2A3C, + 0xF0794F05, 0xE9627E44, 0xC24F2D87, 0xDB541CC6, + 0x94158A01, 0x8D0EBB40, 0xA623E883, 0xBF38D9C2, + 0x38A0C50D, 0x21BBF44C, 0x0A96A78F, 0x138D96CE, + 0x5CCC0009, 0x45D73148, 0x6EFA628B, 0x77E153CA, + 0xBABB5D54, 0xA3A06C15, 0x888D3FD6, 0x91960E97, + 0xDED79850, 0xC7CCA911, 0xECE1FAD2, 0xF5FACB93, + 0x7262D75C, 0x6B79E61D, 0x4054B5DE, 0x594F849F, + 0x160E1258, 0x0F152319, 0x243870DA, 0x3D23419B, + 0x65FD6BA7, 0x7CE65AE6, 0x57CB0925, 0x4ED03864, + 0x0191AEA3, 0x188A9FE2, 0x33A7CC21, 0x2ABCFD60, + 0xAD24E1AF, 0xB43FD0EE, 0x9F12832D, 0x8609B26C, + 0xC94824AB, 0xD05315EA, 0xFB7E4629, 0xE2657768, + 0x2F3F79F6, 0x362448B7, 0x1D091B74, 0x04122A35, + 0x4B53BCF2, 0x52488DB3, 0x7965DE70, 0x607EEF31, + 0xE7E6F3FE, 0xFEFDC2BF, 0xD5D0917C, 0xCCCBA03D, + 0x838A36FA, 0x9A9107BB, 0xB1BC5478, 0xA8A76539, + 0x3B83984B, 0x2298A90A, 0x09B5FAC9, 0x10AECB88, + 0x5FEF5D4F, 0x46F46C0E, 0x6DD93FCD, 0x74C20E8C, + 0xF35A1243, 0xEA412302, 0xC16C70C1, 0xD8774180, + 0x9736D747, 0x8E2DE606, 0xA500B5C5, 0xBC1B8484, + 0x71418A1A, 0x685ABB5B, 0x4377E898, 0x5A6CD9D9, + 0x152D4F1E, 0x0C367E5F, 0x271B2D9C, 0x3E001CDD, + 0xB9980012, 0xA0833153, 0x8BAE6290, 0x92B553D1, + 0xDDF4C516, 0xC4EFF457, 0xEFC2A794, 0xF6D996D5, + 0xAE07BCE9, 0xB71C8DA8, 0x9C31DE6B, 0x852AEF2A, + 0xCA6B79ED, 0xD37048AC, 0xF85D1B6F, 0xE1462A2E, + 0x66DE36E1, 0x7FC507A0, 0x54E85463, 0x4DF36522, + 0x02B2F3E5, 0x1BA9C2A4, 0x30849167, 0x299FA026, + 0xE4C5AEB8, 0xFDDE9FF9, 0xD6F3CC3A, 0xCFE8FD7B, + 0x80A96BBC, 0x99B25AFD, 0xB29F093E, 0xAB84387F, + 0x2C1C24B0, 0x350715F1, 0x1E2A4632, 0x07317773, + 0x4870E1B4, 0x516BD0F5, 0x7A468336, 0x635DB277, + 0xCBFAD74E, 0xD2E1E60F, 0xF9CCB5CC, 0xE0D7848D, + 0xAF96124A, 0xB68D230B, 0x9DA070C8, 0x84BB4189, + 0x03235D46, 0x1A386C07, 0x31153FC4, 0x280E0E85, + 0x674F9842, 0x7E54A903, 0x5579FAC0, 0x4C62CB81, + 0x8138C51F, 0x9823F45E, 0xB30EA79D, 0xAA1596DC, + 0xE554001B, 0xFC4F315A, 0xD7626299, 0xCE7953D8, + 0x49E14F17, 0x50FA7E56, 0x7BD72D95, 0x62CC1CD4, + 0x2D8D8A13, 0x3496BB52, 0x1FBBE891, 0x06A0D9D0, + 0x5E7EF3EC, 0x4765C2AD, 0x6C48916E, 0x7553A02F, + 0x3A1236E8, 0x230907A9, 0x0824546A, 0x113F652B, + 0x96A779E4, 0x8FBC48A5, 0xA4911B66, 0xBD8A2A27, + 0xF2CBBCE0, 0xEBD08DA1, 0xC0FDDE62, 0xD9E6EF23, + 0x14BCE1BD, 0x0DA7D0FC, 0x268A833F, 0x3F91B27E, + 0x70D024B9, 0x69CB15F8, 0x42E6463B, 0x5BFD777A, + 0xDC656BB5, 0xC57E5AF4, 0xEE530937, 0xF7483876, + 0xB809AEB1, 0xA1129FF0, 0x8A3FCC33, 0x9324FD72, + /* T8_2 */ + 0x00000000, 0x01C26A37, 0x0384D46E, 0x0246BE59, + 0x0709A8DC, 0x06CBC2EB, 0x048D7CB2, 0x054F1685, + 0x0E1351B8, 0x0FD13B8F, 0x0D9785D6, 0x0C55EFE1, + 0x091AF964, 0x08D89353, 0x0A9E2D0A, 0x0B5C473D, + 0x1C26A370, 0x1DE4C947, 0x1FA2771E, 0x1E601D29, + 0x1B2F0BAC, 0x1AED619B, 0x18ABDFC2, 0x1969B5F5, + 0x1235F2C8, 0x13F798FF, 0x11B126A6, 0x10734C91, + 0x153C5A14, 0x14FE3023, 0x16B88E7A, 0x177AE44D, + 0x384D46E0, 0x398F2CD7, 0x3BC9928E, 0x3A0BF8B9, + 0x3F44EE3C, 0x3E86840B, 0x3CC03A52, 0x3D025065, + 0x365E1758, 0x379C7D6F, 0x35DAC336, 0x3418A901, + 0x3157BF84, 0x3095D5B3, 0x32D36BEA, 0x331101DD, + 0x246BE590, 0x25A98FA7, 0x27EF31FE, 0x262D5BC9, + 0x23624D4C, 0x22A0277B, 0x20E69922, 0x2124F315, + 0x2A78B428, 0x2BBADE1F, 0x29FC6046, 0x283E0A71, + 0x2D711CF4, 0x2CB376C3, 0x2EF5C89A, 0x2F37A2AD, + 0x709A8DC0, 0x7158E7F7, 0x731E59AE, 0x72DC3399, + 0x7793251C, 0x76514F2B, 0x7417F172, 0x75D59B45, + 0x7E89DC78, 0x7F4BB64F, 0x7D0D0816, 0x7CCF6221, + 0x798074A4, 0x78421E93, 0x7A04A0CA, 0x7BC6CAFD, + 0x6CBC2EB0, 0x6D7E4487, 0x6F38FADE, 0x6EFA90E9, + 0x6BB5866C, 0x6A77EC5B, 0x68315202, 0x69F33835, + 0x62AF7F08, 0x636D153F, 0x612BAB66, 0x60E9C151, + 0x65A6D7D4, 0x6464BDE3, 0x662203BA, 0x67E0698D, + 0x48D7CB20, 0x4915A117, 0x4B531F4E, 0x4A917579, + 0x4FDE63FC, 0x4E1C09CB, 0x4C5AB792, 0x4D98DDA5, + 0x46C49A98, 0x4706F0AF, 0x45404EF6, 0x448224C1, + 0x41CD3244, 0x400F5873, 0x4249E62A, 0x438B8C1D, + 0x54F16850, 0x55330267, 0x5775BC3E, 0x56B7D609, + 0x53F8C08C, 0x523AAABB, 0x507C14E2, 0x51BE7ED5, + 0x5AE239E8, 0x5B2053DF, 0x5966ED86, 0x58A487B1, + 0x5DEB9134, 0x5C29FB03, 0x5E6F455A, 0x5FAD2F6D, + 0xE1351B80, 0xE0F771B7, 0xE2B1CFEE, 0xE373A5D9, + 0xE63CB35C, 0xE7FED96B, 0xE5B86732, 0xE47A0D05, + 0xEF264A38, 0xEEE4200F, 0xECA29E56, 0xED60F461, + 0xE82FE2E4, 0xE9ED88D3, 0xEBAB368A, 0xEA695CBD, + 0xFD13B8F0, 0xFCD1D2C7, 0xFE976C9E, 0xFF5506A9, + 0xFA1A102C, 0xFBD87A1B, 0xF99EC442, 0xF85CAE75, + 0xF300E948, 0xF2C2837F, 0xF0843D26, 0xF1465711, + 0xF4094194, 0xF5CB2BA3, 0xF78D95FA, 0xF64FFFCD, + 0xD9785D60, 0xD8BA3757, 0xDAFC890E, 0xDB3EE339, + 0xDE71F5BC, 0xDFB39F8B, 0xDDF521D2, 0xDC374BE5, + 0xD76B0CD8, 0xD6A966EF, 0xD4EFD8B6, 0xD52DB281, + 0xD062A404, 0xD1A0CE33, 0xD3E6706A, 0xD2241A5D, + 0xC55EFE10, 0xC49C9427, 0xC6DA2A7E, 0xC7184049, + 0xC25756CC, 0xC3953CFB, 0xC1D382A2, 0xC011E895, + 0xCB4DAFA8, 0xCA8FC59F, 0xC8C97BC6, 0xC90B11F1, + 0xCC440774, 0xCD866D43, 0xCFC0D31A, 0xCE02B92D, + 0x91AF9640, 0x906DFC77, 0x922B422E, 0x93E92819, + 0x96A63E9C, 0x976454AB, 0x9522EAF2, 0x94E080C5, + 0x9FBCC7F8, 0x9E7EADCF, 0x9C381396, 0x9DFA79A1, + 0x98B56F24, 0x99770513, 0x9B31BB4A, 0x9AF3D17D, + 0x8D893530, 0x8C4B5F07, 0x8E0DE15E, 0x8FCF8B69, + 0x8A809DEC, 0x8B42F7DB, 0x89044982, 0x88C623B5, + 0x839A6488, 0x82580EBF, 0x801EB0E6, 0x81DCDAD1, + 0x8493CC54, 0x8551A663, 0x8717183A, 0x86D5720D, + 0xA9E2D0A0, 0xA820BA97, 0xAA6604CE, 0xABA46EF9, + 0xAEEB787C, 0xAF29124B, 0xAD6FAC12, 0xACADC625, + 0xA7F18118, 0xA633EB2F, 0xA4755576, 0xA5B73F41, + 0xA0F829C4, 0xA13A43F3, 0xA37CFDAA, 0xA2BE979D, + 0xB5C473D0, 0xB40619E7, 0xB640A7BE, 0xB782CD89, + 0xB2CDDB0C, 0xB30FB13B, 0xB1490F62, 0xB08B6555, + 0xBBD72268, 0xBA15485F, 0xB853F606, 0xB9919C31, + 0xBCDE8AB4, 0xBD1CE083, 0xBF5A5EDA, 0xBE9834ED, + /* T8_3 */ + 0x00000000, 0xB8BC6765, 0xAA09C88B, 0x12B5AFEE, + 0x8F629757, 0x37DEF032, 0x256B5FDC, 0x9DD738B9, + 0xC5B428EF, 0x7D084F8A, 0x6FBDE064, 0xD7018701, + 0x4AD6BFB8, 0xF26AD8DD, 0xE0DF7733, 0x58631056, + 0x5019579F, 0xE8A530FA, 0xFA109F14, 0x42ACF871, + 0xDF7BC0C8, 0x67C7A7AD, 0x75720843, 0xCDCE6F26, + 0x95AD7F70, 0x2D111815, 0x3FA4B7FB, 0x8718D09E, + 0x1ACFE827, 0xA2738F42, 0xB0C620AC, 0x087A47C9, + 0xA032AF3E, 0x188EC85B, 0x0A3B67B5, 0xB28700D0, + 0x2F503869, 0x97EC5F0C, 0x8559F0E2, 0x3DE59787, + 0x658687D1, 0xDD3AE0B4, 0xCF8F4F5A, 0x7733283F, + 0xEAE41086, 0x525877E3, 0x40EDD80D, 0xF851BF68, + 0xF02BF8A1, 0x48979FC4, 0x5A22302A, 0xE29E574F, + 0x7F496FF6, 0xC7F50893, 0xD540A77D, 0x6DFCC018, + 0x359FD04E, 0x8D23B72B, 0x9F9618C5, 0x272A7FA0, + 0xBAFD4719, 0x0241207C, 0x10F48F92, 0xA848E8F7, + 0x9B14583D, 0x23A83F58, 0x311D90B6, 0x89A1F7D3, + 0x1476CF6A, 0xACCAA80F, 0xBE7F07E1, 0x06C36084, + 0x5EA070D2, 0xE61C17B7, 0xF4A9B859, 0x4C15DF3C, + 0xD1C2E785, 0x697E80E0, 0x7BCB2F0E, 0xC377486B, + 0xCB0D0FA2, 0x73B168C7, 0x6104C729, 0xD9B8A04C, + 0x446F98F5, 0xFCD3FF90, 0xEE66507E, 0x56DA371B, + 0x0EB9274D, 0xB6054028, 0xA4B0EFC6, 0x1C0C88A3, + 0x81DBB01A, 0x3967D77F, 0x2BD27891, 0x936E1FF4, + 0x3B26F703, 0x839A9066, 0x912F3F88, 0x299358ED, + 0xB4446054, 0x0CF80731, 0x1E4DA8DF, 0xA6F1CFBA, + 0xFE92DFEC, 0x462EB889, 0x549B1767, 0xEC277002, + 0x71F048BB, 0xC94C2FDE, 0xDBF98030, 0x6345E755, + 0x6B3FA09C, 0xD383C7F9, 0xC1366817, 0x798A0F72, + 0xE45D37CB, 0x5CE150AE, 0x4E54FF40, 0xF6E89825, + 0xAE8B8873, 0x1637EF16, 0x048240F8, 0xBC3E279D, + 0x21E91F24, 0x99557841, 0x8BE0D7AF, 0x335CB0CA, + 0xED59B63B, 0x55E5D15E, 0x47507EB0, 0xFFEC19D5, + 0x623B216C, 0xDA874609, 0xC832E9E7, 0x708E8E82, + 0x28ED9ED4, 0x9051F9B1, 0x82E4565F, 0x3A58313A, + 0xA78F0983, 0x1F336EE6, 0x0D86C108, 0xB53AA66D, + 0xBD40E1A4, 0x05FC86C1, 0x1749292F, 0xAFF54E4A, + 0x322276F3, 0x8A9E1196, 0x982BBE78, 0x2097D91D, + 0x78F4C94B, 0xC048AE2E, 0xD2FD01C0, 0x6A4166A5, + 0xF7965E1C, 0x4F2A3979, 0x5D9F9697, 0xE523F1F2, + 0x4D6B1905, 0xF5D77E60, 0xE762D18E, 0x5FDEB6EB, + 0xC2098E52, 0x7AB5E937, 0x680046D9, 0xD0BC21BC, + 0x88DF31EA, 0x3063568F, 0x22D6F961, 0x9A6A9E04, + 0x07BDA6BD, 0xBF01C1D8, 0xADB46E36, 0x15080953, + 0x1D724E9A, 0xA5CE29FF, 0xB77B8611, 0x0FC7E174, + 0x9210D9CD, 0x2AACBEA8, 0x38191146, 0x80A57623, + 0xD8C66675, 0x607A0110, 0x72CFAEFE, 0xCA73C99B, + 0x57A4F122, 0xEF189647, 0xFDAD39A9, 0x45115ECC, + 0x764DEE06, 0xCEF18963, 0xDC44268D, 0x64F841E8, + 0xF92F7951, 0x41931E34, 0x5326B1DA, 0xEB9AD6BF, + 0xB3F9C6E9, 0x0B45A18C, 0x19F00E62, 0xA14C6907, + 0x3C9B51BE, 0x842736DB, 0x96929935, 0x2E2EFE50, + 0x2654B999, 0x9EE8DEFC, 0x8C5D7112, 0x34E11677, + 0xA9362ECE, 0x118A49AB, 0x033FE645, 0xBB838120, + 0xE3E09176, 0x5B5CF613, 0x49E959FD, 0xF1553E98, + 0x6C820621, 0xD43E6144, 0xC68BCEAA, 0x7E37A9CF, + 0xD67F4138, 0x6EC3265D, 0x7C7689B3, 0xC4CAEED6, + 0x591DD66F, 0xE1A1B10A, 0xF3141EE4, 0x4BA87981, + 0x13CB69D7, 0xAB770EB2, 0xB9C2A15C, 0x017EC639, + 0x9CA9FE80, 0x241599E5, 0x36A0360B, 0x8E1C516E, + 0x866616A7, 0x3EDA71C2, 0x2C6FDE2C, 0x94D3B949, + 0x090481F0, 0xB1B8E695, 0xA30D497B, 0x1BB12E1E, + 0x43D23E48, 0xFB6E592D, 0xE9DBF6C3, 0x516791A6, + 0xCCB0A91F, 0x740CCE7A, 0x66B96194, 0xDE0506F1, + /* T8_4 */ + 0x00000000, 0x3D6029B0, 0x7AC05360, 0x47A07AD0, + 0xF580A6C0, 0xC8E08F70, 0x8F40F5A0, 0xB220DC10, + 0x30704BC1, 0x0D106271, 0x4AB018A1, 0x77D03111, + 0xC5F0ED01, 0xF890C4B1, 0xBF30BE61, 0x825097D1, + 0x60E09782, 0x5D80BE32, 0x1A20C4E2, 0x2740ED52, + 0x95603142, 0xA80018F2, 0xEFA06222, 0xD2C04B92, + 0x5090DC43, 0x6DF0F5F3, 0x2A508F23, 0x1730A693, + 0xA5107A83, 0x98705333, 0xDFD029E3, 0xE2B00053, + 0xC1C12F04, 0xFCA106B4, 0xBB017C64, 0x866155D4, + 0x344189C4, 0x0921A074, 0x4E81DAA4, 0x73E1F314, + 0xF1B164C5, 0xCCD14D75, 0x8B7137A5, 0xB6111E15, + 0x0431C205, 0x3951EBB5, 0x7EF19165, 0x4391B8D5, + 0xA121B886, 0x9C419136, 0xDBE1EBE6, 0xE681C256, + 0x54A11E46, 0x69C137F6, 0x2E614D26, 0x13016496, + 0x9151F347, 0xAC31DAF7, 0xEB91A027, 0xD6F18997, + 0x64D15587, 0x59B17C37, 0x1E1106E7, 0x23712F57, + 0x58F35849, 0x659371F9, 0x22330B29, 0x1F532299, + 0xAD73FE89, 0x9013D739, 0xD7B3ADE9, 0xEAD38459, + 0x68831388, 0x55E33A38, 0x124340E8, 0x2F236958, + 0x9D03B548, 0xA0639CF8, 0xE7C3E628, 0xDAA3CF98, + 0x3813CFCB, 0x0573E67B, 0x42D39CAB, 0x7FB3B51B, + 0xCD93690B, 0xF0F340BB, 0xB7533A6B, 0x8A3313DB, + 0x0863840A, 0x3503ADBA, 0x72A3D76A, 0x4FC3FEDA, + 0xFDE322CA, 0xC0830B7A, 0x872371AA, 0xBA43581A, + 0x9932774D, 0xA4525EFD, 0xE3F2242D, 0xDE920D9D, + 0x6CB2D18D, 0x51D2F83D, 0x167282ED, 0x2B12AB5D, + 0xA9423C8C, 0x9422153C, 0xD3826FEC, 0xEEE2465C, + 0x5CC29A4C, 0x61A2B3FC, 0x2602C92C, 0x1B62E09C, + 0xF9D2E0CF, 0xC4B2C97F, 0x8312B3AF, 0xBE729A1F, + 0x0C52460F, 0x31326FBF, 0x7692156F, 0x4BF23CDF, + 0xC9A2AB0E, 0xF4C282BE, 0xB362F86E, 0x8E02D1DE, + 0x3C220DCE, 0x0142247E, 0x46E25EAE, 0x7B82771E, + 0xB1E6B092, 0x8C869922, 0xCB26E3F2, 0xF646CA42, + 0x44661652, 0x79063FE2, 0x3EA64532, 0x03C66C82, + 0x8196FB53, 0xBCF6D2E3, 0xFB56A833, 0xC6368183, + 0x74165D93, 0x49767423, 0x0ED60EF3, 0x33B62743, + 0xD1062710, 0xEC660EA0, 0xABC67470, 0x96A65DC0, + 0x248681D0, 0x19E6A860, 0x5E46D2B0, 0x6326FB00, + 0xE1766CD1, 0xDC164561, 0x9BB63FB1, 0xA6D61601, + 0x14F6CA11, 0x2996E3A1, 0x6E369971, 0x5356B0C1, + 0x70279F96, 0x4D47B626, 0x0AE7CCF6, 0x3787E546, + 0x85A73956, 0xB8C710E6, 0xFF676A36, 0xC2074386, + 0x4057D457, 0x7D37FDE7, 0x3A978737, 0x07F7AE87, + 0xB5D77297, 0x88B75B27, 0xCF1721F7, 0xF2770847, + 0x10C70814, 0x2DA721A4, 0x6A075B74, 0x576772C4, + 0xE547AED4, 0xD8278764, 0x9F87FDB4, 0xA2E7D404, + 0x20B743D5, 0x1DD76A65, 0x5A7710B5, 0x67173905, + 0xD537E515, 0xE857CCA5, 0xAFF7B675, 0x92979FC5, + 0xE915E8DB, 0xD475C16B, 0x93D5BBBB, 0xAEB5920B, + 0x1C954E1B, 0x21F567AB, 0x66551D7B, 0x5B3534CB, + 0xD965A31A, 0xE4058AAA, 0xA3A5F07A, 0x9EC5D9CA, + 0x2CE505DA, 0x11852C6A, 0x562556BA, 0x6B457F0A, + 0x89F57F59, 0xB49556E9, 0xF3352C39, 0xCE550589, + 0x7C75D999, 0x4115F029, 0x06B58AF9, 0x3BD5A349, + 0xB9853498, 0x84E51D28, 0xC34567F8, 0xFE254E48, + 0x4C059258, 0x7165BBE8, 0x36C5C138, 0x0BA5E888, + 0x28D4C7DF, 0x15B4EE6F, 0x521494BF, 0x6F74BD0F, + 0xDD54611F, 0xE03448AF, 0xA794327F, 0x9AF41BCF, + 0x18A48C1E, 0x25C4A5AE, 0x6264DF7E, 0x5F04F6CE, + 0xED242ADE, 0xD044036E, 0x97E479BE, 0xAA84500E, + 0x4834505D, 0x755479ED, 0x32F4033D, 0x0F942A8D, + 0xBDB4F69D, 0x80D4DF2D, 0xC774A5FD, 0xFA148C4D, + 0x78441B9C, 0x4524322C, 0x028448FC, 0x3FE4614C, + 0x8DC4BD5C, 0xB0A494EC, 0xF704EE3C, 0xCA64C78C, + /* T8_5 */ + 0x00000000, 0xCB5CD3A5, 0x4DC8A10B, 0x869472AE, + 0x9B914216, 0x50CD91B3, 0xD659E31D, 0x1D0530B8, + 0xEC53826D, 0x270F51C8, 0xA19B2366, 0x6AC7F0C3, + 0x77C2C07B, 0xBC9E13DE, 0x3A0A6170, 0xF156B2D5, + 0x03D6029B, 0xC88AD13E, 0x4E1EA390, 0x85427035, + 0x9847408D, 0x531B9328, 0xD58FE186, 0x1ED33223, + 0xEF8580F6, 0x24D95353, 0xA24D21FD, 0x6911F258, + 0x7414C2E0, 0xBF481145, 0x39DC63EB, 0xF280B04E, + 0x07AC0536, 0xCCF0D693, 0x4A64A43D, 0x81387798, + 0x9C3D4720, 0x57619485, 0xD1F5E62B, 0x1AA9358E, + 0xEBFF875B, 0x20A354FE, 0xA6372650, 0x6D6BF5F5, + 0x706EC54D, 0xBB3216E8, 0x3DA66446, 0xF6FAB7E3, + 0x047A07AD, 0xCF26D408, 0x49B2A6A6, 0x82EE7503, + 0x9FEB45BB, 0x54B7961E, 0xD223E4B0, 0x197F3715, + 0xE82985C0, 0x23755665, 0xA5E124CB, 0x6EBDF76E, + 0x73B8C7D6, 0xB8E41473, 0x3E7066DD, 0xF52CB578, + 0x0F580A6C, 0xC404D9C9, 0x4290AB67, 0x89CC78C2, + 0x94C9487A, 0x5F959BDF, 0xD901E971, 0x125D3AD4, + 0xE30B8801, 0x28575BA4, 0xAEC3290A, 0x659FFAAF, + 0x789ACA17, 0xB3C619B2, 0x35526B1C, 0xFE0EB8B9, + 0x0C8E08F7, 0xC7D2DB52, 0x4146A9FC, 0x8A1A7A59, + 0x971F4AE1, 0x5C439944, 0xDAD7EBEA, 0x118B384F, + 0xE0DD8A9A, 0x2B81593F, 0xAD152B91, 0x6649F834, + 0x7B4CC88C, 0xB0101B29, 0x36846987, 0xFDD8BA22, + 0x08F40F5A, 0xC3A8DCFF, 0x453CAE51, 0x8E607DF4, + 0x93654D4C, 0x58399EE9, 0xDEADEC47, 0x15F13FE2, + 0xE4A78D37, 0x2FFB5E92, 0xA96F2C3C, 0x6233FF99, + 0x7F36CF21, 0xB46A1C84, 0x32FE6E2A, 0xF9A2BD8F, + 0x0B220DC1, 0xC07EDE64, 0x46EAACCA, 0x8DB67F6F, + 0x90B34FD7, 0x5BEF9C72, 0xDD7BEEDC, 0x16273D79, + 0xE7718FAC, 0x2C2D5C09, 0xAAB92EA7, 0x61E5FD02, + 0x7CE0CDBA, 0xB7BC1E1F, 0x31286CB1, 0xFA74BF14, + 0x1EB014D8, 0xD5ECC77D, 0x5378B5D3, 0x98246676, + 0x852156CE, 0x4E7D856B, 0xC8E9F7C5, 0x03B52460, + 0xF2E396B5, 0x39BF4510, 0xBF2B37BE, 0x7477E41B, + 0x6972D4A3, 0xA22E0706, 0x24BA75A8, 0xEFE6A60D, + 0x1D661643, 0xD63AC5E6, 0x50AEB748, 0x9BF264ED, + 0x86F75455, 0x4DAB87F0, 0xCB3FF55E, 0x006326FB, + 0xF135942E, 0x3A69478B, 0xBCFD3525, 0x77A1E680, + 0x6AA4D638, 0xA1F8059D, 0x276C7733, 0xEC30A496, + 0x191C11EE, 0xD240C24B, 0x54D4B0E5, 0x9F886340, + 0x828D53F8, 0x49D1805D, 0xCF45F2F3, 0x04192156, + 0xF54F9383, 0x3E134026, 0xB8873288, 0x73DBE12D, + 0x6EDED195, 0xA5820230, 0x2316709E, 0xE84AA33B, + 0x1ACA1375, 0xD196C0D0, 0x5702B27E, 0x9C5E61DB, + 0x815B5163, 0x4A0782C6, 0xCC93F068, 0x07CF23CD, + 0xF6999118, 0x3DC542BD, 0xBB513013, 0x700DE3B6, + 0x6D08D30E, 0xA65400AB, 0x20C07205, 0xEB9CA1A0, + 0x11E81EB4, 0xDAB4CD11, 0x5C20BFBF, 0x977C6C1A, + 0x8A795CA2, 0x41258F07, 0xC7B1FDA9, 0x0CED2E0C, + 0xFDBB9CD9, 0x36E74F7C, 0xB0733DD2, 0x7B2FEE77, + 0x662ADECF, 0xAD760D6A, 0x2BE27FC4, 0xE0BEAC61, + 0x123E1C2F, 0xD962CF8A, 0x5FF6BD24, 0x94AA6E81, + 0x89AF5E39, 0x42F38D9C, 0xC467FF32, 0x0F3B2C97, + 0xFE6D9E42, 0x35314DE7, 0xB3A53F49, 0x78F9ECEC, + 0x65FCDC54, 0xAEA00FF1, 0x28347D5F, 0xE368AEFA, + 0x16441B82, 0xDD18C827, 0x5B8CBA89, 0x90D0692C, + 0x8DD55994, 0x46898A31, 0xC01DF89F, 0x0B412B3A, + 0xFA1799EF, 0x314B4A4A, 0xB7DF38E4, 0x7C83EB41, + 0x6186DBF9, 0xAADA085C, 0x2C4E7AF2, 0xE712A957, + 0x15921919, 0xDECECABC, 0x585AB812, 0x93066BB7, + 0x8E035B0F, 0x455F88AA, 0xC3CBFA04, 0x089729A1, + 0xF9C19B74, 0x329D48D1, 0xB4093A7F, 0x7F55E9DA, + 0x6250D962, 0xA90C0AC7, 0x2F987869, 0xE4C4ABCC, + /* T8_6 */ + 0x00000000, 0xA6770BB4, 0x979F1129, 0x31E81A9D, + 0xF44F2413, 0x52382FA7, 0x63D0353A, 0xC5A73E8E, + 0x33EF4E67, 0x959845D3, 0xA4705F4E, 0x020754FA, + 0xC7A06A74, 0x61D761C0, 0x503F7B5D, 0xF64870E9, + 0x67DE9CCE, 0xC1A9977A, 0xF0418DE7, 0x56368653, + 0x9391B8DD, 0x35E6B369, 0x040EA9F4, 0xA279A240, + 0x5431D2A9, 0xF246D91D, 0xC3AEC380, 0x65D9C834, + 0xA07EF6BA, 0x0609FD0E, 0x37E1E793, 0x9196EC27, + 0xCFBD399C, 0x69CA3228, 0x582228B5, 0xFE552301, + 0x3BF21D8F, 0x9D85163B, 0xAC6D0CA6, 0x0A1A0712, + 0xFC5277FB, 0x5A257C4F, 0x6BCD66D2, 0xCDBA6D66, + 0x081D53E8, 0xAE6A585C, 0x9F8242C1, 0x39F54975, + 0xA863A552, 0x0E14AEE6, 0x3FFCB47B, 0x998BBFCF, + 0x5C2C8141, 0xFA5B8AF5, 0xCBB39068, 0x6DC49BDC, + 0x9B8CEB35, 0x3DFBE081, 0x0C13FA1C, 0xAA64F1A8, + 0x6FC3CF26, 0xC9B4C492, 0xF85CDE0F, 0x5E2BD5BB, + 0x440B7579, 0xE27C7ECD, 0xD3946450, 0x75E36FE4, + 0xB044516A, 0x16335ADE, 0x27DB4043, 0x81AC4BF7, + 0x77E43B1E, 0xD19330AA, 0xE07B2A37, 0x460C2183, + 0x83AB1F0D, 0x25DC14B9, 0x14340E24, 0xB2430590, + 0x23D5E9B7, 0x85A2E203, 0xB44AF89E, 0x123DF32A, + 0xD79ACDA4, 0x71EDC610, 0x4005DC8D, 0xE672D739, + 0x103AA7D0, 0xB64DAC64, 0x87A5B6F9, 0x21D2BD4D, + 0xE47583C3, 0x42028877, 0x73EA92EA, 0xD59D995E, + 0x8BB64CE5, 0x2DC14751, 0x1C295DCC, 0xBA5E5678, + 0x7FF968F6, 0xD98E6342, 0xE86679DF, 0x4E11726B, + 0xB8590282, 0x1E2E0936, 0x2FC613AB, 0x89B1181F, + 0x4C162691, 0xEA612D25, 0xDB8937B8, 0x7DFE3C0C, + 0xEC68D02B, 0x4A1FDB9F, 0x7BF7C102, 0xDD80CAB6, + 0x1827F438, 0xBE50FF8C, 0x8FB8E511, 0x29CFEEA5, + 0xDF879E4C, 0x79F095F8, 0x48188F65, 0xEE6F84D1, + 0x2BC8BA5F, 0x8DBFB1EB, 0xBC57AB76, 0x1A20A0C2, + 0x8816EAF2, 0x2E61E146, 0x1F89FBDB, 0xB9FEF06F, + 0x7C59CEE1, 0xDA2EC555, 0xEBC6DFC8, 0x4DB1D47C, + 0xBBF9A495, 0x1D8EAF21, 0x2C66B5BC, 0x8A11BE08, + 0x4FB68086, 0xE9C18B32, 0xD82991AF, 0x7E5E9A1B, + 0xEFC8763C, 0x49BF7D88, 0x78576715, 0xDE206CA1, + 0x1B87522F, 0xBDF0599B, 0x8C184306, 0x2A6F48B2, + 0xDC27385B, 0x7A5033EF, 0x4BB82972, 0xEDCF22C6, + 0x28681C48, 0x8E1F17FC, 0xBFF70D61, 0x198006D5, + 0x47ABD36E, 0xE1DCD8DA, 0xD034C247, 0x7643C9F3, + 0xB3E4F77D, 0x1593FCC9, 0x247BE654, 0x820CEDE0, + 0x74449D09, 0xD23396BD, 0xE3DB8C20, 0x45AC8794, + 0x800BB91A, 0x267CB2AE, 0x1794A833, 0xB1E3A387, + 0x20754FA0, 0x86024414, 0xB7EA5E89, 0x119D553D, + 0xD43A6BB3, 0x724D6007, 0x43A57A9A, 0xE5D2712E, + 0x139A01C7, 0xB5ED0A73, 0x840510EE, 0x22721B5A, + 0xE7D525D4, 0x41A22E60, 0x704A34FD, 0xD63D3F49, + 0xCC1D9F8B, 0x6A6A943F, 0x5B828EA2, 0xFDF58516, + 0x3852BB98, 0x9E25B02C, 0xAFCDAAB1, 0x09BAA105, + 0xFFF2D1EC, 0x5985DA58, 0x686DC0C5, 0xCE1ACB71, + 0x0BBDF5FF, 0xADCAFE4B, 0x9C22E4D6, 0x3A55EF62, + 0xABC30345, 0x0DB408F1, 0x3C5C126C, 0x9A2B19D8, + 0x5F8C2756, 0xF9FB2CE2, 0xC813367F, 0x6E643DCB, + 0x982C4D22, 0x3E5B4696, 0x0FB35C0B, 0xA9C457BF, + 0x6C636931, 0xCA146285, 0xFBFC7818, 0x5D8B73AC, + 0x03A0A617, 0xA5D7ADA3, 0x943FB73E, 0x3248BC8A, + 0xF7EF8204, 0x519889B0, 0x6070932D, 0xC6079899, + 0x304FE870, 0x9638E3C4, 0xA7D0F959, 0x01A7F2ED, + 0xC400CC63, 0x6277C7D7, 0x539FDD4A, 0xF5E8D6FE, + 0x647E3AD9, 0xC209316D, 0xF3E12BF0, 0x55962044, + 0x90311ECA, 0x3646157E, 0x07AE0FE3, 0xA1D90457, + 0x579174BE, 0xF1E67F0A, 0xC00E6597, 0x66796E23, + 0xA3DE50AD, 0x05A95B19, 0x34414184, 0x92364A30, + /* T8_7 */ + 0x00000000, 0xCCAA009E, 0x4225077D, 0x8E8F07E3, + 0x844A0EFA, 0x48E00E64, 0xC66F0987, 0x0AC50919, + 0xD3E51BB5, 0x1F4F1B2B, 0x91C01CC8, 0x5D6A1C56, + 0x57AF154F, 0x9B0515D1, 0x158A1232, 0xD92012AC, + 0x7CBB312B, 0xB01131B5, 0x3E9E3656, 0xF23436C8, + 0xF8F13FD1, 0x345B3F4F, 0xBAD438AC, 0x767E3832, + 0xAF5E2A9E, 0x63F42A00, 0xED7B2DE3, 0x21D12D7D, + 0x2B142464, 0xE7BE24FA, 0x69312319, 0xA59B2387, + 0xF9766256, 0x35DC62C8, 0xBB53652B, 0x77F965B5, + 0x7D3C6CAC, 0xB1966C32, 0x3F196BD1, 0xF3B36B4F, + 0x2A9379E3, 0xE639797D, 0x68B67E9E, 0xA41C7E00, + 0xAED97719, 0x62737787, 0xECFC7064, 0x205670FA, + 0x85CD537D, 0x496753E3, 0xC7E85400, 0x0B42549E, + 0x01875D87, 0xCD2D5D19, 0x43A25AFA, 0x8F085A64, + 0x562848C8, 0x9A824856, 0x140D4FB5, 0xD8A74F2B, + 0xD2624632, 0x1EC846AC, 0x9047414F, 0x5CED41D1, + 0x299DC2ED, 0xE537C273, 0x6BB8C590, 0xA712C50E, + 0xADD7CC17, 0x617DCC89, 0xEFF2CB6A, 0x2358CBF4, + 0xFA78D958, 0x36D2D9C6, 0xB85DDE25, 0x74F7DEBB, + 0x7E32D7A2, 0xB298D73C, 0x3C17D0DF, 0xF0BDD041, + 0x5526F3C6, 0x998CF358, 0x1703F4BB, 0xDBA9F425, + 0xD16CFD3C, 0x1DC6FDA2, 0x9349FA41, 0x5FE3FADF, + 0x86C3E873, 0x4A69E8ED, 0xC4E6EF0E, 0x084CEF90, + 0x0289E689, 0xCE23E617, 0x40ACE1F4, 0x8C06E16A, + 0xD0EBA0BB, 0x1C41A025, 0x92CEA7C6, 0x5E64A758, + 0x54A1AE41, 0x980BAEDF, 0x1684A93C, 0xDA2EA9A2, + 0x030EBB0E, 0xCFA4BB90, 0x412BBC73, 0x8D81BCED, + 0x8744B5F4, 0x4BEEB56A, 0xC561B289, 0x09CBB217, + 0xAC509190, 0x60FA910E, 0xEE7596ED, 0x22DF9673, + 0x281A9F6A, 0xE4B09FF4, 0x6A3F9817, 0xA6959889, + 0x7FB58A25, 0xB31F8ABB, 0x3D908D58, 0xF13A8DC6, + 0xFBFF84DF, 0x37558441, 0xB9DA83A2, 0x7570833C, + 0x533B85DA, 0x9F918544, 0x111E82A7, 0xDDB48239, + 0xD7718B20, 0x1BDB8BBE, 0x95548C5D, 0x59FE8CC3, + 0x80DE9E6F, 0x4C749EF1, 0xC2FB9912, 0x0E51998C, + 0x04949095, 0xC83E900B, 0x46B197E8, 0x8A1B9776, + 0x2F80B4F1, 0xE32AB46F, 0x6DA5B38C, 0xA10FB312, + 0xABCABA0B, 0x6760BA95, 0xE9EFBD76, 0x2545BDE8, + 0xFC65AF44, 0x30CFAFDA, 0xBE40A839, 0x72EAA8A7, + 0x782FA1BE, 0xB485A120, 0x3A0AA6C3, 0xF6A0A65D, + 0xAA4DE78C, 0x66E7E712, 0xE868E0F1, 0x24C2E06F, + 0x2E07E976, 0xE2ADE9E8, 0x6C22EE0B, 0xA088EE95, + 0x79A8FC39, 0xB502FCA7, 0x3B8DFB44, 0xF727FBDA, + 0xFDE2F2C3, 0x3148F25D, 0xBFC7F5BE, 0x736DF520, + 0xD6F6D6A7, 0x1A5CD639, 0x94D3D1DA, 0x5879D144, + 0x52BCD85D, 0x9E16D8C3, 0x1099DF20, 0xDC33DFBE, + 0x0513CD12, 0xC9B9CD8C, 0x4736CA6F, 0x8B9CCAF1, + 0x8159C3E8, 0x4DF3C376, 0xC37CC495, 0x0FD6C40B, + 0x7AA64737, 0xB60C47A9, 0x3883404A, 0xF42940D4, + 0xFEEC49CD, 0x32464953, 0xBCC94EB0, 0x70634E2E, + 0xA9435C82, 0x65E95C1C, 0xEB665BFF, 0x27CC5B61, + 0x2D095278, 0xE1A352E6, 0x6F2C5505, 0xA386559B, + 0x061D761C, 0xCAB77682, 0x44387161, 0x889271FF, + 0x825778E6, 0x4EFD7878, 0xC0727F9B, 0x0CD87F05, + 0xD5F86DA9, 0x19526D37, 0x97DD6AD4, 0x5B776A4A, + 0x51B26353, 0x9D1863CD, 0x1397642E, 0xDF3D64B0, + 0x83D02561, 0x4F7A25FF, 0xC1F5221C, 0x0D5F2282, + 0x079A2B9B, 0xCB302B05, 0x45BF2CE6, 0x89152C78, + 0x50353ED4, 0x9C9F3E4A, 0x121039A9, 0xDEBA3937, + 0xD47F302E, 0x18D530B0, 0x965A3753, 0x5AF037CD, + 0xFF6B144A, 0x33C114D4, 0xBD4E1337, 0x71E413A9, + 0x7B211AB0, 0xB78B1A2E, 0x39041DCD, 0xF5AE1D53, + 0x2C8E0FFF, 0xE0240F61, 0x6EAB0882, 0xA201081C, + 0xA8C40105, 0x646E019B, 0xEAE10678, 0x264B06E6 + }; +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/PureJavaCrc32CByteBuffer.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/PureJavaCrc32CByteBuffer.java new file mode 100644 index 0000000000000..1c443575f8179 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/PureJavaCrc32CByteBuffer.java @@ -0,0 +1,559 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + * Some portions of this file Copyright (c) 2004-2006 Intel Corportation + * and licensed under the BSD license. + */ +package org.apache.hadoop.ozone.common; + +/** + * Similar to {@link org.apache.hadoop.util.PureJavaCrc32C} + * except that this class implement {@link ChecksumByteBuffer}. + */ +final class PureJavaCrc32CByteBuffer extends ChecksumByteBuffer.CrcIntTable { + @Override + int[] getTable() { + return T; + } + + /** + * CRC-32C lookup table generated by the polynomial 0x82F63B78. + * See also org.apache.hadoop.util.TestPureJavaCrc32.Table. + */ + private static final int[] T = { + /* T8_0 */ + 0x00000000, 0xF26B8303, 0xE13B70F7, 0x1350F3F4, + 0xC79A971F, 0x35F1141C, 0x26A1E7E8, 0xD4CA64EB, + 0x8AD958CF, 0x78B2DBCC, 0x6BE22838, 0x9989AB3B, + 0x4D43CFD0, 0xBF284CD3, 0xAC78BF27, 0x5E133C24, + 0x105EC76F, 0xE235446C, 0xF165B798, 0x030E349B, + 0xD7C45070, 0x25AFD373, 0x36FF2087, 0xC494A384, + 0x9A879FA0, 0x68EC1CA3, 0x7BBCEF57, 0x89D76C54, + 0x5D1D08BF, 0xAF768BBC, 0xBC267848, 0x4E4DFB4B, + 0x20BD8EDE, 0xD2D60DDD, 0xC186FE29, 0x33ED7D2A, + 0xE72719C1, 0x154C9AC2, 0x061C6936, 0xF477EA35, + 0xAA64D611, 0x580F5512, 0x4B5FA6E6, 0xB93425E5, + 0x6DFE410E, 0x9F95C20D, 0x8CC531F9, 0x7EAEB2FA, + 0x30E349B1, 0xC288CAB2, 0xD1D83946, 0x23B3BA45, + 0xF779DEAE, 0x05125DAD, 0x1642AE59, 0xE4292D5A, + 0xBA3A117E, 0x4851927D, 0x5B016189, 0xA96AE28A, + 0x7DA08661, 0x8FCB0562, 0x9C9BF696, 0x6EF07595, + 0x417B1DBC, 0xB3109EBF, 0xA0406D4B, 0x522BEE48, + 0x86E18AA3, 0x748A09A0, 0x67DAFA54, 0x95B17957, + 0xCBA24573, 0x39C9C670, 0x2A993584, 0xD8F2B687, + 0x0C38D26C, 0xFE53516F, 0xED03A29B, 0x1F682198, + 0x5125DAD3, 0xA34E59D0, 0xB01EAA24, 0x42752927, + 0x96BF4DCC, 0x64D4CECF, 0x77843D3B, 0x85EFBE38, + 0xDBFC821C, 0x2997011F, 0x3AC7F2EB, 0xC8AC71E8, + 0x1C661503, 0xEE0D9600, 0xFD5D65F4, 0x0F36E6F7, + 0x61C69362, 0x93AD1061, 0x80FDE395, 0x72966096, + 0xA65C047D, 0x5437877E, 0x4767748A, 0xB50CF789, + 0xEB1FCBAD, 0x197448AE, 0x0A24BB5A, 0xF84F3859, + 0x2C855CB2, 0xDEEEDFB1, 0xCDBE2C45, 0x3FD5AF46, + 0x7198540D, 0x83F3D70E, 0x90A324FA, 0x62C8A7F9, + 0xB602C312, 0x44694011, 0x5739B3E5, 0xA55230E6, + 0xFB410CC2, 0x092A8FC1, 0x1A7A7C35, 0xE811FF36, + 0x3CDB9BDD, 0xCEB018DE, 0xDDE0EB2A, 0x2F8B6829, + 0x82F63B78, 0x709DB87B, 0x63CD4B8F, 0x91A6C88C, + 0x456CAC67, 0xB7072F64, 0xA457DC90, 0x563C5F93, + 0x082F63B7, 0xFA44E0B4, 0xE9141340, 0x1B7F9043, + 0xCFB5F4A8, 0x3DDE77AB, 0x2E8E845F, 0xDCE5075C, + 0x92A8FC17, 0x60C37F14, 0x73938CE0, 0x81F80FE3, + 0x55326B08, 0xA759E80B, 0xB4091BFF, 0x466298FC, + 0x1871A4D8, 0xEA1A27DB, 0xF94AD42F, 0x0B21572C, + 0xDFEB33C7, 0x2D80B0C4, 0x3ED04330, 0xCCBBC033, + 0xA24BB5A6, 0x502036A5, 0x4370C551, 0xB11B4652, + 0x65D122B9, 0x97BAA1BA, 0x84EA524E, 0x7681D14D, + 0x2892ED69, 0xDAF96E6A, 0xC9A99D9E, 0x3BC21E9D, + 0xEF087A76, 0x1D63F975, 0x0E330A81, 0xFC588982, + 0xB21572C9, 0x407EF1CA, 0x532E023E, 0xA145813D, + 0x758FE5D6, 0x87E466D5, 0x94B49521, 0x66DF1622, + 0x38CC2A06, 0xCAA7A905, 0xD9F75AF1, 0x2B9CD9F2, + 0xFF56BD19, 0x0D3D3E1A, 0x1E6DCDEE, 0xEC064EED, + 0xC38D26C4, 0x31E6A5C7, 0x22B65633, 0xD0DDD530, + 0x0417B1DB, 0xF67C32D8, 0xE52CC12C, 0x1747422F, + 0x49547E0B, 0xBB3FFD08, 0xA86F0EFC, 0x5A048DFF, + 0x8ECEE914, 0x7CA56A17, 0x6FF599E3, 0x9D9E1AE0, + 0xD3D3E1AB, 0x21B862A8, 0x32E8915C, 0xC083125F, + 0x144976B4, 0xE622F5B7, 0xF5720643, 0x07198540, + 0x590AB964, 0xAB613A67, 0xB831C993, 0x4A5A4A90, + 0x9E902E7B, 0x6CFBAD78, 0x7FAB5E8C, 0x8DC0DD8F, + 0xE330A81A, 0x115B2B19, 0x020BD8ED, 0xF0605BEE, + 0x24AA3F05, 0xD6C1BC06, 0xC5914FF2, 0x37FACCF1, + 0x69E9F0D5, 0x9B8273D6, 0x88D28022, 0x7AB90321, + 0xAE7367CA, 0x5C18E4C9, 0x4F48173D, 0xBD23943E, + 0xF36E6F75, 0x0105EC76, 0x12551F82, 0xE03E9C81, + 0x34F4F86A, 0xC69F7B69, 0xD5CF889D, 0x27A40B9E, + 0x79B737BA, 0x8BDCB4B9, 0x988C474D, 0x6AE7C44E, + 0xBE2DA0A5, 0x4C4623A6, 0x5F16D052, 0xAD7D5351, + /* T8_1 */ + 0x00000000, 0x13A29877, 0x274530EE, 0x34E7A899, + 0x4E8A61DC, 0x5D28F9AB, 0x69CF5132, 0x7A6DC945, + 0x9D14C3B8, 0x8EB65BCF, 0xBA51F356, 0xA9F36B21, + 0xD39EA264, 0xC03C3A13, 0xF4DB928A, 0xE7790AFD, + 0x3FC5F181, 0x2C6769F6, 0x1880C16F, 0x0B225918, + 0x714F905D, 0x62ED082A, 0x560AA0B3, 0x45A838C4, + 0xA2D13239, 0xB173AA4E, 0x859402D7, 0x96369AA0, + 0xEC5B53E5, 0xFFF9CB92, 0xCB1E630B, 0xD8BCFB7C, + 0x7F8BE302, 0x6C297B75, 0x58CED3EC, 0x4B6C4B9B, + 0x310182DE, 0x22A31AA9, 0x1644B230, 0x05E62A47, + 0xE29F20BA, 0xF13DB8CD, 0xC5DA1054, 0xD6788823, + 0xAC154166, 0xBFB7D911, 0x8B507188, 0x98F2E9FF, + 0x404E1283, 0x53EC8AF4, 0x670B226D, 0x74A9BA1A, + 0x0EC4735F, 0x1D66EB28, 0x298143B1, 0x3A23DBC6, + 0xDD5AD13B, 0xCEF8494C, 0xFA1FE1D5, 0xE9BD79A2, + 0x93D0B0E7, 0x80722890, 0xB4958009, 0xA737187E, + 0xFF17C604, 0xECB55E73, 0xD852F6EA, 0xCBF06E9D, + 0xB19DA7D8, 0xA23F3FAF, 0x96D89736, 0x857A0F41, + 0x620305BC, 0x71A19DCB, 0x45463552, 0x56E4AD25, + 0x2C896460, 0x3F2BFC17, 0x0BCC548E, 0x186ECCF9, + 0xC0D23785, 0xD370AFF2, 0xE797076B, 0xF4359F1C, + 0x8E585659, 0x9DFACE2E, 0xA91D66B7, 0xBABFFEC0, + 0x5DC6F43D, 0x4E646C4A, 0x7A83C4D3, 0x69215CA4, + 0x134C95E1, 0x00EE0D96, 0x3409A50F, 0x27AB3D78, + 0x809C2506, 0x933EBD71, 0xA7D915E8, 0xB47B8D9F, + 0xCE1644DA, 0xDDB4DCAD, 0xE9537434, 0xFAF1EC43, + 0x1D88E6BE, 0x0E2A7EC9, 0x3ACDD650, 0x296F4E27, + 0x53028762, 0x40A01F15, 0x7447B78C, 0x67E52FFB, + 0xBF59D487, 0xACFB4CF0, 0x981CE469, 0x8BBE7C1E, + 0xF1D3B55B, 0xE2712D2C, 0xD69685B5, 0xC5341DC2, + 0x224D173F, 0x31EF8F48, 0x050827D1, 0x16AABFA6, + 0x6CC776E3, 0x7F65EE94, 0x4B82460D, 0x5820DE7A, + 0xFBC3FAF9, 0xE861628E, 0xDC86CA17, 0xCF245260, + 0xB5499B25, 0xA6EB0352, 0x920CABCB, 0x81AE33BC, + 0x66D73941, 0x7575A136, 0x419209AF, 0x523091D8, + 0x285D589D, 0x3BFFC0EA, 0x0F186873, 0x1CBAF004, + 0xC4060B78, 0xD7A4930F, 0xE3433B96, 0xF0E1A3E1, + 0x8A8C6AA4, 0x992EF2D3, 0xADC95A4A, 0xBE6BC23D, + 0x5912C8C0, 0x4AB050B7, 0x7E57F82E, 0x6DF56059, + 0x1798A91C, 0x043A316B, 0x30DD99F2, 0x237F0185, + 0x844819FB, 0x97EA818C, 0xA30D2915, 0xB0AFB162, + 0xCAC27827, 0xD960E050, 0xED8748C9, 0xFE25D0BE, + 0x195CDA43, 0x0AFE4234, 0x3E19EAAD, 0x2DBB72DA, + 0x57D6BB9F, 0x447423E8, 0x70938B71, 0x63311306, + 0xBB8DE87A, 0xA82F700D, 0x9CC8D894, 0x8F6A40E3, + 0xF50789A6, 0xE6A511D1, 0xD242B948, 0xC1E0213F, + 0x26992BC2, 0x353BB3B5, 0x01DC1B2C, 0x127E835B, + 0x68134A1E, 0x7BB1D269, 0x4F567AF0, 0x5CF4E287, + 0x04D43CFD, 0x1776A48A, 0x23910C13, 0x30339464, + 0x4A5E5D21, 0x59FCC556, 0x6D1B6DCF, 0x7EB9F5B8, + 0x99C0FF45, 0x8A626732, 0xBE85CFAB, 0xAD2757DC, + 0xD74A9E99, 0xC4E806EE, 0xF00FAE77, 0xE3AD3600, + 0x3B11CD7C, 0x28B3550B, 0x1C54FD92, 0x0FF665E5, + 0x759BACA0, 0x663934D7, 0x52DE9C4E, 0x417C0439, + 0xA6050EC4, 0xB5A796B3, 0x81403E2A, 0x92E2A65D, + 0xE88F6F18, 0xFB2DF76F, 0xCFCA5FF6, 0xDC68C781, + 0x7B5FDFFF, 0x68FD4788, 0x5C1AEF11, 0x4FB87766, + 0x35D5BE23, 0x26772654, 0x12908ECD, 0x013216BA, + 0xE64B1C47, 0xF5E98430, 0xC10E2CA9, 0xD2ACB4DE, + 0xA8C17D9B, 0xBB63E5EC, 0x8F844D75, 0x9C26D502, + 0x449A2E7E, 0x5738B609, 0x63DF1E90, 0x707D86E7, + 0x0A104FA2, 0x19B2D7D5, 0x2D557F4C, 0x3EF7E73B, + 0xD98EEDC6, 0xCA2C75B1, 0xFECBDD28, 0xED69455F, + 0x97048C1A, 0x84A6146D, 0xB041BCF4, 0xA3E32483, + /* T8_2 */ + 0x00000000, 0xA541927E, 0x4F6F520D, 0xEA2EC073, + 0x9EDEA41A, 0x3B9F3664, 0xD1B1F617, 0x74F06469, + 0x38513EC5, 0x9D10ACBB, 0x773E6CC8, 0xD27FFEB6, + 0xA68F9ADF, 0x03CE08A1, 0xE9E0C8D2, 0x4CA15AAC, + 0x70A27D8A, 0xD5E3EFF4, 0x3FCD2F87, 0x9A8CBDF9, + 0xEE7CD990, 0x4B3D4BEE, 0xA1138B9D, 0x045219E3, + 0x48F3434F, 0xEDB2D131, 0x079C1142, 0xA2DD833C, + 0xD62DE755, 0x736C752B, 0x9942B558, 0x3C032726, + 0xE144FB14, 0x4405696A, 0xAE2BA919, 0x0B6A3B67, + 0x7F9A5F0E, 0xDADBCD70, 0x30F50D03, 0x95B49F7D, + 0xD915C5D1, 0x7C5457AF, 0x967A97DC, 0x333B05A2, + 0x47CB61CB, 0xE28AF3B5, 0x08A433C6, 0xADE5A1B8, + 0x91E6869E, 0x34A714E0, 0xDE89D493, 0x7BC846ED, + 0x0F382284, 0xAA79B0FA, 0x40577089, 0xE516E2F7, + 0xA9B7B85B, 0x0CF62A25, 0xE6D8EA56, 0x43997828, + 0x37691C41, 0x92288E3F, 0x78064E4C, 0xDD47DC32, + 0xC76580D9, 0x622412A7, 0x880AD2D4, 0x2D4B40AA, + 0x59BB24C3, 0xFCFAB6BD, 0x16D476CE, 0xB395E4B0, + 0xFF34BE1C, 0x5A752C62, 0xB05BEC11, 0x151A7E6F, + 0x61EA1A06, 0xC4AB8878, 0x2E85480B, 0x8BC4DA75, + 0xB7C7FD53, 0x12866F2D, 0xF8A8AF5E, 0x5DE93D20, + 0x29195949, 0x8C58CB37, 0x66760B44, 0xC337993A, + 0x8F96C396, 0x2AD751E8, 0xC0F9919B, 0x65B803E5, + 0x1148678C, 0xB409F5F2, 0x5E273581, 0xFB66A7FF, + 0x26217BCD, 0x8360E9B3, 0x694E29C0, 0xCC0FBBBE, + 0xB8FFDFD7, 0x1DBE4DA9, 0xF7908DDA, 0x52D11FA4, + 0x1E704508, 0xBB31D776, 0x511F1705, 0xF45E857B, + 0x80AEE112, 0x25EF736C, 0xCFC1B31F, 0x6A802161, + 0x56830647, 0xF3C29439, 0x19EC544A, 0xBCADC634, + 0xC85DA25D, 0x6D1C3023, 0x8732F050, 0x2273622E, + 0x6ED23882, 0xCB93AAFC, 0x21BD6A8F, 0x84FCF8F1, + 0xF00C9C98, 0x554D0EE6, 0xBF63CE95, 0x1A225CEB, + 0x8B277743, 0x2E66E53D, 0xC448254E, 0x6109B730, + 0x15F9D359, 0xB0B84127, 0x5A968154, 0xFFD7132A, + 0xB3764986, 0x1637DBF8, 0xFC191B8B, 0x595889F5, + 0x2DA8ED9C, 0x88E97FE2, 0x62C7BF91, 0xC7862DEF, + 0xFB850AC9, 0x5EC498B7, 0xB4EA58C4, 0x11ABCABA, + 0x655BAED3, 0xC01A3CAD, 0x2A34FCDE, 0x8F756EA0, + 0xC3D4340C, 0x6695A672, 0x8CBB6601, 0x29FAF47F, + 0x5D0A9016, 0xF84B0268, 0x1265C21B, 0xB7245065, + 0x6A638C57, 0xCF221E29, 0x250CDE5A, 0x804D4C24, + 0xF4BD284D, 0x51FCBA33, 0xBBD27A40, 0x1E93E83E, + 0x5232B292, 0xF77320EC, 0x1D5DE09F, 0xB81C72E1, + 0xCCEC1688, 0x69AD84F6, 0x83834485, 0x26C2D6FB, + 0x1AC1F1DD, 0xBF8063A3, 0x55AEA3D0, 0xF0EF31AE, + 0x841F55C7, 0x215EC7B9, 0xCB7007CA, 0x6E3195B4, + 0x2290CF18, 0x87D15D66, 0x6DFF9D15, 0xC8BE0F6B, + 0xBC4E6B02, 0x190FF97C, 0xF321390F, 0x5660AB71, + 0x4C42F79A, 0xE90365E4, 0x032DA597, 0xA66C37E9, + 0xD29C5380, 0x77DDC1FE, 0x9DF3018D, 0x38B293F3, + 0x7413C95F, 0xD1525B21, 0x3B7C9B52, 0x9E3D092C, + 0xEACD6D45, 0x4F8CFF3B, 0xA5A23F48, 0x00E3AD36, + 0x3CE08A10, 0x99A1186E, 0x738FD81D, 0xD6CE4A63, + 0xA23E2E0A, 0x077FBC74, 0xED517C07, 0x4810EE79, + 0x04B1B4D5, 0xA1F026AB, 0x4BDEE6D8, 0xEE9F74A6, + 0x9A6F10CF, 0x3F2E82B1, 0xD50042C2, 0x7041D0BC, + 0xAD060C8E, 0x08479EF0, 0xE2695E83, 0x4728CCFD, + 0x33D8A894, 0x96993AEA, 0x7CB7FA99, 0xD9F668E7, + 0x9557324B, 0x3016A035, 0xDA386046, 0x7F79F238, + 0x0B899651, 0xAEC8042F, 0x44E6C45C, 0xE1A75622, + 0xDDA47104, 0x78E5E37A, 0x92CB2309, 0x378AB177, + 0x437AD51E, 0xE63B4760, 0x0C158713, 0xA954156D, + 0xE5F54FC1, 0x40B4DDBF, 0xAA9A1DCC, 0x0FDB8FB2, + 0x7B2BEBDB, 0xDE6A79A5, 0x3444B9D6, 0x91052BA8, + /* T8_3 */ + 0x00000000, 0xDD45AAB8, 0xBF672381, 0x62228939, + 0x7B2231F3, 0xA6679B4B, 0xC4451272, 0x1900B8CA, + 0xF64463E6, 0x2B01C95E, 0x49234067, 0x9466EADF, + 0x8D665215, 0x5023F8AD, 0x32017194, 0xEF44DB2C, + 0xE964B13D, 0x34211B85, 0x560392BC, 0x8B463804, + 0x924680CE, 0x4F032A76, 0x2D21A34F, 0xF06409F7, + 0x1F20D2DB, 0xC2657863, 0xA047F15A, 0x7D025BE2, + 0x6402E328, 0xB9474990, 0xDB65C0A9, 0x06206A11, + 0xD725148B, 0x0A60BE33, 0x6842370A, 0xB5079DB2, + 0xAC072578, 0x71428FC0, 0x136006F9, 0xCE25AC41, + 0x2161776D, 0xFC24DDD5, 0x9E0654EC, 0x4343FE54, + 0x5A43469E, 0x8706EC26, 0xE524651F, 0x3861CFA7, + 0x3E41A5B6, 0xE3040F0E, 0x81268637, 0x5C632C8F, + 0x45639445, 0x98263EFD, 0xFA04B7C4, 0x27411D7C, + 0xC805C650, 0x15406CE8, 0x7762E5D1, 0xAA274F69, + 0xB327F7A3, 0x6E625D1B, 0x0C40D422, 0xD1057E9A, + 0xABA65FE7, 0x76E3F55F, 0x14C17C66, 0xC984D6DE, + 0xD0846E14, 0x0DC1C4AC, 0x6FE34D95, 0xB2A6E72D, + 0x5DE23C01, 0x80A796B9, 0xE2851F80, 0x3FC0B538, + 0x26C00DF2, 0xFB85A74A, 0x99A72E73, 0x44E284CB, + 0x42C2EEDA, 0x9F874462, 0xFDA5CD5B, 0x20E067E3, + 0x39E0DF29, 0xE4A57591, 0x8687FCA8, 0x5BC25610, + 0xB4868D3C, 0x69C32784, 0x0BE1AEBD, 0xD6A40405, + 0xCFA4BCCF, 0x12E11677, 0x70C39F4E, 0xAD8635F6, + 0x7C834B6C, 0xA1C6E1D4, 0xC3E468ED, 0x1EA1C255, + 0x07A17A9F, 0xDAE4D027, 0xB8C6591E, 0x6583F3A6, + 0x8AC7288A, 0x57828232, 0x35A00B0B, 0xE8E5A1B3, + 0xF1E51979, 0x2CA0B3C1, 0x4E823AF8, 0x93C79040, + 0x95E7FA51, 0x48A250E9, 0x2A80D9D0, 0xF7C57368, + 0xEEC5CBA2, 0x3380611A, 0x51A2E823, 0x8CE7429B, + 0x63A399B7, 0xBEE6330F, 0xDCC4BA36, 0x0181108E, + 0x1881A844, 0xC5C402FC, 0xA7E68BC5, 0x7AA3217D, + 0x52A0C93F, 0x8FE56387, 0xEDC7EABE, 0x30824006, + 0x2982F8CC, 0xF4C75274, 0x96E5DB4D, 0x4BA071F5, + 0xA4E4AAD9, 0x79A10061, 0x1B838958, 0xC6C623E0, + 0xDFC69B2A, 0x02833192, 0x60A1B8AB, 0xBDE41213, + 0xBBC47802, 0x6681D2BA, 0x04A35B83, 0xD9E6F13B, + 0xC0E649F1, 0x1DA3E349, 0x7F816A70, 0xA2C4C0C8, + 0x4D801BE4, 0x90C5B15C, 0xF2E73865, 0x2FA292DD, + 0x36A22A17, 0xEBE780AF, 0x89C50996, 0x5480A32E, + 0x8585DDB4, 0x58C0770C, 0x3AE2FE35, 0xE7A7548D, + 0xFEA7EC47, 0x23E246FF, 0x41C0CFC6, 0x9C85657E, + 0x73C1BE52, 0xAE8414EA, 0xCCA69DD3, 0x11E3376B, + 0x08E38FA1, 0xD5A62519, 0xB784AC20, 0x6AC10698, + 0x6CE16C89, 0xB1A4C631, 0xD3864F08, 0x0EC3E5B0, + 0x17C35D7A, 0xCA86F7C2, 0xA8A47EFB, 0x75E1D443, + 0x9AA50F6F, 0x47E0A5D7, 0x25C22CEE, 0xF8878656, + 0xE1873E9C, 0x3CC29424, 0x5EE01D1D, 0x83A5B7A5, + 0xF90696D8, 0x24433C60, 0x4661B559, 0x9B241FE1, + 0x8224A72B, 0x5F610D93, 0x3D4384AA, 0xE0062E12, + 0x0F42F53E, 0xD2075F86, 0xB025D6BF, 0x6D607C07, + 0x7460C4CD, 0xA9256E75, 0xCB07E74C, 0x16424DF4, + 0x106227E5, 0xCD278D5D, 0xAF050464, 0x7240AEDC, + 0x6B401616, 0xB605BCAE, 0xD4273597, 0x09629F2F, + 0xE6264403, 0x3B63EEBB, 0x59416782, 0x8404CD3A, + 0x9D0475F0, 0x4041DF48, 0x22635671, 0xFF26FCC9, + 0x2E238253, 0xF36628EB, 0x9144A1D2, 0x4C010B6A, + 0x5501B3A0, 0x88441918, 0xEA669021, 0x37233A99, + 0xD867E1B5, 0x05224B0D, 0x6700C234, 0xBA45688C, + 0xA345D046, 0x7E007AFE, 0x1C22F3C7, 0xC167597F, + 0xC747336E, 0x1A0299D6, 0x782010EF, 0xA565BA57, + 0xBC65029D, 0x6120A825, 0x0302211C, 0xDE478BA4, + 0x31035088, 0xEC46FA30, 0x8E647309, 0x5321D9B1, + 0x4A21617B, 0x9764CBC3, 0xF54642FA, 0x2803E842, + /* T8_4 */ + 0x00000000, 0x38116FAC, 0x7022DF58, 0x4833B0F4, + 0xE045BEB0, 0xD854D11C, 0x906761E8, 0xA8760E44, + 0xC5670B91, 0xFD76643D, 0xB545D4C9, 0x8D54BB65, + 0x2522B521, 0x1D33DA8D, 0x55006A79, 0x6D1105D5, + 0x8F2261D3, 0xB7330E7F, 0xFF00BE8B, 0xC711D127, + 0x6F67DF63, 0x5776B0CF, 0x1F45003B, 0x27546F97, + 0x4A456A42, 0x725405EE, 0x3A67B51A, 0x0276DAB6, + 0xAA00D4F2, 0x9211BB5E, 0xDA220BAA, 0xE2336406, + 0x1BA8B557, 0x23B9DAFB, 0x6B8A6A0F, 0x539B05A3, + 0xFBED0BE7, 0xC3FC644B, 0x8BCFD4BF, 0xB3DEBB13, + 0xDECFBEC6, 0xE6DED16A, 0xAEED619E, 0x96FC0E32, + 0x3E8A0076, 0x069B6FDA, 0x4EA8DF2E, 0x76B9B082, + 0x948AD484, 0xAC9BBB28, 0xE4A80BDC, 0xDCB96470, + 0x74CF6A34, 0x4CDE0598, 0x04EDB56C, 0x3CFCDAC0, + 0x51EDDF15, 0x69FCB0B9, 0x21CF004D, 0x19DE6FE1, + 0xB1A861A5, 0x89B90E09, 0xC18ABEFD, 0xF99BD151, + 0x37516AAE, 0x0F400502, 0x4773B5F6, 0x7F62DA5A, + 0xD714D41E, 0xEF05BBB2, 0xA7360B46, 0x9F2764EA, + 0xF236613F, 0xCA270E93, 0x8214BE67, 0xBA05D1CB, + 0x1273DF8F, 0x2A62B023, 0x625100D7, 0x5A406F7B, + 0xB8730B7D, 0x806264D1, 0xC851D425, 0xF040BB89, + 0x5836B5CD, 0x6027DA61, 0x28146A95, 0x10050539, + 0x7D1400EC, 0x45056F40, 0x0D36DFB4, 0x3527B018, + 0x9D51BE5C, 0xA540D1F0, 0xED736104, 0xD5620EA8, + 0x2CF9DFF9, 0x14E8B055, 0x5CDB00A1, 0x64CA6F0D, + 0xCCBC6149, 0xF4AD0EE5, 0xBC9EBE11, 0x848FD1BD, + 0xE99ED468, 0xD18FBBC4, 0x99BC0B30, 0xA1AD649C, + 0x09DB6AD8, 0x31CA0574, 0x79F9B580, 0x41E8DA2C, + 0xA3DBBE2A, 0x9BCAD186, 0xD3F96172, 0xEBE80EDE, + 0x439E009A, 0x7B8F6F36, 0x33BCDFC2, 0x0BADB06E, + 0x66BCB5BB, 0x5EADDA17, 0x169E6AE3, 0x2E8F054F, + 0x86F90B0B, 0xBEE864A7, 0xF6DBD453, 0xCECABBFF, + 0x6EA2D55C, 0x56B3BAF0, 0x1E800A04, 0x269165A8, + 0x8EE76BEC, 0xB6F60440, 0xFEC5B4B4, 0xC6D4DB18, + 0xABC5DECD, 0x93D4B161, 0xDBE70195, 0xE3F66E39, + 0x4B80607D, 0x73910FD1, 0x3BA2BF25, 0x03B3D089, + 0xE180B48F, 0xD991DB23, 0x91A26BD7, 0xA9B3047B, + 0x01C50A3F, 0x39D46593, 0x71E7D567, 0x49F6BACB, + 0x24E7BF1E, 0x1CF6D0B2, 0x54C56046, 0x6CD40FEA, + 0xC4A201AE, 0xFCB36E02, 0xB480DEF6, 0x8C91B15A, + 0x750A600B, 0x4D1B0FA7, 0x0528BF53, 0x3D39D0FF, + 0x954FDEBB, 0xAD5EB117, 0xE56D01E3, 0xDD7C6E4F, + 0xB06D6B9A, 0x887C0436, 0xC04FB4C2, 0xF85EDB6E, + 0x5028D52A, 0x6839BA86, 0x200A0A72, 0x181B65DE, + 0xFA2801D8, 0xC2396E74, 0x8A0ADE80, 0xB21BB12C, + 0x1A6DBF68, 0x227CD0C4, 0x6A4F6030, 0x525E0F9C, + 0x3F4F0A49, 0x075E65E5, 0x4F6DD511, 0x777CBABD, + 0xDF0AB4F9, 0xE71BDB55, 0xAF286BA1, 0x9739040D, + 0x59F3BFF2, 0x61E2D05E, 0x29D160AA, 0x11C00F06, + 0xB9B60142, 0x81A76EEE, 0xC994DE1A, 0xF185B1B6, + 0x9C94B463, 0xA485DBCF, 0xECB66B3B, 0xD4A70497, + 0x7CD10AD3, 0x44C0657F, 0x0CF3D58B, 0x34E2BA27, + 0xD6D1DE21, 0xEEC0B18D, 0xA6F30179, 0x9EE26ED5, + 0x36946091, 0x0E850F3D, 0x46B6BFC9, 0x7EA7D065, + 0x13B6D5B0, 0x2BA7BA1C, 0x63940AE8, 0x5B856544, + 0xF3F36B00, 0xCBE204AC, 0x83D1B458, 0xBBC0DBF4, + 0x425B0AA5, 0x7A4A6509, 0x3279D5FD, 0x0A68BA51, + 0xA21EB415, 0x9A0FDBB9, 0xD23C6B4D, 0xEA2D04E1, + 0x873C0134, 0xBF2D6E98, 0xF71EDE6C, 0xCF0FB1C0, + 0x6779BF84, 0x5F68D028, 0x175B60DC, 0x2F4A0F70, + 0xCD796B76, 0xF56804DA, 0xBD5BB42E, 0x854ADB82, + 0x2D3CD5C6, 0x152DBA6A, 0x5D1E0A9E, 0x650F6532, + 0x081E60E7, 0x300F0F4B, 0x783CBFBF, 0x402DD013, + 0xE85BDE57, 0xD04AB1FB, 0x9879010F, 0xA0686EA3, + /* T8_5 */ + 0x00000000, 0xEF306B19, 0xDB8CA0C3, 0x34BCCBDA, + 0xB2F53777, 0x5DC55C6E, 0x697997B4, 0x8649FCAD, + 0x6006181F, 0x8F367306, 0xBB8AB8DC, 0x54BAD3C5, + 0xD2F32F68, 0x3DC34471, 0x097F8FAB, 0xE64FE4B2, + 0xC00C303E, 0x2F3C5B27, 0x1B8090FD, 0xF4B0FBE4, + 0x72F90749, 0x9DC96C50, 0xA975A78A, 0x4645CC93, + 0xA00A2821, 0x4F3A4338, 0x7B8688E2, 0x94B6E3FB, + 0x12FF1F56, 0xFDCF744F, 0xC973BF95, 0x2643D48C, + 0x85F4168D, 0x6AC47D94, 0x5E78B64E, 0xB148DD57, + 0x370121FA, 0xD8314AE3, 0xEC8D8139, 0x03BDEA20, + 0xE5F20E92, 0x0AC2658B, 0x3E7EAE51, 0xD14EC548, + 0x570739E5, 0xB83752FC, 0x8C8B9926, 0x63BBF23F, + 0x45F826B3, 0xAAC84DAA, 0x9E748670, 0x7144ED69, + 0xF70D11C4, 0x183D7ADD, 0x2C81B107, 0xC3B1DA1E, + 0x25FE3EAC, 0xCACE55B5, 0xFE729E6F, 0x1142F576, + 0x970B09DB, 0x783B62C2, 0x4C87A918, 0xA3B7C201, + 0x0E045BEB, 0xE13430F2, 0xD588FB28, 0x3AB89031, + 0xBCF16C9C, 0x53C10785, 0x677DCC5F, 0x884DA746, + 0x6E0243F4, 0x813228ED, 0xB58EE337, 0x5ABE882E, + 0xDCF77483, 0x33C71F9A, 0x077BD440, 0xE84BBF59, + 0xCE086BD5, 0x213800CC, 0x1584CB16, 0xFAB4A00F, + 0x7CFD5CA2, 0x93CD37BB, 0xA771FC61, 0x48419778, + 0xAE0E73CA, 0x413E18D3, 0x7582D309, 0x9AB2B810, + 0x1CFB44BD, 0xF3CB2FA4, 0xC777E47E, 0x28478F67, + 0x8BF04D66, 0x64C0267F, 0x507CEDA5, 0xBF4C86BC, + 0x39057A11, 0xD6351108, 0xE289DAD2, 0x0DB9B1CB, + 0xEBF65579, 0x04C63E60, 0x307AF5BA, 0xDF4A9EA3, + 0x5903620E, 0xB6330917, 0x828FC2CD, 0x6DBFA9D4, + 0x4BFC7D58, 0xA4CC1641, 0x9070DD9B, 0x7F40B682, + 0xF9094A2F, 0x16392136, 0x2285EAEC, 0xCDB581F5, + 0x2BFA6547, 0xC4CA0E5E, 0xF076C584, 0x1F46AE9D, + 0x990F5230, 0x763F3929, 0x4283F2F3, 0xADB399EA, + 0x1C08B7D6, 0xF338DCCF, 0xC7841715, 0x28B47C0C, + 0xAEFD80A1, 0x41CDEBB8, 0x75712062, 0x9A414B7B, + 0x7C0EAFC9, 0x933EC4D0, 0xA7820F0A, 0x48B26413, + 0xCEFB98BE, 0x21CBF3A7, 0x1577387D, 0xFA475364, + 0xDC0487E8, 0x3334ECF1, 0x0788272B, 0xE8B84C32, + 0x6EF1B09F, 0x81C1DB86, 0xB57D105C, 0x5A4D7B45, + 0xBC029FF7, 0x5332F4EE, 0x678E3F34, 0x88BE542D, + 0x0EF7A880, 0xE1C7C399, 0xD57B0843, 0x3A4B635A, + 0x99FCA15B, 0x76CCCA42, 0x42700198, 0xAD406A81, + 0x2B09962C, 0xC439FD35, 0xF08536EF, 0x1FB55DF6, + 0xF9FAB944, 0x16CAD25D, 0x22761987, 0xCD46729E, + 0x4B0F8E33, 0xA43FE52A, 0x90832EF0, 0x7FB345E9, + 0x59F09165, 0xB6C0FA7C, 0x827C31A6, 0x6D4C5ABF, + 0xEB05A612, 0x0435CD0B, 0x308906D1, 0xDFB96DC8, + 0x39F6897A, 0xD6C6E263, 0xE27A29B9, 0x0D4A42A0, + 0x8B03BE0D, 0x6433D514, 0x508F1ECE, 0xBFBF75D7, + 0x120CEC3D, 0xFD3C8724, 0xC9804CFE, 0x26B027E7, + 0xA0F9DB4A, 0x4FC9B053, 0x7B757B89, 0x94451090, + 0x720AF422, 0x9D3A9F3B, 0xA98654E1, 0x46B63FF8, + 0xC0FFC355, 0x2FCFA84C, 0x1B736396, 0xF443088F, + 0xD200DC03, 0x3D30B71A, 0x098C7CC0, 0xE6BC17D9, + 0x60F5EB74, 0x8FC5806D, 0xBB794BB7, 0x544920AE, + 0xB206C41C, 0x5D36AF05, 0x698A64DF, 0x86BA0FC6, + 0x00F3F36B, 0xEFC39872, 0xDB7F53A8, 0x344F38B1, + 0x97F8FAB0, 0x78C891A9, 0x4C745A73, 0xA344316A, + 0x250DCDC7, 0xCA3DA6DE, 0xFE816D04, 0x11B1061D, + 0xF7FEE2AF, 0x18CE89B6, 0x2C72426C, 0xC3422975, + 0x450BD5D8, 0xAA3BBEC1, 0x9E87751B, 0x71B71E02, + 0x57F4CA8E, 0xB8C4A197, 0x8C786A4D, 0x63480154, + 0xE501FDF9, 0x0A3196E0, 0x3E8D5D3A, 0xD1BD3623, + 0x37F2D291, 0xD8C2B988, 0xEC7E7252, 0x034E194B, + 0x8507E5E6, 0x6A378EFF, 0x5E8B4525, 0xB1BB2E3C, + /* T8_6 */ + 0x00000000, 0x68032CC8, 0xD0065990, 0xB8057558, + 0xA5E0C5D1, 0xCDE3E919, 0x75E69C41, 0x1DE5B089, + 0x4E2DFD53, 0x262ED19B, 0x9E2BA4C3, 0xF628880B, + 0xEBCD3882, 0x83CE144A, 0x3BCB6112, 0x53C84DDA, + 0x9C5BFAA6, 0xF458D66E, 0x4C5DA336, 0x245E8FFE, + 0x39BB3F77, 0x51B813BF, 0xE9BD66E7, 0x81BE4A2F, + 0xD27607F5, 0xBA752B3D, 0x02705E65, 0x6A7372AD, + 0x7796C224, 0x1F95EEEC, 0xA7909BB4, 0xCF93B77C, + 0x3D5B83BD, 0x5558AF75, 0xED5DDA2D, 0x855EF6E5, + 0x98BB466C, 0xF0B86AA4, 0x48BD1FFC, 0x20BE3334, + 0x73767EEE, 0x1B755226, 0xA370277E, 0xCB730BB6, + 0xD696BB3F, 0xBE9597F7, 0x0690E2AF, 0x6E93CE67, + 0xA100791B, 0xC90355D3, 0x7106208B, 0x19050C43, + 0x04E0BCCA, 0x6CE39002, 0xD4E6E55A, 0xBCE5C992, + 0xEF2D8448, 0x872EA880, 0x3F2BDDD8, 0x5728F110, + 0x4ACD4199, 0x22CE6D51, 0x9ACB1809, 0xF2C834C1, + 0x7AB7077A, 0x12B42BB2, 0xAAB15EEA, 0xC2B27222, + 0xDF57C2AB, 0xB754EE63, 0x0F519B3B, 0x6752B7F3, + 0x349AFA29, 0x5C99D6E1, 0xE49CA3B9, 0x8C9F8F71, + 0x917A3FF8, 0xF9791330, 0x417C6668, 0x297F4AA0, + 0xE6ECFDDC, 0x8EEFD114, 0x36EAA44C, 0x5EE98884, + 0x430C380D, 0x2B0F14C5, 0x930A619D, 0xFB094D55, + 0xA8C1008F, 0xC0C22C47, 0x78C7591F, 0x10C475D7, + 0x0D21C55E, 0x6522E996, 0xDD279CCE, 0xB524B006, + 0x47EC84C7, 0x2FEFA80F, 0x97EADD57, 0xFFE9F19F, + 0xE20C4116, 0x8A0F6DDE, 0x320A1886, 0x5A09344E, + 0x09C17994, 0x61C2555C, 0xD9C72004, 0xB1C40CCC, + 0xAC21BC45, 0xC422908D, 0x7C27E5D5, 0x1424C91D, + 0xDBB77E61, 0xB3B452A9, 0x0BB127F1, 0x63B20B39, + 0x7E57BBB0, 0x16549778, 0xAE51E220, 0xC652CEE8, + 0x959A8332, 0xFD99AFFA, 0x459CDAA2, 0x2D9FF66A, + 0x307A46E3, 0x58796A2B, 0xE07C1F73, 0x887F33BB, + 0xF56E0EF4, 0x9D6D223C, 0x25685764, 0x4D6B7BAC, + 0x508ECB25, 0x388DE7ED, 0x808892B5, 0xE88BBE7D, + 0xBB43F3A7, 0xD340DF6F, 0x6B45AA37, 0x034686FF, + 0x1EA33676, 0x76A01ABE, 0xCEA56FE6, 0xA6A6432E, + 0x6935F452, 0x0136D89A, 0xB933ADC2, 0xD130810A, + 0xCCD53183, 0xA4D61D4B, 0x1CD36813, 0x74D044DB, + 0x27180901, 0x4F1B25C9, 0xF71E5091, 0x9F1D7C59, + 0x82F8CCD0, 0xEAFBE018, 0x52FE9540, 0x3AFDB988, + 0xC8358D49, 0xA036A181, 0x1833D4D9, 0x7030F811, + 0x6DD54898, 0x05D66450, 0xBDD31108, 0xD5D03DC0, + 0x8618701A, 0xEE1B5CD2, 0x561E298A, 0x3E1D0542, + 0x23F8B5CB, 0x4BFB9903, 0xF3FEEC5B, 0x9BFDC093, + 0x546E77EF, 0x3C6D5B27, 0x84682E7F, 0xEC6B02B7, + 0xF18EB23E, 0x998D9EF6, 0x2188EBAE, 0x498BC766, + 0x1A438ABC, 0x7240A674, 0xCA45D32C, 0xA246FFE4, + 0xBFA34F6D, 0xD7A063A5, 0x6FA516FD, 0x07A63A35, + 0x8FD9098E, 0xE7DA2546, 0x5FDF501E, 0x37DC7CD6, + 0x2A39CC5F, 0x423AE097, 0xFA3F95CF, 0x923CB907, + 0xC1F4F4DD, 0xA9F7D815, 0x11F2AD4D, 0x79F18185, + 0x6414310C, 0x0C171DC4, 0xB412689C, 0xDC114454, + 0x1382F328, 0x7B81DFE0, 0xC384AAB8, 0xAB878670, + 0xB66236F9, 0xDE611A31, 0x66646F69, 0x0E6743A1, + 0x5DAF0E7B, 0x35AC22B3, 0x8DA957EB, 0xE5AA7B23, + 0xF84FCBAA, 0x904CE762, 0x2849923A, 0x404ABEF2, + 0xB2828A33, 0xDA81A6FB, 0x6284D3A3, 0x0A87FF6B, + 0x17624FE2, 0x7F61632A, 0xC7641672, 0xAF673ABA, + 0xFCAF7760, 0x94AC5BA8, 0x2CA92EF0, 0x44AA0238, + 0x594FB2B1, 0x314C9E79, 0x8949EB21, 0xE14AC7E9, + 0x2ED97095, 0x46DA5C5D, 0xFEDF2905, 0x96DC05CD, + 0x8B39B544, 0xE33A998C, 0x5B3FECD4, 0x333CC01C, + 0x60F48DC6, 0x08F7A10E, 0xB0F2D456, 0xD8F1F89E, + 0xC5144817, 0xAD1764DF, 0x15121187, 0x7D113D4F, + /* T8_7 */ + 0x00000000, 0x493C7D27, 0x9278FA4E, 0xDB448769, + 0x211D826D, 0x6821FF4A, 0xB3657823, 0xFA590504, + 0x423B04DA, 0x0B0779FD, 0xD043FE94, 0x997F83B3, + 0x632686B7, 0x2A1AFB90, 0xF15E7CF9, 0xB86201DE, + 0x847609B4, 0xCD4A7493, 0x160EF3FA, 0x5F328EDD, + 0xA56B8BD9, 0xEC57F6FE, 0x37137197, 0x7E2F0CB0, + 0xC64D0D6E, 0x8F717049, 0x5435F720, 0x1D098A07, + 0xE7508F03, 0xAE6CF224, 0x7528754D, 0x3C14086A, + 0x0D006599, 0x443C18BE, 0x9F789FD7, 0xD644E2F0, + 0x2C1DE7F4, 0x65219AD3, 0xBE651DBA, 0xF759609D, + 0x4F3B6143, 0x06071C64, 0xDD439B0D, 0x947FE62A, + 0x6E26E32E, 0x271A9E09, 0xFC5E1960, 0xB5626447, + 0x89766C2D, 0xC04A110A, 0x1B0E9663, 0x5232EB44, + 0xA86BEE40, 0xE1579367, 0x3A13140E, 0x732F6929, + 0xCB4D68F7, 0x827115D0, 0x593592B9, 0x1009EF9E, + 0xEA50EA9A, 0xA36C97BD, 0x782810D4, 0x31146DF3, + 0x1A00CB32, 0x533CB615, 0x8878317C, 0xC1444C5B, + 0x3B1D495F, 0x72213478, 0xA965B311, 0xE059CE36, + 0x583BCFE8, 0x1107B2CF, 0xCA4335A6, 0x837F4881, + 0x79264D85, 0x301A30A2, 0xEB5EB7CB, 0xA262CAEC, + 0x9E76C286, 0xD74ABFA1, 0x0C0E38C8, 0x453245EF, + 0xBF6B40EB, 0xF6573DCC, 0x2D13BAA5, 0x642FC782, + 0xDC4DC65C, 0x9571BB7B, 0x4E353C12, 0x07094135, + 0xFD504431, 0xB46C3916, 0x6F28BE7F, 0x2614C358, + 0x1700AEAB, 0x5E3CD38C, 0x857854E5, 0xCC4429C2, + 0x361D2CC6, 0x7F2151E1, 0xA465D688, 0xED59ABAF, + 0x553BAA71, 0x1C07D756, 0xC743503F, 0x8E7F2D18, + 0x7426281C, 0x3D1A553B, 0xE65ED252, 0xAF62AF75, + 0x9376A71F, 0xDA4ADA38, 0x010E5D51, 0x48322076, + 0xB26B2572, 0xFB575855, 0x2013DF3C, 0x692FA21B, + 0xD14DA3C5, 0x9871DEE2, 0x4335598B, 0x0A0924AC, + 0xF05021A8, 0xB96C5C8F, 0x6228DBE6, 0x2B14A6C1, + 0x34019664, 0x7D3DEB43, 0xA6796C2A, 0xEF45110D, + 0x151C1409, 0x5C20692E, 0x8764EE47, 0xCE589360, + 0x763A92BE, 0x3F06EF99, 0xE44268F0, 0xAD7E15D7, + 0x572710D3, 0x1E1B6DF4, 0xC55FEA9D, 0x8C6397BA, + 0xB0779FD0, 0xF94BE2F7, 0x220F659E, 0x6B3318B9, + 0x916A1DBD, 0xD856609A, 0x0312E7F3, 0x4A2E9AD4, + 0xF24C9B0A, 0xBB70E62D, 0x60346144, 0x29081C63, + 0xD3511967, 0x9A6D6440, 0x4129E329, 0x08159E0E, + 0x3901F3FD, 0x703D8EDA, 0xAB7909B3, 0xE2457494, + 0x181C7190, 0x51200CB7, 0x8A648BDE, 0xC358F6F9, + 0x7B3AF727, 0x32068A00, 0xE9420D69, 0xA07E704E, + 0x5A27754A, 0x131B086D, 0xC85F8F04, 0x8163F223, + 0xBD77FA49, 0xF44B876E, 0x2F0F0007, 0x66337D20, + 0x9C6A7824, 0xD5560503, 0x0E12826A, 0x472EFF4D, + 0xFF4CFE93, 0xB67083B4, 0x6D3404DD, 0x240879FA, + 0xDE517CFE, 0x976D01D9, 0x4C2986B0, 0x0515FB97, + 0x2E015D56, 0x673D2071, 0xBC79A718, 0xF545DA3F, + 0x0F1CDF3B, 0x4620A21C, 0x9D642575, 0xD4585852, + 0x6C3A598C, 0x250624AB, 0xFE42A3C2, 0xB77EDEE5, + 0x4D27DBE1, 0x041BA6C6, 0xDF5F21AF, 0x96635C88, + 0xAA7754E2, 0xE34B29C5, 0x380FAEAC, 0x7133D38B, + 0x8B6AD68F, 0xC256ABA8, 0x19122CC1, 0x502E51E6, + 0xE84C5038, 0xA1702D1F, 0x7A34AA76, 0x3308D751, + 0xC951D255, 0x806DAF72, 0x5B29281B, 0x1215553C, + 0x230138CF, 0x6A3D45E8, 0xB179C281, 0xF845BFA6, + 0x021CBAA2, 0x4B20C785, 0x906440EC, 0xD9583DCB, + 0x613A3C15, 0x28064132, 0xF342C65B, 0xBA7EBB7C, + 0x4027BE78, 0x091BC35F, 0xD25F4436, 0x9B633911, + 0xA777317B, 0xEE4B4C5C, 0x350FCB35, 0x7C33B612, + 0x866AB316, 0xCF56CE31, 0x14124958, 0x5D2E347F, + 0xE54C35A1, 0xAC704886, 0x7734CFEF, 0x3E08B2C8, + 0xC451B7CC, 0x8D6DCAEB, 0x56294D82, 0x1F1530A5 + }; +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Storage.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Storage.java index f393ed9c3a4f4..7992dad78db5a 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Storage.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Storage.java @@ -81,7 +81,7 @@ public Storage(NodeType type, File root, String sdName) /** * Gets the path of the Storage dir. - * @return Stoarge dir path + * @return Storage dir path */ public String getStorageDir() { return storageDir.getAbsoluteFile().toString(); @@ -117,7 +117,7 @@ public void setClusterId(String clusterId) throws IOException { } /** - * Retreives the storageInfo instance to read/write the common + * Retrieves the storageInfo instance to read/write the common * version file properties. * @return the instance of the storageInfo class */ @@ -128,7 +128,7 @@ protected StorageInfo getStorageInfo() { abstract protected Properties getNodeProperties(); /** - * Sets the Node properties spaecific to OM/SCM. + * Sets the Node properties specific to OM/SCM. */ private void setNodeProperties() { Properties nodeProperties = getNodeProperties(); diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerCommandRequestPBHelper.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerCommandRequestPBHelper.java index 0b7ae2d7b31ed..11d9028f1900c 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerCommandRequestPBHelper.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerCommandRequestPBHelper.java @@ -79,9 +79,11 @@ public static Map getAuditParams( auditParams.put("blockData", BlockData.getFromProtoBuf(msg.getPutBlock().getBlockData()) .toString()); - }catch (IOException ex){ - LOG.trace("Encountered error parsing BlockData from protobuf:" - + ex.getMessage()); + } catch (IOException ex){ + if (LOG.isTraceEnabled()) { + LOG.trace("Encountered error parsing BlockData from protobuf: " + + ex.getMessage()); + } return null; } return auditParams; @@ -134,9 +136,11 @@ public static Map getAuditParams( auditParams.put("blockData", BlockData.getFromProtoBuf(msg.getPutSmallFile() .getBlock().getBlockData()).toString()); - }catch (IOException ex){ - LOG.trace("Encountered error parsing BlockData from protobuf:" - + ex.getMessage()); + } catch (IOException ex){ + if (LOG.isTraceEnabled()) { + LOG.trace("Encountered error parsing BlockData from protobuf: " + + ex.getMessage()); + } } return auditParams; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lease/LeaseCallbackExecutor.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lease/LeaseCallbackExecutor.java index 1b7391bf5d878..e2ca455ef0c57 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lease/LeaseCallbackExecutor.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lease/LeaseCallbackExecutor.java @@ -49,7 +49,7 @@ public LeaseCallbackExecutor(T resource, List> callbacks) { @Override public void run() { - if(LOG.isDebugEnabled()) { + if (LOG.isDebugEnabled()) { LOG.debug("Executing callbacks for lease on {}", resource); } for(Callable callback : callbacks) { diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lease/LeaseManager.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lease/LeaseManager.java index 756a41af08998..02befaef9804f 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lease/LeaseManager.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lease/LeaseManager.java @@ -110,7 +110,7 @@ public synchronized Lease acquire(T resource) public synchronized Lease acquire(T resource, long timeout) throws LeaseAlreadyExistException { checkStatus(); - if(LOG.isDebugEnabled()) { + if (LOG.isDebugEnabled()) { LOG.debug("Acquiring lease on {} for {} milliseconds", resource, timeout); } if(activeLeases.containsKey(resource)) { @@ -150,7 +150,7 @@ public Lease get(T resource) throws LeaseNotFoundException { public synchronized void release(T resource) throws LeaseNotFoundException { checkStatus(); - if(LOG.isDebugEnabled()) { + if (LOG.isDebugEnabled()) { LOG.debug("Releasing lease on {}", resource); } Lease lease = activeLeases.remove(resource); @@ -206,7 +206,7 @@ private LeaseMonitor() { @Override public void run() { - while(monitor) { + while (monitor) { LOG.debug("{}-LeaseMonitor: checking for lease expiry", name); long sleepTime = Long.MAX_VALUE; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/ActiveLock.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/ActiveLock.java index c3020844927c8..95dfd6c393cac 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/ActiveLock.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/ActiveLock.java @@ -18,22 +18,25 @@ package org.apache.hadoop.ozone.lock; import java.util.concurrent.atomic.AtomicInteger; -import java.util.concurrent.locks.Lock; -import java.util.concurrent.locks.ReentrantLock; +import java.util.concurrent.locks.ReadWriteLock; +import java.util.concurrent.locks.ReentrantReadWriteLock; /** * Lock implementation which also maintains counter. */ public final class ActiveLock { - private Lock lock; + private ReadWriteLock lock; private AtomicInteger count; /** * Use ActiveLock#newInstance to create instance. + * + * @param fairness - if true the lock uses a fair ordering policy, else + * non-fair ordering. */ - private ActiveLock() { - this.lock = new ReentrantLock(); + private ActiveLock(boolean fairness) { + this.lock = new ReentrantReadWriteLock(fairness); this.count = new AtomicInteger(0); } @@ -42,26 +45,63 @@ private ActiveLock() { * * @return new ActiveLock */ - public static ActiveLock newInstance() { - return new ActiveLock(); + public static ActiveLock newInstance(boolean fairness) { + return new ActiveLock(fairness); + } + + /** + * Acquires read lock. + * + *

    Acquires the read lock if the write lock is not held by + * another thread and returns immediately. + * + *

    If the write lock is held by another thread then + * the current thread becomes disabled for thread scheduling + * purposes and lies dormant until the read lock has been acquired. + */ + void readLock() { + lock.readLock().lock(); } /** - * Acquires the lock. + * Attempts to release the read lock. * - *

    If the lock is not available then the current thread becomes - * disabled for thread scheduling purposes and lies dormant until the - * lock has been acquired. + *

    If the number of readers is now zero then the lock + * is made available for write lock attempts. */ - public void lock() { - lock.lock(); + void readUnlock() { + lock.readLock().unlock(); } /** - * Releases the lock. + * Acquires write lock. + * + *

    Acquires the write lock if neither the read nor write lock + * are held by another thread + * and returns immediately, setting the write lock hold count to + * one. + * + *

    If the current thread already holds the write lock then the + * hold count is incremented by one and the method returns + * immediately. + * + *

    If the lock is held by another thread then the current + * thread becomes disabled for thread scheduling purposes and + * lies dormant until the write lock has been acquired. + */ + void writeLock() { + lock.writeLock().lock(); + } + + /** + * Attempts to release the write lock. + * + *

    If the current thread is the holder of this lock then + * the hold count is decremented. If the hold count is now + * zero then the lock is released. */ - public void unlock() { - lock.unlock(); + void writeUnlock() { + lock.writeLock().unlock(); } /** diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/LockManager.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/LockManager.java index 49cf544626b7e..3c2b5d4a394c2 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/LockManager.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/LockManager.java @@ -25,42 +25,169 @@ import java.util.Map; import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Consumer; /** * Manages the locks on a given resource. A new lock is created for each * and every unique resource. Uniqueness of resource depends on the * {@code equals} implementation of it. */ -public class LockManager { +public class LockManager { private static final Logger LOG = LoggerFactory.getLogger(LockManager.class); - private final Map activeLocks = new ConcurrentHashMap<>(); - private final GenericObjectPool lockPool = - new GenericObjectPool<>(new PooledLockFactory()); + private final Map activeLocks = new ConcurrentHashMap<>(); + private final GenericObjectPool lockPool; /** - * Creates new LockManager instance. + * Creates new LockManager instance with the given Configuration.and uses + * non-fair mode for locks. * * @param conf Configuration object */ - public LockManager(Configuration conf) { - int maxPoolSize = conf.getInt(HddsConfigKeys.HDDS_LOCK_MAX_CONCURRENCY, + public LockManager(final Configuration conf) { + this(conf, false); + } + + + /** + * Creates new LockManager instance with the given Configuration. + * + * @param conf Configuration object + * @param fair - true to use fair lock ordering, else non-fair lock ordering. + */ + public LockManager(final Configuration conf, boolean fair) { + final int maxPoolSize = conf.getInt( + HddsConfigKeys.HDDS_LOCK_MAX_CONCURRENCY, HddsConfigKeys.HDDS_LOCK_MAX_CONCURRENCY_DEFAULT); + lockPool = + new GenericObjectPool<>(new PooledLockFactory(fair)); lockPool.setMaxTotal(maxPoolSize); } - /** * Acquires the lock on given resource. * *

    If the lock is not available then the current thread becomes * disabled for thread scheduling purposes and lies dormant until the * lock has been acquired. + * + * @param resource on which the lock has to be acquired + * @deprecated Use {@link LockManager#writeLock} instead + */ + public void lock(final R resource) { + writeLock(resource); + } + + /** + * Releases the lock on given resource. + * + * @param resource for which the lock has to be released + * @deprecated Use {@link LockManager#writeUnlock} instead + */ + public void unlock(final R resource) { + writeUnlock(resource); + } + + /** + * Acquires the read lock on given resource. + * + *

    Acquires the read lock on resource if the write lock is not held by + * another thread and returns immediately. + * + *

    If the write lock on resource is held by another thread then + * the current thread becomes disabled for thread scheduling + * purposes and lies dormant until the read lock has been acquired. + * + * @param resource on which the read lock has to be acquired + */ + public void readLock(final R resource) { + acquire(resource, ActiveLock::readLock); + } + + /** + * Releases the read lock on given resource. + * + * @param resource for which the read lock has to be released + * @throws IllegalMonitorStateException if the current thread does not + * hold this lock */ - public void lock(T resource) { - activeLocks.compute(resource, (k, v) -> { - ActiveLock lock; + public void readUnlock(final R resource) throws IllegalMonitorStateException { + release(resource, ActiveLock::readUnlock); + } + + /** + * Acquires the write lock on given resource. + * + *

    Acquires the write lock on resource if neither the read nor write lock + * are held by another thread and returns immediately. + * + *

    If the current thread already holds the write lock then the + * hold count is incremented by one and the method returns + * immediately. + * + *

    If the lock is held by another thread then the current + * thread becomes disabled for thread scheduling purposes and + * lies dormant until the write lock has been acquired. + * + * @param resource on which the lock has to be acquired + */ + public void writeLock(final R resource) { + acquire(resource, ActiveLock::writeLock); + } + + /** + * Releases the write lock on given resource. + * + * @param resource for which the lock has to be released + * @throws IllegalMonitorStateException if the current thread does not + * hold this lock + */ + public void writeUnlock(final R resource) + throws IllegalMonitorStateException { + release(resource, ActiveLock::writeUnlock); + } + + /** + * Acquires the lock on given resource using the provided lock function. + * + * @param resource on which the lock has to be acquired + * @param lockFn function to acquire the lock + */ + private void acquire(final R resource, final Consumer lockFn) { + lockFn.accept(getLockForLocking(resource)); + } + + /** + * Releases the lock on given resource using the provided release function. + * + * @param resource for which the lock has to be released + * @param releaseFn function to release the lock + */ + private void release(final R resource, final Consumer releaseFn) { + final ActiveLock lock = getLockForReleasing(resource); + releaseFn.accept(lock); + decrementActiveLockCount(resource); + } + + /** + * Returns {@link ActiveLock} instance for the given resource, + * on which the lock can be acquired. + * + * @param resource on which the lock has to be acquired + * @return {@link ActiveLock} instance + */ + private ActiveLock getLockForLocking(final R resource) { + /* + * While getting a lock object for locking we should + * atomically increment the active count of the lock. + * + * This is to avoid cases where the selected lock could + * be removed from the activeLocks map and returned to + * the object pool. + */ + return activeLocks.compute(resource, (k, v) -> { + final ActiveLock lock; try { if (v == null) { lock = lockPool.borrowObject(); @@ -73,21 +200,34 @@ public void lock(T resource) { throw new RuntimeException(ex); } return lock; - }).lock(); + }); } /** - * Releases the lock on given resource. + * Returns {@link ActiveLock} instance for the given resource, + * for which the lock has to be released. + * + * @param resource for which the lock has to be released + * @return {@link ActiveLock} instance */ - public void unlock(T resource) { - ActiveLock lock = activeLocks.get(resource); - if (lock == null) { - // Someone is releasing a lock which was never acquired. Log and return. - LOG.warn("Trying to release the lock on {}, which was never acquired.", - resource); - return; + private ActiveLock getLockForReleasing(final R resource) { + if (activeLocks.containsKey(resource)) { + return activeLocks.get(resource); } - lock.unlock(); + // Someone is releasing a lock which was never acquired. + LOG.error("Trying to release the lock on {}, which was never acquired.", + resource); + throw new IllegalMonitorStateException("Releasing lock on resource " + + resource + " without acquiring lock"); + } + + /** + * Decrements the active lock count and returns the {@link ActiveLock} + * object to pool if the active count is 0. + * + * @param resource resource to which the ActiveLock is associated + */ + private void decrementActiveLockCount(final R resource) { activeLocks.computeIfPresent(resource, (k, v) -> { v.decrementActiveCount(); if (v.getActiveLockCount() != 0) { diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/PooledLockFactory.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/PooledLockFactory.java index 4c24ef74b2831..1e3ba05a3a2b2 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/PooledLockFactory.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lock/PooledLockFactory.java @@ -26,9 +26,14 @@ */ public class PooledLockFactory extends BasePooledObjectFactory { + private boolean fairness; + + PooledLockFactory(boolean fair) { + this.fairness = fair; + } @Override public ActiveLock create() throws Exception { - return ActiveLock.newInstance(); + return ActiveLock.newInstance(fairness); } @Override diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/ProtocolMessageMetrics.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/ProtocolMessageMetrics.java new file mode 100644 index 0000000000000..96725f269a124 --- /dev/null +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/ProtocolMessageMetrics.java @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ozone.protocolPB; + +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicLong; + +import org.apache.hadoop.metrics2.MetricsCollector; +import org.apache.hadoop.metrics2.MetricsInfo; +import org.apache.hadoop.metrics2.MetricsRecordBuilder; +import org.apache.hadoop.metrics2.MetricsSource; +import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; + +import com.google.protobuf.ProtocolMessageEnum; + +/** + * Metrics to count all the subtypes of a specific message. + */ +public class ProtocolMessageMetrics implements MetricsSource { + + private String name; + + private String description; + + private Map counters = + new ConcurrentHashMap<>(); + + public static ProtocolMessageMetrics create(String name, + String description, ProtocolMessageEnum[] types) { + ProtocolMessageMetrics protocolMessageMetrics = + new ProtocolMessageMetrics(name, description, + types); + return protocolMessageMetrics; + } + + public ProtocolMessageMetrics(String name, String description, + ProtocolMessageEnum[] values) { + this.name = name; + this.description = description; + for (ProtocolMessageEnum value : values) { + counters.put(value, new AtomicLong(0)); + } + } + + public void increment(ProtocolMessageEnum key) { + counters.get(key).incrementAndGet(); + } + + public void register() { + DefaultMetricsSystem.instance() + .register(name, description, this); + } + + public void unregister() { + DefaultMetricsSystem.instance().unregisterSource(name); + } + + @Override + public void getMetrics(MetricsCollector collector, boolean all) { + MetricsRecordBuilder builder = collector.addRecord(name); + counters.forEach((key, value) -> { + builder.addCounter(new MetricName(key.toString(), ""), value.longValue()); + }); + builder.endRecord(); + } + + /** + * Simple metrics info implementation. + */ + public static class MetricName implements MetricsInfo { + private String name; + private String description; + + public MetricName(String name, String description) { + this.name = name; + this.description = description; + } + + @Override + public String name() { + return name; + } + + @Override + public String description() { + return description; + } + } +} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/ScmBlockLocationProtocolServerSideTranslatorPB.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/ScmBlockLocationProtocolServerSideTranslatorPB.java deleted file mode 100644 index 65f0a973ce412..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/ScmBlockLocationProtocolServerSideTranslatorPB.java +++ /dev/null @@ -1,150 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.ozone.protocolPB; - -import com.google.protobuf.RpcController; -import com.google.protobuf.ServiceException; -import io.opentracing.Scope; - -import org.apache.hadoop.classification.InterfaceAudience; -import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos - .AllocateBlockResponse; -import org.apache.hadoop.hdds.scm.ScmInfo; -import org.apache.hadoop.hdds.scm.container.common.helpers.AllocatedBlock; -import org.apache.hadoop.hdds.scm.container.common.helpers.ExcludeList; -import org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol; -import org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol; -import org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolPB; -import org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolPB; -import org.apache.hadoop.hdds.protocol.proto.HddsProtos; -import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos - .AllocateScmBlockRequestProto; -import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos - .AllocateScmBlockResponseProto; -import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos - .DeleteKeyBlocksResultProto; -import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos - .DeleteScmKeyBlocksRequestProto; -import org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos - .DeleteScmKeyBlocksResponseProto; -import org.apache.hadoop.hdds.tracing.TracingUtil; -import org.apache.hadoop.ozone.common.BlockGroup; -import org.apache.hadoop.ozone.common.DeleteBlockGroupResult; - -import java.io.IOException; -import java.util.List; -import java.util.stream.Collectors; - -/** - * This class is the server-side translator that forwards requests received on - * {@link StorageContainerLocationProtocolPB} to the - * {@link StorageContainerLocationProtocol} server implementation. - */ -@InterfaceAudience.Private -public final class ScmBlockLocationProtocolServerSideTranslatorPB - implements ScmBlockLocationProtocolPB { - - private final ScmBlockLocationProtocol impl; - - /** - * Creates a new ScmBlockLocationProtocolServerSideTranslatorPB. - * - * @param impl {@link ScmBlockLocationProtocol} server implementation - */ - public ScmBlockLocationProtocolServerSideTranslatorPB( - ScmBlockLocationProtocol impl) throws IOException { - this.impl = impl; - } - - @Override - public AllocateScmBlockResponseProto allocateScmBlock( - RpcController controller, AllocateScmBlockRequestProto request) - throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("ScmBlockLocationProtocol.allocateBlock", - request.getTraceID())) { - List allocatedBlocks = - impl.allocateBlock(request.getSize(), - request.getNumBlocks(), request.getType(), - request.getFactor(), request.getOwner(), - ExcludeList.getFromProtoBuf(request.getExcludeList())); - - AllocateScmBlockResponseProto.Builder builder = - AllocateScmBlockResponseProto.newBuilder(); - - if (allocatedBlocks.size() < request.getNumBlocks()) { - return builder - .setErrorCode(AllocateScmBlockResponseProto.Error.unknownFailure) - .build(); - } - - for (AllocatedBlock block : allocatedBlocks) { - builder.addBlocks(AllocateBlockResponse.newBuilder() - .setContainerBlockID(block.getBlockID().getProtobuf()) - .setPipeline(block.getPipeline().getProtobufMessage())); - } - - return builder - .setErrorCode(AllocateScmBlockResponseProto.Error.success) - .build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } - - @Override - public DeleteScmKeyBlocksResponseProto deleteScmKeyBlocks( - RpcController controller, DeleteScmKeyBlocksRequestProto req) - throws ServiceException { - DeleteScmKeyBlocksResponseProto.Builder resp = - DeleteScmKeyBlocksResponseProto.newBuilder(); - try { - List infoList = req.getKeyBlocksList().stream() - .map(BlockGroup::getFromProto).collect(Collectors.toList()); - final List results = - impl.deleteKeyBlocks(infoList); - for (DeleteBlockGroupResult result: results) { - DeleteKeyBlocksResultProto.Builder deleteResult = - DeleteKeyBlocksResultProto - .newBuilder() - .setObjectKey(result.getObjectKey()) - .addAllBlockResults(result.getBlockResultProtoList()); - resp.addResults(deleteResult.build()); - } - } catch (IOException ex) { - throw new ServiceException(ex); - } - return resp.build(); - } - - @Override - public HddsProtos.GetScmInfoResponseProto getScmInfo( - RpcController controller, HddsProtos.GetScmInfoRequestProto req) - throws ServiceException { - ScmInfo scmInfo; - try { - scmInfo = impl.getScmInfo(); - } catch (IOException ex) { - throw new ServiceException(ex); - } - return HddsProtos.GetScmInfoResponseProto.newBuilder() - .setClusterId(scmInfo.getClusterId()) - .setScmId(scmInfo.getScmId()) - .build(); - } -} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerLocationProtocolServerSideTranslatorPB.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerLocationProtocolServerSideTranslatorPB.java deleted file mode 100644 index ea96cfa0aaea1..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerLocationProtocolServerSideTranslatorPB.java +++ /dev/null @@ -1,309 +0,0 @@ - -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.ozone.protocolPB; - -import com.google.protobuf.RpcController; -import com.google.protobuf.ServiceException; -import io.opentracing.Scope; - -import org.apache.hadoop.classification.InterfaceAudience; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.InSafeModeRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.InSafeModeResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ForceExitSafeModeRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ForceExitSafeModeResponseProto; -import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.GetContainerWithPipelineRequestProto; -import org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos.GetContainerWithPipelineResponseProto; -import org.apache.hadoop.hdds.scm.ScmInfo; -import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerWithPipeline; -import org.apache.hadoop.hdds.scm.container.ContainerInfo; -import org.apache.hadoop.hdds.scm.pipeline.Pipeline; -import org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol; -import org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolPB; -import org.apache.hadoop.hdds.protocol.proto.HddsProtos; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ContainerRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ContainerResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ClosePipelineRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ClosePipelineResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ListPipelineRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ListPipelineResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.GetContainerRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.GetContainerResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ObjectStageChangeRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.ObjectStageChangeResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.PipelineRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.PipelineResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.SCMDeleteContainerRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.SCMDeleteContainerResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.SCMListContainerRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerLocationProtocolProtos.SCMListContainerResponseProto; -import org.apache.hadoop.hdds.tracing.TracingUtil; - -import java.io.IOException; -import java.util.List; - -/** - * This class is the server-side translator that forwards requests received on - * {@link StorageContainerLocationProtocolPB} to the - * {@link StorageContainerLocationProtocol} server implementation. - */ -@InterfaceAudience.Private -public final class StorageContainerLocationProtocolServerSideTranslatorPB - implements StorageContainerLocationProtocolPB { - - private final StorageContainerLocationProtocol impl; - - /** - * Creates a new StorageContainerLocationProtocolServerSideTranslatorPB. - * - * @param impl {@link StorageContainerLocationProtocol} server implementation - */ - public StorageContainerLocationProtocolServerSideTranslatorPB( - StorageContainerLocationProtocol impl) throws IOException { - this.impl = impl; - } - - @Override - public ContainerResponseProto allocateContainer(RpcController unused, - ContainerRequestProto request) throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("allocateContainer", request.getTraceID())) { - ContainerWithPipeline containerWithPipeline = impl - .allocateContainer(request.getReplicationType(), - request.getReplicationFactor(), request.getOwner()); - return ContainerResponseProto.newBuilder() - .setContainerWithPipeline(containerWithPipeline.getProtobuf()) - .setErrorCode(ContainerResponseProto.Error.success) - .build(); - - } catch (IOException e) { - throw new ServiceException(e); - } - } - - @Override - public GetContainerResponseProto getContainer( - RpcController controller, GetContainerRequestProto request) - throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("getContainer", request.getTraceID())) { - ContainerInfo container = impl.getContainer(request.getContainerID()); - return GetContainerResponseProto.newBuilder() - .setContainerInfo(container.getProtobuf()) - .build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } - - @Override - public GetContainerWithPipelineResponseProto getContainerWithPipeline( - RpcController controller, GetContainerWithPipelineRequestProto request) - throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("getContainerWithPipeline", - request.getTraceID())) { - ContainerWithPipeline container = impl - .getContainerWithPipeline(request.getContainerID()); - return GetContainerWithPipelineResponseProto.newBuilder() - .setContainerWithPipeline(container.getProtobuf()) - .build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } - - @Override - public SCMListContainerResponseProto listContainer(RpcController controller, - SCMListContainerRequestProto request) throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("listContainer", request.getTraceID())) { - long startContainerID = 0; - int count = -1; - - // Arguments check. - if (request.hasStartContainerID()) { - // End container name is given. - startContainerID = request.getStartContainerID(); - } - count = request.getCount(); - List containerList = - impl.listContainer(startContainerID, count); - SCMListContainerResponseProto.Builder builder = - SCMListContainerResponseProto.newBuilder(); - for (ContainerInfo container : containerList) { - builder.addContainers(container.getProtobuf()); - } - return builder.build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } - - @Override - public SCMDeleteContainerResponseProto deleteContainer( - RpcController controller, SCMDeleteContainerRequestProto request) - throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("deleteContainer", request.getTraceID())) { - impl.deleteContainer(request.getContainerID()); - return SCMDeleteContainerResponseProto.newBuilder().build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } - - @Override - public StorageContainerLocationProtocolProtos.NodeQueryResponseProto - queryNode(RpcController controller, - StorageContainerLocationProtocolProtos.NodeQueryRequestProto request) - throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("queryNode", request.getTraceID())) { - HddsProtos.NodeState nodeState = request.getState(); - List datanodes = impl.queryNode(nodeState, - request.getScope(), request.getPoolName()); - return StorageContainerLocationProtocolProtos - .NodeQueryResponseProto.newBuilder() - .addAllDatanodes(datanodes) - .build(); - } catch (Exception e) { - throw new ServiceException(e); - } - } - - @Override - public ObjectStageChangeResponseProto notifyObjectStageChange( - RpcController controller, ObjectStageChangeRequestProto request) - throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("notifyObjectStageChange", - request.getTraceID())) { - impl.notifyObjectStageChange(request.getType(), request.getId(), - request.getOp(), request.getStage()); - return ObjectStageChangeResponseProto.newBuilder().build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } - - @Override - public PipelineResponseProto allocatePipeline( - RpcController controller, PipelineRequestProto request) - throws ServiceException { - // TODO : Wiring this up requires one more patch. - return null; - } - - @Override - public ListPipelineResponseProto listPipelines( - RpcController controller, ListPipelineRequestProto request) - throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("listPipelines", request.getTraceID())) { - ListPipelineResponseProto.Builder builder = ListPipelineResponseProto - .newBuilder(); - List pipelines = impl.listPipelines(); - for (Pipeline pipeline : pipelines) { - HddsProtos.Pipeline protobufMessage = pipeline.getProtobufMessage(); - builder.addPipelines(protobufMessage); - } - return builder.build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } - - @Override - public ClosePipelineResponseProto closePipeline( - RpcController controller, ClosePipelineRequestProto request) - throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("closePipeline", request.getTraceID())) { - impl.closePipeline(request.getPipelineID()); - return ClosePipelineResponseProto.newBuilder().build(); - } catch (IOException e) { - throw new ServiceException(e); - } - } - - @Override - public HddsProtos.GetScmInfoResponseProto getScmInfo( - RpcController controller, HddsProtos.GetScmInfoRequestProto req) - throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("getScmInfo", req.getTraceID())) { - ScmInfo scmInfo = impl.getScmInfo(); - return HddsProtos.GetScmInfoResponseProto.newBuilder() - .setClusterId(scmInfo.getClusterId()) - .setScmId(scmInfo.getScmId()) - .build(); - } catch (IOException ex) { - throw new ServiceException(ex); - } - - } - - @Override - public InSafeModeResponseProto inSafeMode( - RpcController controller, - InSafeModeRequestProto request) throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("inSafeMode", request.getTraceID())) { - return InSafeModeResponseProto.newBuilder() - .setInSafeMode(impl.inSafeMode()).build(); - } catch (IOException ex) { - throw new ServiceException(ex); - } - } - - @Override - public ForceExitSafeModeResponseProto forceExitSafeMode( - RpcController controller, ForceExitSafeModeRequestProto request) - throws ServiceException { - try (Scope scope = TracingUtil - .importAndCreateScope("forceExitSafeMode", request.getTraceID())) { - return ForceExitSafeModeResponseProto.newBuilder() - .setExitedSafeMode(impl.forceExitSafeMode()).build(); - } catch (IOException ex) { - throw new ServiceException(ex); - } - } -} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/web/utils/JsonUtils.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/web/utils/JsonUtils.java index af56da394cd48..4177b96a354c3 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/web/utils/JsonUtils.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/web/utils/JsonUtils.java @@ -43,10 +43,9 @@ private JsonUtils() { // Never constructed } - public static String toJsonStringWithDefaultPrettyPrinter(String jsonString) + public static String toJsonStringWithDefaultPrettyPrinter(Object obj) throws IOException { - Object json = READER.readValue(jsonString); - return WRITTER.writeValueAsString(json); + return WRITTER.writeValueAsString(obj); } public static String toJsonString(Object obj) throws IOException { diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BatchOperation.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BatchOperation.java deleted file mode 100644 index 47699ebba8086..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BatchOperation.java +++ /dev/null @@ -1,90 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.utils; - -import com.google.common.collect.Lists; - -import java.util.List; - -/** - * An utility class to store a batch of DB write operations. - */ -public class BatchOperation { - - /** - * Enum for write operations. - */ - public enum Operation { - DELETE, PUT - } - - private List operations = - Lists.newArrayList(); - - /** - * Add a PUT operation into the batch. - */ - public void put(byte[] key, byte[] value) { - operations.add(new SingleOperation(Operation.PUT, key, value)); - } - - /** - * Add a DELETE operation into the batch. - */ - public void delete(byte[] key) { - operations.add(new SingleOperation(Operation.DELETE, key, null)); - - } - - public List getOperations() { - return operations; - } - - /** - * A SingleOperation represents a PUT or DELETE operation - * and the data the operation needs to manipulates. - */ - public static class SingleOperation { - - private Operation opt; - private byte[] key; - private byte[] value; - - public SingleOperation(Operation opt, byte[] key, byte[] value) { - this.opt = opt; - if (key == null) { - throw new IllegalArgumentException("key cannot be null"); - } - this.key = key.clone(); - this.value = value == null ? null : value.clone(); - } - - public Operation getOpt() { - return opt; - } - - public byte[] getKey() { - return key.clone(); - } - - public byte[] getValue() { - return value == null ? null : value.clone(); - } - } -} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreMBean.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreMBean.java deleted file mode 100644 index 88c093e62bfc7..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreMBean.java +++ /dev/null @@ -1,144 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.utils; - -import org.rocksdb.HistogramData; -import org.rocksdb.HistogramType; -import org.rocksdb.Statistics; -import org.rocksdb.TickerType; - -import javax.management.Attribute; -import javax.management.AttributeList; -import javax.management.AttributeNotFoundException; -import javax.management.DynamicMBean; -import javax.management.InvalidAttributeValueException; -import javax.management.MBeanAttributeInfo; -import javax.management.MBeanException; -import javax.management.MBeanInfo; -import javax.management.ReflectionException; -import java.lang.reflect.Method; -import java.util.ArrayList; -import java.util.HashSet; -import java.util.List; -import java.util.Set; - -/** - * Adapter JMX bean to publish all the Rocksdb metrics. - */ -public class RocksDBStoreMBean implements DynamicMBean { - - private Statistics statistics; - - private Set histogramAttributes = new HashSet<>(); - - public RocksDBStoreMBean(Statistics statistics) { - this.statistics = statistics; - histogramAttributes.add("Average"); - histogramAttributes.add("Median"); - histogramAttributes.add("Percentile95"); - histogramAttributes.add("Percentile99"); - histogramAttributes.add("StandardDeviation"); - } - - @Override - public Object getAttribute(String attribute) - throws AttributeNotFoundException, MBeanException, ReflectionException { - for (String histogramAttribute : histogramAttributes) { - if (attribute.endsWith("_" + histogramAttribute.toUpperCase())) { - String keyName = attribute - .substring(0, attribute.length() - histogramAttribute.length() - 1); - try { - HistogramData histogram = - statistics.getHistogramData(HistogramType.valueOf(keyName)); - try { - Method method = - HistogramData.class.getMethod("get" + histogramAttribute); - return method.invoke(histogram); - } catch (Exception e) { - throw new ReflectionException(e, - "Can't read attribute " + attribute); - } - } catch (IllegalArgumentException exception) { - throw new AttributeNotFoundException( - "No such attribute in RocksDB stats: " + attribute); - } - } - } - try { - return statistics.getTickerCount(TickerType.valueOf(attribute)); - } catch (IllegalArgumentException ex) { - throw new AttributeNotFoundException( - "No such attribute in RocksDB stats: " + attribute); - } - } - - @Override - public void setAttribute(Attribute attribute) - throws AttributeNotFoundException, InvalidAttributeValueException, - MBeanException, ReflectionException { - - } - - @Override - public AttributeList getAttributes(String[] attributes) { - AttributeList result = new AttributeList(); - for (String attributeName : attributes) { - try { - Object value = getAttribute(attributeName); - result.add(value); - } catch (Exception e) { - //TODO - } - } - return result; - } - - @Override - public AttributeList setAttributes(AttributeList attributes) { - return null; - } - - @Override - public Object invoke(String actionName, Object[] params, String[] signature) - throws MBeanException, ReflectionException { - return null; - } - - @Override - public MBeanInfo getMBeanInfo() { - - List attributes = new ArrayList<>(); - for (TickerType tickerType : TickerType.values()) { - attributes.add(new MBeanAttributeInfo(tickerType.name(), "long", - "RocksDBStat: " + tickerType.name(), true, false, false)); - } - for (HistogramType histogramType : HistogramType.values()) { - for (String histogramAttribute : histogramAttributes) { - attributes.add(new MBeanAttributeInfo( - histogramType.name() + "_" + histogramAttribute.toUpperCase(), - "long", "RocksDBStat: " + histogramType.name(), true, false, - false)); - } - } - - return new MBeanInfo("", "RocksDBStat", - attributes.toArray(new MBeanAttributeInfo[0]), null, null, null); - - } -} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/BatchOperation.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/BatchOperation.java deleted file mode 100644 index 2c65736a7059d..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/BatchOperation.java +++ /dev/null @@ -1,27 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -package org.apache.hadoop.utils.db; - -/** - * Class represents a batch operation, collects multiple db operation. - */ -public interface BatchOperation extends AutoCloseable { - - void close(); -} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBCheckpoint.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBCheckpoint.java deleted file mode 100644 index a3b197a55f178..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBCheckpoint.java +++ /dev/null @@ -1,58 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package org.apache.hadoop.utils.db; - -import java.io.IOException; -import java.nio.file.Path; - -/** - * Generic DB Checkpoint interface. - */ -public interface DBCheckpoint { - - /** - * Get Snapshot location. - */ - Path getCheckpointLocation(); - - /** - * Get Snapshot creation timestamp. - */ - long getCheckpointTimestamp(); - - /** - * Get last sequence number of Snapshot. - */ - long getLatestSequenceNumber(); - - /** - * Time taken in milliseconds for the checkpoint to be created. - */ - long checkpointCreationTimeTaken(); - - /** - * Destroy the contents of the specified checkpoint to ensure - * proper cleanup of the footprint on disk. - * - * @throws IOException if I/O error happens - */ - void cleanupCheckpoint() throws IOException; - -} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java deleted file mode 100644 index 5bb0fa41399ba..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java +++ /dev/null @@ -1,301 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package org.apache.hadoop.utils.db; - -import static org.apache.hadoop.ozone.OzoneConsts.OM_DB_CHECKPOINTS_DIR_NAME; - -import javax.management.ObjectName; -import java.io.File; -import java.io.IOException; -import java.nio.file.Paths; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.Hashtable; -import java.util.List; -import java.util.Map; -import java.util.Set; - -import org.apache.hadoop.hdds.HddsUtils; -import org.apache.hadoop.hdfs.DFSUtil; -import org.apache.hadoop.metrics2.util.MBeans; -import org.apache.hadoop.utils.RocksDBStoreMBean; - -import com.google.common.base.Preconditions; -import org.apache.ratis.thirdparty.com.google.common.annotations.VisibleForTesting; -import org.rocksdb.ColumnFamilyDescriptor; -import org.rocksdb.ColumnFamilyHandle; -import org.rocksdb.DBOptions; -import org.rocksdb.FlushOptions; -import org.rocksdb.RocksDB; -import org.rocksdb.RocksDBException; -import org.rocksdb.WriteOptions; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * RocksDB Store that supports creating Tables in DB. - */ -public class RDBStore implements DBStore { - private static final Logger LOG = - LoggerFactory.getLogger(RDBStore.class); - private RocksDB db; - private File dbLocation; - private final WriteOptions writeOptions; - private final DBOptions dbOptions; - private final CodecRegistry codecRegistry; - private final Hashtable handleTable; - private ObjectName statMBeanName; - private RDBCheckpointManager checkPointManager; - private String checkpointsParentDir; - - @VisibleForTesting - public RDBStore(File dbFile, DBOptions options, - Set families) throws IOException { - this(dbFile, options, families, new CodecRegistry(), false); - } - - public RDBStore(File dbFile, DBOptions options, Set families, - CodecRegistry registry, boolean readOnly) - throws IOException { - Preconditions.checkNotNull(dbFile, "DB file location cannot be null"); - Preconditions.checkNotNull(families); - Preconditions.checkArgument(families.size() > 0); - handleTable = new Hashtable<>(); - codecRegistry = registry; - final List columnFamilyDescriptors = - new ArrayList<>(); - final List columnFamilyHandles = new ArrayList<>(); - - for (TableConfig family : families) { - columnFamilyDescriptors.add(family.getDescriptor()); - } - - dbOptions = options; - dbLocation = dbFile; - // TODO: Read from the next Config. - writeOptions = new WriteOptions(); - - try { - if (readOnly) { - db = RocksDB.openReadOnly(dbOptions, dbLocation.getAbsolutePath(), - columnFamilyDescriptors, columnFamilyHandles); - } else { - db = RocksDB.open(dbOptions, dbLocation.getAbsolutePath(), - columnFamilyDescriptors, columnFamilyHandles); - } - - for (int x = 0; x < columnFamilyHandles.size(); x++) { - handleTable.put( - DFSUtil.bytes2String(columnFamilyHandles.get(x).getName()), - columnFamilyHandles.get(x)); - } - - if (dbOptions.statistics() != null) { - Map jmxProperties = new HashMap<>(); - jmxProperties.put("dbName", dbFile.getName()); - statMBeanName = HddsUtils.registerWithJmxProperties( - "Ozone", "RocksDbStore", jmxProperties, - new RocksDBStoreMBean(dbOptions.statistics())); - if (statMBeanName == null) { - LOG.warn("jmx registration failed during RocksDB init, db path :{}", - dbFile.getAbsolutePath()); - } - } - - //create checkpoints directory if not exists. - checkpointsParentDir = Paths.get(dbLocation.getParent(), - OM_DB_CHECKPOINTS_DIR_NAME).toString(); - File checkpointsDir = new File(checkpointsParentDir); - if (!checkpointsDir.exists()) { - boolean success = checkpointsDir.mkdir(); - if (!success) { - LOG.warn("Unable to create RocksDB checkpoint directory"); - } - } - - //Initialize checkpoint manager - checkPointManager = new RDBCheckpointManager(db, "om"); - - } catch (RocksDBException e) { - throw toIOException( - "Failed init RocksDB, db path : " + dbFile.getAbsolutePath(), e); - } - - if (LOG.isDebugEnabled()) { - LOG.debug("RocksDB successfully opened."); - LOG.debug("[Option] dbLocation= {}", dbLocation.getAbsolutePath()); - LOG.debug("[Option] createIfMissing = {}", options.createIfMissing()); - LOG.debug("[Option] maxOpenFiles= {}", options.maxOpenFiles()); - } - } - - public static IOException toIOException(String msg, RocksDBException e) { - String statusCode = e.getStatus() == null ? "N/A" : - e.getStatus().getCodeString(); - String errMessage = e.getMessage() == null ? "Unknown error" : - e.getMessage(); - String output = msg + "; status : " + statusCode - + "; message : " + errMessage; - return new IOException(output, e); - } - - @Override - public void compactDB() throws IOException { - if (db != null) { - try { - db.compactRange(); - } catch (RocksDBException e) { - throw toIOException("Failed to compact db", e); - } - } - } - - @Override - public void close() throws IOException { - - for (final ColumnFamilyHandle handle : handleTable.values()) { - handle.close(); - } - - if (statMBeanName != null) { - MBeans.unregister(statMBeanName); - statMBeanName = null; - } - - if (db != null) { - db.close(); - } - - if (dbOptions != null) { - dbOptions.close(); - } - - if (writeOptions != null) { - writeOptions.close(); - } - } - - @Override - public void move(KEY key, Table source, - Table dest) throws IOException { - try (BatchOperation batchOperation = initBatchOperation()) { - - VALUE value = source.get(key); - dest.putWithBatch(batchOperation, key, value); - source.deleteWithBatch(batchOperation, key); - commitBatchOperation(batchOperation); - } - } - - @Override - public void move(KEY key, VALUE value, Table source, - Table dest) throws IOException { - move(key, key, value, source, dest); - } - - @Override - public void move(KEY sourceKey, KEY destKey, VALUE value, - Table source, - Table dest) throws IOException { - try (BatchOperation batchOperation = initBatchOperation()) { - dest.putWithBatch(batchOperation, destKey, value); - source.deleteWithBatch(batchOperation, sourceKey); - commitBatchOperation(batchOperation); - } - } - - @Override - public long getEstimatedKeyCount() throws IOException { - try { - return db.getLongProperty("rocksdb.estimate-num-keys"); - } catch (RocksDBException e) { - throw toIOException("Unable to get the estimated count.", e); - } - } - - @Override - public BatchOperation initBatchOperation() { - return new RDBBatchOperation(); - } - - @Override - public void commitBatchOperation(BatchOperation operation) - throws IOException { - ((RDBBatchOperation) operation).commit(db, writeOptions); - } - - - @VisibleForTesting - protected ObjectName getStatMBeanName() { - return statMBeanName; - } - - @Override - public Table getTable(String name) throws IOException { - ColumnFamilyHandle handle = handleTable.get(name); - if (handle == null) { - throw new IOException("No such table in this DB. TableName : " + name); - } - return new RDBTable(this.db, handle, this.writeOptions); - } - - @Override - public Table getTable(String name, - Class keyType, Class valueType) throws IOException { - return new TypedTable(getTable(name), codecRegistry, keyType, - valueType); - } - - @Override - public ArrayList

    listTables() throws IOException { - ArrayList
    returnList = new ArrayList<>(); - for (ColumnFamilyHandle handle : handleTable.values()) { - returnList.add(new RDBTable(db, handle, writeOptions)); - } - return returnList; - } - - @Override - public void flush() throws IOException { - final FlushOptions flushOptions = new FlushOptions().setWaitForFlush(true); - try { - db.flush(flushOptions); - } catch (RocksDBException e) { - LOG.error("Unable to Flush RocksDB data", e); - throw toIOException("Unable to Flush RocksDB data", e); - } - } - - @Override - public DBCheckpoint getCheckpoint(boolean flush) { - final FlushOptions flushOptions = new FlushOptions().setWaitForFlush(flush); - try { - db.flush(flushOptions); - } catch (RocksDBException e) { - LOG.error("Unable to Flush RocksDB data before creating snapshot", e); - } - return checkPointManager.createCheckpoint(checkpointsParentDir); - } - - @Override - public File getDbLocation() { - return dbLocation; - } -} \ No newline at end of file diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java deleted file mode 100644 index 6de65090a92fa..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java +++ /dev/null @@ -1,279 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -package org.apache.hadoop.utils.db; - -import java.io.IOException; - -import com.google.common.annotations.VisibleForTesting; -import org.apache.hadoop.utils.db.cache.CacheKey; -import org.apache.hadoop.utils.db.cache.CacheValue; -import org.apache.hadoop.utils.db.cache.PartialTableCache; -import org.apache.hadoop.utils.db.cache.TableCache; - -/** - * Strongly typed table implementation. - *

    - * Automatically converts values and keys using a raw byte[] based table - * implementation and registered converters. - * - * @param type of the keys in the store. - * @param type of the values in the store. - */ -public class TypedTable implements Table { - - private final Table rawTable; - - private final CodecRegistry codecRegistry; - - private final Class keyType; - - private final Class valueType; - - private final TableCache, CacheValue> cache; - - - public TypedTable( - Table rawTable, - CodecRegistry codecRegistry, Class keyType, - Class valueType) { - this.rawTable = rawTable; - this.codecRegistry = codecRegistry; - this.keyType = keyType; - this.valueType = valueType; - cache = new PartialTableCache<>(); - } - - @Override - public void put(KEY key, VALUE value) throws IOException { - byte[] keyData = codecRegistry.asRawData(key); - byte[] valueData = codecRegistry.asRawData(value); - rawTable.put(keyData, valueData); - } - - @Override - public void putWithBatch(BatchOperation batch, KEY key, VALUE value) - throws IOException { - byte[] keyData = codecRegistry.asRawData(key); - byte[] valueData = codecRegistry.asRawData(value); - rawTable.putWithBatch(batch, keyData, valueData); - } - - @Override - public boolean isEmpty() throws IOException { - return rawTable.isEmpty(); - } - - /** - * Returns the value mapped to the given key in byte array or returns null - * if the key is not found. - * - * Caller's of this method should use synchronization mechanism, when - * accessing. First it will check from cache, if it has entry return the - * value, otherwise get from the RocksDB table. - * - * @param key metadata key - * @return VALUE - * @throws IOException - */ - @Override - public VALUE get(KEY key) throws IOException { - // Here the metadata lock will guarantee that cache is not updated for same - // key during get key. - CacheValue< VALUE > cacheValue = cache.get(new CacheKey<>(key)); - if (cacheValue == null) { - // If no cache for the table or if it does not exist in cache get from - // RocksDB table. - return getFromTable(key); - } else { - // We have a value in cache, return the value. - return cacheValue.getValue(); - } - } - - private VALUE getFromTable(KEY key) throws IOException { - byte[] keyBytes = codecRegistry.asRawData(key); - byte[] valueBytes = rawTable.get(keyBytes); - return codecRegistry.asObject(valueBytes, valueType); - } - - @Override - public void delete(KEY key) throws IOException { - rawTable.delete(codecRegistry.asRawData(key)); - } - - @Override - public void deleteWithBatch(BatchOperation batch, KEY key) - throws IOException { - rawTable.deleteWithBatch(batch, codecRegistry.asRawData(key)); - - } - - @Override - public TableIterator iterator() { - TableIterator> iterator = - rawTable.iterator(); - return new TypedTableIterator(iterator, keyType, valueType); - } - - @Override - public String getName() throws IOException { - return rawTable.getName(); - } - - @Override - public void close() throws Exception { - rawTable.close(); - - } - - @Override - public void addCacheEntry(CacheKey cacheKey, - CacheValue cacheValue) { - // This will override the entry if there is already entry for this key. - cache.put(cacheKey, cacheValue); - } - - - @Override - public void cleanupCache(long epoch) { - cache.cleanup(epoch); - } - - @VisibleForTesting - TableCache, CacheValue> getCache() { - return cache; - } - - public Table getRawTable() { - return rawTable; - } - - public CodecRegistry getCodecRegistry() { - return codecRegistry; - } - - public Class getKeyType() { - return keyType; - } - - public Class getValueType() { - return valueType; - } - - /** - * Key value implementation for strongly typed tables. - */ - public class TypedKeyValue implements KeyValue { - - private KeyValue rawKeyValue; - - public TypedKeyValue(KeyValue rawKeyValue) { - this.rawKeyValue = rawKeyValue; - } - - public TypedKeyValue(KeyValue rawKeyValue, - Class keyType, Class valueType) { - this.rawKeyValue = rawKeyValue; - } - - @Override - public KEY getKey() throws IOException { - return codecRegistry.asObject(rawKeyValue.getKey(), keyType); - } - - @Override - public VALUE getValue() throws IOException { - return codecRegistry.asObject(rawKeyValue.getValue(), valueType); - } - } - - /** - * Table Iterator implementation for strongly typed tables. - */ - public class TypedTableIterator implements TableIterator { - - private TableIterator> - rawIterator; - private final Class keyClass; - private final Class valueClass; - - public TypedTableIterator( - TableIterator> rawIterator, - Class keyType, - Class valueType) { - this.rawIterator = rawIterator; - keyClass = keyType; - valueClass = valueType; - } - - @Override - public void seekToFirst() { - rawIterator.seekToFirst(); - } - - @Override - public void seekToLast() { - rawIterator.seekToLast(); - } - - @Override - public TypedKeyValue seek(KEY key) throws IOException { - byte[] keyBytes = codecRegistry.asRawData(key); - KeyValue result = rawIterator.seek(keyBytes); - if (result == null) { - return null; - } - return new TypedKeyValue(result); - } - - @Override - public KEY key() throws IOException { - byte[] result = rawIterator.key(); - if (result == null) { - return null; - } - return codecRegistry.asObject(result, keyClass); - } - - @Override - public TypedKeyValue value() { - KeyValue keyValue = rawIterator.value(); - if(keyValue != null) { - return new TypedKeyValue(keyValue, keyClass, valueClass); - } - return null; - } - - @Override - public void close() throws IOException { - rawIterator.close(); - } - - @Override - public boolean hasNext() { - return rawIterator.hasNext(); - } - - @Override - public TypedKeyValue next() { - return new TypedKeyValue(rawIterator.next(), keyType, - valueType); - } - } -} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java deleted file mode 100644 index 4d3711269a168..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java +++ /dev/null @@ -1,97 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package org.apache.hadoop.utils.db.cache; - -import java.util.Iterator; -import java.util.TreeSet; -import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.concurrent.ThreadFactory; - -import com.google.common.util.concurrent.ThreadFactoryBuilder; -import org.apache.hadoop.classification.InterfaceAudience.Private; -import org.apache.hadoop.classification.InterfaceStability.Evolving; - -/** - * Cache implementation for the table, this cache is partial cache, this will - * be cleaned up, after entries are flushed to DB. - */ -@Private -@Evolving -public class PartialTableCache implements TableCache { - - private final ConcurrentHashMap cache; - private final TreeSet> epochEntries; - private ExecutorService executorService; - - - - public PartialTableCache() { - cache = new ConcurrentHashMap<>(); - epochEntries = new TreeSet<>(); - // Created a singleThreadExecutor, so one cleanup will be running at a - // time. - ThreadFactory build = new ThreadFactoryBuilder().setDaemon(true) - .setNameFormat("PartialTableCache Cleanup Thread - %d").build(); - executorService = Executors.newSingleThreadExecutor(build); - - } - - @Override - public CACHEVALUE get(CACHEKEY cachekey) { - return cache.get(cachekey); - } - - @Override - public void put(CACHEKEY cacheKey, CACHEVALUE value) { - cache.put(cacheKey, value); - epochEntries.add(new EpochEntry<>(value.getEpoch(), cacheKey)); - } - - @Override - public void cleanup(long epoch) { - executorService.submit(() -> evictCache(epoch)); - } - - @Override - public int size() { - return cache.size(); - } - - private void evictCache(long epoch) { - EpochEntry currentEntry = null; - for (Iterator> iterator = epochEntries.iterator(); - iterator.hasNext();) { - currentEntry = iterator.next(); - CACHEKEY cachekey = currentEntry.getCachekey(); - CacheValue cacheValue = cache.get(cachekey); - if (cacheValue.getEpoch() <= epoch) { - cache.remove(cachekey); - iterator.remove(); - } else { - // If currentEntry epoch is greater than epoch, we have deleted all - // entries less than specified epoch. So, we can break. - break; - } - } - } -} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java deleted file mode 100644 index 70e0b33e92974..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java +++ /dev/null @@ -1,63 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package org.apache.hadoop.utils.db.cache; - -import org.apache.hadoop.classification.InterfaceAudience.Private; -import org.apache.hadoop.classification.InterfaceStability.Evolving; - -/** - * Cache used for RocksDB tables. - * @param - * @param - */ - -@Private -@Evolving -public interface TableCache { - - /** - * Return the value for the key if it is present, otherwise return null. - * @param cacheKey - * @return CACHEVALUE - */ - CACHEVALUE get(CACHEKEY cacheKey); - - /** - * Add an entry to the cache, if the key already exists it overrides. - * @param cacheKey - * @param value - */ - void put(CACHEKEY cacheKey, CACHEVALUE value); - - /** - * Removes all the entries from the cache which are having epoch value less - * than or equal to specified epoch value. For FullTable Cache this is a - * do-nothing operation. - * @param epoch - */ - void cleanup(long epoch); - - /** - * Return the size of the cache. - * @return size - */ - int size(); -} diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/package-info.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/package-info.java deleted file mode 100644 index 8d2506a9bfee3..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/package-info.java +++ /dev/null @@ -1,18 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.utils.db.cache; diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/package-info.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/package-info.java deleted file mode 100644 index 17d676d928345..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/package-info.java +++ /dev/null @@ -1,22 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -/** - * Database interfaces for Ozone. - */ -package org.apache.hadoop.utils.db; \ No newline at end of file diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/package-info.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/package-info.java deleted file mode 100644 index 44663379129b6..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/package-info.java +++ /dev/null @@ -1,18 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.utils; diff --git a/hadoop-hdds/common/src/main/java/org/apache/ratis/RatisHelper.java b/hadoop-hdds/common/src/main/java/org/apache/ratis/RatisHelper.java deleted file mode 100644 index a63d18a726bd4..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/ratis/RatisHelper.java +++ /dev/null @@ -1,244 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.ratis; - -import java.io.IOException; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collection; -import java.util.Collections; -import java.util.List; -import java.util.UUID; -import java.util.concurrent.TimeUnit; -import java.util.stream.Collectors; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hdds.protocol.DatanodeDetails; -import org.apache.hadoop.hdds.scm.pipeline.Pipeline; -import org.apache.hadoop.hdds.security.x509.SecurityConfig; -import org.apache.hadoop.ozone.OzoneConfigKeys; -import org.apache.hadoop.ozone.OzoneConsts; - -import org.apache.ratis.client.RaftClient; -import org.apache.ratis.client.RaftClientConfigKeys; -import org.apache.ratis.conf.RaftProperties; -import org.apache.ratis.grpc.GrpcConfigKeys; -import org.apache.ratis.grpc.GrpcFactory; -import org.apache.ratis.grpc.GrpcTlsConfig; -import org.apache.ratis.proto.RaftProtos; -import org.apache.ratis.protocol.RaftGroup; -import org.apache.ratis.protocol.RaftGroupId; -import org.apache.ratis.protocol.RaftPeer; -import org.apache.ratis.protocol.RaftPeerId; -import org.apache.ratis.retry.RetryPolicies; -import org.apache.ratis.retry.RetryPolicy; -import org.apache.ratis.rpc.RpcType; -import org.apache.ratis.rpc.SupportedRpcType; -import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; -import org.apache.ratis.util.SizeInBytes; -import org.apache.ratis.util.TimeDuration; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * Ratis helper methods. - */ -public interface RatisHelper { - Logger LOG = LoggerFactory.getLogger(RatisHelper.class); - - static String toRaftPeerIdString(DatanodeDetails id) { - return id.getUuidString(); - } - - static UUID toDatanodeId(String peerIdString) { - return UUID.fromString(peerIdString); - } - - static UUID toDatanodeId(RaftPeerId peerId) { - return toDatanodeId(peerId.toString()); - } - - static UUID toDatanodeId(RaftProtos.RaftPeerProto peerId) { - return toDatanodeId(RaftPeerId.valueOf(peerId.getId())); - } - - static String toRaftPeerAddressString(DatanodeDetails id) { - return id.getIpAddress() + ":" + - id.getPort(DatanodeDetails.Port.Name.RATIS).getValue(); - } - - static RaftPeerId toRaftPeerId(DatanodeDetails id) { - return RaftPeerId.valueOf(toRaftPeerIdString(id)); - } - - static RaftPeer toRaftPeer(DatanodeDetails id) { - return new RaftPeer(toRaftPeerId(id), toRaftPeerAddressString(id)); - } - - static List toRaftPeers(Pipeline pipeline) { - return toRaftPeers(pipeline.getNodes()); - } - - static List toRaftPeers( - List datanodes) { - return datanodes.stream().map(RatisHelper::toRaftPeer) - .collect(Collectors.toList()); - } - - /* TODO: use a dummy id for all groups for the moment. - * It should be changed to a unique id for each group. - */ - RaftGroupId DUMMY_GROUP_ID = - RaftGroupId.valueOf(ByteString.copyFromUtf8("AOzoneRatisGroup")); - - RaftGroup EMPTY_GROUP = RaftGroup.valueOf(DUMMY_GROUP_ID, - Collections.emptyList()); - - static RaftGroup emptyRaftGroup() { - return EMPTY_GROUP; - } - - static RaftGroup newRaftGroup(Collection peers) { - return peers.isEmpty()? emptyRaftGroup() - : RaftGroup.valueOf(DUMMY_GROUP_ID, peers); - } - - static RaftGroup newRaftGroup(RaftGroupId groupId, - Collection peers) { - final List newPeers = peers.stream() - .map(RatisHelper::toRaftPeer) - .collect(Collectors.toList()); - return peers.isEmpty() ? RaftGroup.valueOf(groupId, Collections.emptyList()) - : RaftGroup.valueOf(groupId, newPeers); - } - - static RaftGroup newRaftGroup(Pipeline pipeline) { - return RaftGroup.valueOf(RaftGroupId.valueOf(pipeline.getId().getId()), - toRaftPeers(pipeline)); - } - - static RaftClient newRaftClient(RpcType rpcType, Pipeline pipeline, - RetryPolicy retryPolicy, int maxOutStandingRequest, - GrpcTlsConfig tlsConfig, TimeDuration timeout) throws IOException { - return newRaftClient(rpcType, toRaftPeerId(pipeline.getFirstNode()), - newRaftGroup(RaftGroupId.valueOf(pipeline.getId().getId()), - pipeline.getNodes()), retryPolicy, maxOutStandingRequest, tlsConfig, - timeout); - } - - static TimeDuration getClientRequestTimeout(Configuration conf) { - // Set the client requestTimeout - final TimeUnit timeUnit = - OzoneConfigKeys.DFS_RATIS_CLIENT_REQUEST_TIMEOUT_DURATION_DEFAULT - .getUnit(); - final long duration = conf.getTimeDuration( - OzoneConfigKeys.DFS_RATIS_CLIENT_REQUEST_TIMEOUT_DURATION_KEY, - OzoneConfigKeys.DFS_RATIS_CLIENT_REQUEST_TIMEOUT_DURATION_DEFAULT - .getDuration(), timeUnit); - final TimeDuration clientRequestTimeout = - TimeDuration.valueOf(duration, timeUnit); - return clientRequestTimeout; - } - - static RaftClient newRaftClient(RpcType rpcType, RaftPeer leader, - RetryPolicy retryPolicy, int maxOutstandingRequests, - GrpcTlsConfig tlsConfig, TimeDuration clientRequestTimeout) { - return newRaftClient(rpcType, leader.getId(), - newRaftGroup(new ArrayList<>(Arrays.asList(leader))), retryPolicy, - maxOutstandingRequests, tlsConfig, clientRequestTimeout); - } - - static RaftClient newRaftClient(RpcType rpcType, RaftPeer leader, - RetryPolicy retryPolicy, int maxOutstandingRequests, - TimeDuration clientRequestTimeout) { - return newRaftClient(rpcType, leader.getId(), - newRaftGroup(new ArrayList<>(Arrays.asList(leader))), retryPolicy, - maxOutstandingRequests, null, clientRequestTimeout); - } - - static RaftClient newRaftClient(RpcType rpcType, RaftPeerId leader, - RaftGroup group, RetryPolicy retryPolicy, int maxOutStandingRequest, - GrpcTlsConfig tlsConfig, TimeDuration clientRequestTimeout) { - LOG.trace("newRaftClient: {}, leader={}, group={}", rpcType, leader, group); - final RaftProperties properties = new RaftProperties(); - RaftConfigKeys.Rpc.setType(properties, rpcType); - RaftClientConfigKeys.Rpc - .setRequestTimeout(properties, clientRequestTimeout); - - GrpcConfigKeys.setMessageSizeMax(properties, - SizeInBytes.valueOf(OzoneConsts.OZONE_SCM_CHUNK_MAX_SIZE)); - GrpcConfigKeys.OutputStream.setOutstandingAppendsMax(properties, - maxOutStandingRequest); - - RaftClient.Builder builder = RaftClient.newBuilder() - .setRaftGroup(group) - .setLeaderId(leader) - .setProperties(properties) - .setRetryPolicy(retryPolicy); - - // TODO: GRPC TLS only for now, netty/hadoop RPC TLS support later. - if (tlsConfig != null && rpcType == SupportedRpcType.GRPC) { - builder.setParameters(GrpcFactory.newRaftParameters(tlsConfig)); - } - return builder.build(); - } - - static GrpcTlsConfig createTlsClientConfig(SecurityConfig conf) { - if (conf.isGrpcTlsEnabled()) { - if (conf.isGrpcMutualTlsRequired()) { - return new GrpcTlsConfig( - null, null, conf.getTrustStoreFile(), false); - } else { - return new GrpcTlsConfig(conf.getClientPrivateKeyFile(), - conf.getClientCertChainFile(), conf.getTrustStoreFile(), true); - } - } - return null; - } - - static GrpcTlsConfig createTlsServerConfig(SecurityConfig conf) { - if (conf.isGrpcTlsEnabled()) { - if (conf.isGrpcMutualTlsRequired()) { - return new GrpcTlsConfig( - conf.getServerPrivateKeyFile(), conf.getServerCertChainFile(), null, - false); - } else { - return new GrpcTlsConfig(conf.getServerPrivateKeyFile(), - conf.getServerCertChainFile(), conf.getClientCertChainFile(), true); - } - } - return null; - } - - static RetryPolicy createRetryPolicy(Configuration conf) { - int maxRetryCount = - conf.getInt(OzoneConfigKeys.DFS_RATIS_CLIENT_REQUEST_MAX_RETRIES_KEY, - OzoneConfigKeys. - DFS_RATIS_CLIENT_REQUEST_MAX_RETRIES_DEFAULT); - long retryInterval = conf.getTimeDuration(OzoneConfigKeys. - DFS_RATIS_CLIENT_REQUEST_RETRY_INTERVAL_KEY, OzoneConfigKeys. - DFS_RATIS_CLIENT_REQUEST_RETRY_INTERVAL_DEFAULT - .toIntExact(TimeUnit.MILLISECONDS), TimeUnit.MILLISECONDS); - TimeDuration sleepDuration = - TimeDuration.valueOf(retryInterval, TimeUnit.MILLISECONDS); - RetryPolicy retryPolicy = RetryPolicies - .retryUpToMaximumCountWithFixedSleep(maxRetryCount, sleepDuration); - return retryPolicy; - } -} diff --git a/hadoop-hdds/common/src/main/java/org/apache/ratis/package-info.java b/hadoop-hdds/common/src/main/java/org/apache/ratis/package-info.java deleted file mode 100644 index c13c20c60604b..0000000000000 --- a/hadoop-hdds/common/src/main/java/org/apache/ratis/package-info.java +++ /dev/null @@ -1,22 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.ratis; - -/** - * This package contains classes related to Apache Ratis. - */ diff --git a/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto b/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto index fd572ada622bc..1bfe4d1247c5f 100644 --- a/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto +++ b/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto @@ -144,6 +144,7 @@ enum Result { CONTAINER_NOT_OPEN = 39; CONTAINER_MISSING = 40; BLOCK_TOKEN_VERIFICATION_FAILED = 41; + ERROR_IN_DB_SYNC = 42; } /** @@ -247,8 +248,9 @@ message ContainerDataProto { optional ContainerType containerType = 10 [default = KeyValueContainer]; } -message ContainerIdSetProto { - repeated int64 containerId = 1; +message Container2BCSIDMapProto { + // repeated Container2BCSIDMapEntryProto container2BCSID = 1; + map container2BCSID = 1; } enum ContainerType { diff --git a/hadoop-hdds/common/src/main/proto/FSProtos.proto b/hadoop-hdds/common/src/main/proto/FSProtos.proto new file mode 100644 index 0000000000000..c3b768ab67ed1 --- /dev/null +++ b/hadoop-hdds/common/src/main/proto/FSProtos.proto @@ -0,0 +1,78 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * These .proto interfaces are private and stable. + * Please see http://wiki.apache.org/hadoop/Compatibility + * for what changes are allowed for a *stable* .proto interface. + */ + +option java_package = "org.apache.hadoop.fs"; +option java_outer_classname = "FSProtos"; +option java_generic_services = true; +option java_generate_equals_and_hash = true; +package hadoop.fs; + +message FsPermissionProto { + required uint32 perm = 1; // UNIX-style mode bits +} + +/* + * FileStatus encoding. Field IDs match those from HdfsFileStatusProto, but + * cross-serialization is not an explicitly supported use case. Unlike HDFS, + * most fields are optional and do not define defaults. + */ +message FileStatusProto { + enum FileType { + FT_DIR = 1; + FT_FILE = 2; + FT_SYMLINK = 3; + } + enum Flags { + HAS_ACL = 0x01; // has ACLs + HAS_CRYPT = 0x02; // encrypted + HAS_EC = 0x04; // erasure coded + SNAPSHOT_ENABLED = 0x08; // snapshot enabled + } + required FileType fileType = 1; + required string path = 2; + optional uint64 length = 3; + optional FsPermissionProto permission = 4; + optional string owner = 5; + optional string group = 6; + optional uint64 modification_time = 7; + optional uint64 access_time = 8; + optional string symlink = 9; + optional uint32 block_replication = 10; + optional uint64 block_size = 11; + // locations = 12 + // alias = 13 + // childrenNum = 14 + optional bytes encryption_data = 15; + // storagePolicy = 16 + optional bytes ec_data = 17; + optional uint32 flags = 18 [default = 0]; +} + +/** + * Placeholder type for consistent basic FileSystem operations. + */ +message LocalFileSystemPathHandleProto { + optional uint64 mtime = 1; + optional string path = 2; +} diff --git a/hadoop-hdds/common/src/main/proto/SCMSecurityProtocol.proto b/hadoop-hdds/common/src/main/proto/SCMSecurityProtocol.proto index 5fcd98e6b9172..72e0e9f66f7d4 100644 --- a/hadoop-hdds/common/src/main/proto/SCMSecurityProtocol.proto +++ b/hadoop-hdds/common/src/main/proto/SCMSecurityProtocol.proto @@ -30,17 +30,61 @@ option java_generic_services = true; option java_generate_equals_and_hash = true; -package hadoop.hdds; +package hadoop.hdds.security; import "hdds.proto"; +/** +All commands is send as request and all response come back via +Response class. If adding new functions please follow this protocol, since +our tracing and visibility tools depend on this pattern. +*/ +message SCMSecurityRequest { + required Type cmdType = 1; // Type of the command + + optional string traceID = 2; + + optional SCMGetDataNodeCertRequestProto getDataNodeCertRequest = 3; + optional SCMGetOMCertRequestProto getOMCertRequest = 4; + optional SCMGetCertificateRequestProto getCertificateRequest = 5; + optional SCMGetCACertificateRequestProto getCACertificateRequest = 6; + +} + +message SCMSecurityResponse { + required Type cmdType = 1; // Type of the command + + // A string that identifies this command, we generate Trace ID in Ozone + // frontend and this allows us to trace that command all over ozone. + optional string traceID = 2; + + optional bool success = 3 [default = true]; + + optional string message = 4; + + required Status status = 5; + + optional SCMGetCertResponseProto getCertResponseProto = 6; + +} + +enum Type { + GetDataNodeCertificate = 1; + GetOMCertificate = 2; + GetCertificate = 3; + GetCACertificate = 4; +} + +enum Status { + OK = 1; +} /** * This message is send by data node to prove its identity and get an SCM * signed certificate. */ message SCMGetDataNodeCertRequestProto { - required DatanodeDetailsProto datanodeDetails = 1; - required string CSR = 2; + required DatanodeDetailsProto datanodeDetails = 1; + required string CSR = 2; } /** @@ -48,15 +92,15 @@ message SCMGetDataNodeCertRequestProto { * signed certificate. */ message SCMGetOMCertRequestProto { - required OzoneManagerDetailsProto omDetails = 1; - required string CSR = 2; + required OzoneManagerDetailsProto omDetails = 1; + required string CSR = 2; } /** * Proto request to get a certificate with given serial id. */ message SCMGetCertificateRequestProto { - required string certSerialId = 1; + required string certSerialId = 1; } /** @@ -69,38 +113,17 @@ message SCMGetCACertificateRequestProto { * Returns a certificate signed by SCM. */ message SCMGetCertResponseProto { - enum ResponseCode { - success = 1; - authenticationFailed = 2; - invalidCSR = 3; - } - required ResponseCode responseCode = 1; - required string x509Certificate = 2; // Base64 encoded X509 certificate. + enum ResponseCode { + success = 1; + authenticationFailed = 2; + invalidCSR = 3; + } + required ResponseCode responseCode = 1; + required string x509Certificate = 2; // Base64 encoded X509 certificate. + optional string x509CACertificate = 3; // Base64 encoded CA X509 certificate. } service SCMSecurityProtocolService { - /** - * Get SCM signed certificate for DataNode. - */ - rpc getDataNodeCertificate (SCMGetDataNodeCertRequestProto) returns - (SCMGetCertResponseProto); - - /** - * Get SCM signed certificate for DataNode. - */ - rpc getOMCertificate (SCMGetOMCertRequestProto) returns - (SCMGetCertResponseProto); - - /** - * Get SCM signed certificate for DataNode. - */ - rpc getCertificate (SCMGetCertificateRequestProto) returns - (SCMGetCertResponseProto); - - /** - * Get SCM signed certificate for DataNode. - */ - rpc getCACertificate (SCMGetCACertificateRequestProto) returns - (SCMGetCertResponseProto); + rpc submitRequest (SCMSecurityRequest) returns (SCMSecurityResponse); } diff --git a/hadoop-hdds/common/src/main/proto/ScmBlockLocationProtocol.proto b/hadoop-hdds/common/src/main/proto/ScmBlockLocationProtocol.proto index 6745c6ee14bb5..fc7a5988ce669 100644 --- a/hadoop-hdds/common/src/main/proto/ScmBlockLocationProtocol.proto +++ b/hadoop-hdds/common/src/main/proto/ScmBlockLocationProtocol.proto @@ -26,13 +26,96 @@ option java_package = "org.apache.hadoop.hdds.protocol.proto"; option java_outer_classname = "ScmBlockLocationProtocolProtos"; option java_generic_services = true; option java_generate_equals_and_hash = true; -package hadoop.hdds; +package hadoop.hdds.block; import "hdds.proto"; // SCM Block protocol +enum Type { + AllocateScmBlock = 11; + DeleteScmKeyBlocks = 12; + GetScmInfo = 13; + SortDatanodes = 14; +} + +message SCMBlockLocationRequest { + required Type cmdType = 1; // Type of the command + + // A string that identifies this command, we generate Trace ID in Ozone + // frontend and this allows us to trace that command all over ozone. + optional string traceID = 2; + + optional UserInfo userInfo = 3; + + optional AllocateScmBlockRequestProto allocateScmBlockRequest = 11; + optional DeleteScmKeyBlocksRequestProto deleteScmKeyBlocksRequest = 12; + optional hadoop.hdds.GetScmInfoRequestProto getScmInfoRequest = 13; + optional SortDatanodesRequestProto sortDatanodesRequest = 14; +} + +message SCMBlockLocationResponse { + required Type cmdType = 1; // Type of the command + + // A string that identifies this command, we generate Trace ID in Ozone + // frontend and this allows us to trace that command all over ozone. + optional string traceID = 2; + + optional bool success = 3 [default=true]; + + optional string message = 4; + + required Status status = 5; + + optional string leaderOMNodeId = 6; + + optional AllocateScmBlockResponseProto allocateScmBlockResponse = 11; + optional DeleteScmKeyBlocksResponseProto deleteScmKeyBlocksResponse = 12; + optional hadoop.hdds.GetScmInfoResponseProto getScmInfoResponse = 13; + optional SortDatanodesResponseProto sortDatanodesResponse = 14; +} + +/** + User information which will be extracted during RPC context and used + during validating Acl. +*/ +message UserInfo { + optional string userName = 1; + optional string remoteAddress = 3; +} + +enum Status { + OK = 1; + FAILED_TO_LOAD_NODEPOOL = 2; + FAILED_TO_FIND_NODE_IN_POOL = 3; + FAILED_TO_FIND_HEALTHY_NODES = 4; + FAILED_TO_FIND_NODES_WITH_SPACE = 5; + FAILED_TO_FIND_SUITABLE_NODE = 6; + INVALID_CAPACITY = 7; + INVALID_BLOCK_SIZE = 8; + SAFE_MODE_EXCEPTION = 9; + FAILED_TO_LOAD_OPEN_CONTAINER = 10; + FAILED_TO_ALLOCATE_CONTAINER = 11; + FAILED_TO_CHANGE_CONTAINER_STATE = 12; + FAILED_TO_CHANGE_PIPELINE_STATE = 13; + CONTAINER_EXISTS = 14; + FAILED_TO_FIND_CONTAINER = 15; + FAILED_TO_FIND_CONTAINER_WITH_SPACE = 16; + BLOCK_EXISTS = 17; + FAILED_TO_FIND_BLOCK = 18; + IO_EXCEPTION = 19; + UNEXPECTED_CONTAINER_STATE = 20; + SCM_NOT_INITIALIZED = 21; + DUPLICATE_DATANODE = 22; + NO_SUCH_DATANODE = 23; + NO_REPLICA_FOUND = 24; + FAILED_TO_FIND_ACTIVE_PIPELINE = 25; + FAILED_TO_INIT_CONTAINER_PLACEMENT_POLICY = 26; + FAILED_TO_ALLOCATE_ENOUGH_BLOCKS = 27; + INTERNAL_ERROR = 29; +} + /** * Request send to SCM asking allocate block of specified size. */ @@ -42,7 +125,6 @@ message AllocateScmBlockRequestProto { required ReplicationType type = 3; required hadoop.hdds.ReplicationFactor factor = 4; required string owner = 5; - optional string traceID = 6; optional ExcludeListProto excludeList = 7; } @@ -73,8 +155,6 @@ message KeyBlocks { */ message DeleteScmKeyBlocksResponseProto { repeated DeleteKeyBlocksResultProto results = 1; - optional string traceID = 2; - } /** @@ -105,38 +185,28 @@ message AllocateBlockResponse { * Reply from SCM indicating that the container. */ message AllocateScmBlockResponseProto { - enum Error { - success = 1; - errorNotEnoughSpace = 2; - errorSizeTooBig = 3; - unknownFailure = 4; - } - required Error errorCode = 1; - optional string errorMessage = 2; repeated AllocateBlockResponse blocks = 3; } +/** + * Datanode sort request sent by OM to SCM, it contains + * multiple number of datanodes. + */ +message SortDatanodesRequestProto{ + required string client = 1; + repeated string nodeNetworkName = 2; +} + +message SortDatanodesResponseProto{ + repeated DatanodeDetailsProto node = 1; +} + /** * Protocol used from OzoneManager to StorageContainerManager. * See request and response messages for details of the RPC calls. */ service ScmBlockLocationProtocolService { - /** - * Creates a block entry in SCM. - */ - rpc allocateScmBlock(AllocateScmBlockRequestProto) - returns (AllocateScmBlockResponseProto); - - /** - * Deletes blocks for a set of object keys from SCM. - */ - rpc deleteScmKeyBlocks(DeleteScmKeyBlocksRequestProto) - returns (DeleteScmKeyBlocksResponseProto); - - /** - * Gets the scmInfo from SCM. - */ - rpc getScmInfo(hadoop.hdds.GetScmInfoRequestProto) - returns (hadoop.hdds.GetScmInfoResponseProto); + rpc send(SCMBlockLocationRequest) + returns (SCMBlockLocationResponse); } diff --git a/hadoop-hdds/common/src/main/proto/Security.proto b/hadoop-hdds/common/src/main/proto/Security.proto new file mode 100644 index 0000000000000..a3ce7392d0b0f --- /dev/null +++ b/hadoop-hdds/common/src/main/proto/Security.proto @@ -0,0 +1,73 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * These .proto interfaces are private and stable. + * Please see http://wiki.apache.org/hadoop/Compatibility + * for what changes are allowed for a *stable* .proto interface. + */ + +option java_package = "org.apache.hadoop.security.proto"; +option java_outer_classname = "SecurityProtos"; +option java_generic_services = true; +option java_generate_equals_and_hash = true; +package hadoop.common; + +/** + * Security token identifier + */ +message TokenProto { + required bytes identifier = 1; + required bytes password = 2; + required string kind = 3; + required string service = 4; +} + +message CredentialsKVProto { + required string alias = 1; + optional hadoop.common.TokenProto token = 2; + optional bytes secret = 3; +} + +message CredentialsProto { + repeated hadoop.common.CredentialsKVProto tokens = 1; + repeated hadoop.common.CredentialsKVProto secrets = 2; +} + +message GetDelegationTokenRequestProto { + required string renewer = 1; +} + +message GetDelegationTokenResponseProto { + optional hadoop.common.TokenProto token = 1; +} + +message RenewDelegationTokenRequestProto { + required hadoop.common.TokenProto token = 1; +} + +message RenewDelegationTokenResponseProto { + required uint64 newExpiryTime = 1; +} + +message CancelDelegationTokenRequestProto { + required hadoop.common.TokenProto token = 1; +} + +message CancelDelegationTokenResponseProto { // void response +} diff --git a/hadoop-hdds/common/src/main/proto/StorageContainerLocationProtocol.proto b/hadoop-hdds/common/src/main/proto/StorageContainerLocationProtocol.proto index 7ad19b112a6b1..8ea72b6cd1780 100644 --- a/hadoop-hdds/common/src/main/proto/StorageContainerLocationProtocol.proto +++ b/hadoop-hdds/common/src/main/proto/StorageContainerLocationProtocol.proto @@ -26,10 +26,100 @@ option java_package = "org.apache.hadoop.hdds.protocol.proto"; option java_outer_classname = "StorageContainerLocationProtocolProtos"; option java_generic_services = true; option java_generate_equals_and_hash = true; -package hadoop.hdds; +package hadoop.hdds.container; import "hdds.proto"; +/** + All functions are dispatched as Request/Response under Ozone. + if you add newe functions, please add them using same pattern. +*/ +message ScmContainerLocationRequest { + required Type cmdType = 1; // Type of the command + + // A string that identifies this command, we generate Trace ID in Ozone + // frontend and this allows us to trace that command all over ozone. + optional string traceID = 2; + + optional ContainerRequestProto containerRequest = 6; + optional GetContainerRequestProto getContainerRequest = 7; + optional GetContainerWithPipelineRequestProto getContainerWithPipelineRequest = 8; + optional SCMListContainerRequestProto scmListContainerRequest = 9; + optional SCMDeleteContainerRequestProto scmDeleteContainerRequest = 10; + optional NodeQueryRequestProto nodeQueryRequest = 11; + optional ObjectStageChangeRequestProto objectStageChangeRequest = 12; + optional PipelineRequestProto pipelineRequest = 13; + optional ListPipelineRequestProto listPipelineRequest = 14; + optional ActivatePipelineRequestProto activatePipelineRequest = 15; + optional DeactivatePipelineRequestProto deactivatePipelineRequest = 16; + optional ClosePipelineRequestProto closePipelineRequest = 17; + optional GetScmInfoRequestProto getScmInfoRequest = 18; + optional InSafeModeRequestProto inSafeModeRequest = 19; + optional ForceExitSafeModeRequestProto forceExitSafeModeRequest = 20; + optional StartReplicationManagerRequestProto startReplicationManagerRequest = 21; + optional StopReplicationManagerRequestProto stopReplicationManagerRequest = 22; + optional ReplicationManagerStatusRequestProto seplicationManagerStatusRequest = 23; + +} + +message ScmContainerLocationResponse { + required Type cmdType = 1; // Type of the command + + optional string traceID = 2; + + optional bool success = 3 [default = true]; + + optional string message = 4; + + required Status status = 5; + + optional ContainerResponseProto containerResponse = 6; + optional GetContainerResponseProto getContainerResponse = 7; + optional GetContainerWithPipelineResponseProto getContainerWithPipelineResponse = 8; + optional SCMListContainerResponseProto scmListContainerResponse = 9; + optional SCMDeleteContainerResponseProto scmDeleteContainerResponse = 10; + optional NodeQueryResponseProto nodeQueryResponse = 11; + optional ObjectStageChangeResponseProto objectStageChangeResponse = 12; + optional PipelineResponseProto pipelineResponse = 13; + optional ListPipelineResponseProto listPipelineResponse = 14; + optional ActivatePipelineResponseProto activatePipelineResponse = 15; + optional DeactivatePipelineResponseProto deactivatePipelineResponse = 16; + optional ClosePipelineResponseProto closePipelineResponse = 17; + optional GetScmInfoResponseProto getScmInfoResponse = 18; + optional InSafeModeResponseProto inSafeModeResponse = 19; + optional ForceExitSafeModeResponseProto forceExitSafeModeResponse = 20; + optional StartReplicationManagerResponseProto startReplicationManagerResponse = 21; + optional StopReplicationManagerResponseProto stopReplicationManagerResponse = 22; + optional ReplicationManagerStatusResponseProto replicationManagerStatusResponse = 23; + enum Status { + OK = 1; + CONTAINER_ALREADY_EXISTS = 2; + CONTAINER_IS_MISSING = 3; + } +} + +enum Type { + + AllocateContainer = 1; + GetContainer = 2; + GetContainerWithPipeline = 3; + ListContainer = 4; + DeleteContainer = 5; + QueryNode = 6; + NotifyObjectStageChange = 7; + AllocatePipeline = 8; + ListPipelines = 9; + ActivatePipeline = 10; + DeactivatePipeline = 11; + ClosePipeline = 12; + GetScmInfo = 13; + InSafeMode = 14; + ForceExitSafeMode = 15; + StartReplicationManager = 16; + StopReplicationManager = 17; + GetReplicationManagerStatus = 18; +} + /** * Request send to SCM asking where the container should be created. */ @@ -167,6 +257,22 @@ message ListPipelineResponseProto { repeated Pipeline pipelines = 1; } +message ActivatePipelineRequestProto { + required PipelineID pipelineID = 1; + optional string traceID = 2; +} + +message ActivatePipelineResponseProto { +} + +message DeactivatePipelineRequestProto { + required PipelineID pipelineID = 1; + optional string traceID = 2; +} + +message DeactivatePipelineResponseProto { +} + message ClosePipelineRequestProto { required PipelineID pipelineID = 1; optional string traceID = 2; @@ -192,87 +298,33 @@ message ForceExitSafeModeResponseProto { required bool exitedSafeMode = 1; } +message StartReplicationManagerRequestProto { + optional string traceID = 1; +} + +message StartReplicationManagerResponseProto { +} + +message StopReplicationManagerRequestProto { + optional string traceID = 1; +} + +message StopReplicationManagerResponseProto { +} + +message ReplicationManagerStatusRequestProto { + optional string traceID = 1; +} + +message ReplicationManagerStatusResponseProto { + required bool isRunning = 1; +} + /** * Protocol used from an HDFS node to StorageContainerManager. See the request * and response messages for details of the RPC calls. */ service StorageContainerLocationProtocolService { + rpc submitRequest (ScmContainerLocationRequest) returns (ScmContainerLocationResponse); - /** - * Creates a container entry in SCM. - */ - rpc allocateContainer(ContainerRequestProto) returns (ContainerResponseProto); - - /** - * Returns the pipeline for a given container. - */ - rpc getContainer(GetContainerRequestProto) returns (GetContainerResponseProto); - - /** - * Returns the pipeline for a given container. - */ - rpc getContainerWithPipeline(GetContainerWithPipelineRequestProto) returns (GetContainerWithPipelineResponseProto); - - rpc listContainer(SCMListContainerRequestProto) returns (SCMListContainerResponseProto); - - /** - * Deletes a container in SCM. - */ - rpc deleteContainer(SCMDeleteContainerRequestProto) returns (SCMDeleteContainerResponseProto); - - /** - * Returns a set of Nodes that meet a criteria. - */ - rpc queryNode(NodeQueryRequestProto) returns (NodeQueryResponseProto); - - /** - * Notify from client when begin or finish container or pipeline operations on datanodes. - */ - rpc notifyObjectStageChange(ObjectStageChangeRequestProto) returns (ObjectStageChangeResponseProto); - - /* - * Apis that Manage Pipelines. - * - * Pipelines are abstractions offered by SCM and Datanode that allows users - * to create a replication pipeline. - * - * These following APIs allow command line programs like SCM CLI to list - * and manage pipelines. - */ - - /** - * Creates a replication pipeline. - */ - rpc allocatePipeline(PipelineRequestProto) - returns (PipelineResponseProto); - - /** - * Returns the list of Pipelines managed by SCM. - */ - rpc listPipelines(ListPipelineRequestProto) - returns (ListPipelineResponseProto); - - /** - * Closes a pipeline. - */ - rpc closePipeline(ClosePipelineRequestProto) - returns (ClosePipelineResponseProto); - - /** - * Returns information about SCM. - */ - rpc getScmInfo(GetScmInfoRequestProto) - returns (GetScmInfoResponseProto); - - /** - * Checks if SCM is in SafeMode. - */ - rpc inSafeMode(InSafeModeRequestProto) - returns (InSafeModeResponseProto); - - /** - * Returns information about SCM. - */ - rpc forceExitSafeMode(ForceExitSafeModeRequestProto) - returns (ForceExitSafeModeResponseProto); } diff --git a/hadoop-hdds/common/src/main/proto/hdds.proto b/hadoop-hdds/common/src/main/proto/hdds.proto index ddde7ea93acb5..d2bb355ff8a4a 100644 --- a/hadoop-hdds/common/src/main/proto/hdds.proto +++ b/hadoop-hdds/common/src/main/proto/hdds.proto @@ -34,6 +34,9 @@ message DatanodeDetailsProto { required string hostName = 3; // hostname repeated Port ports = 4; optional string certSerialId = 5; // Certificate serial id. + // network name, can be Ip address or host name, depends + optional string networkName = 6; + optional string networkLocation = 7; // Network topology location } /** @@ -59,7 +62,8 @@ message PipelineID { enum PipelineState { PIPELINE_ALLOCATED = 1; PIPELINE_OPEN = 2; - PIPELINE_CLOSED = 3; + PIPELINE_DORMANT = 3; + PIPELINE_CLOSED = 4; } message Pipeline { @@ -70,6 +74,7 @@ message Pipeline { optional ReplicationType type = 4 [default = STAND_ALONE]; optional ReplicationFactor factor = 5 [default = ONE]; required PipelineID id = 6; + repeated uint32 memberOrders = 7; } message KeyValue { diff --git a/hadoop-hdds/config/src/main/resources/META-INF/services/javax.annotation.processing.Processor b/hadoop-hdds/common/src/main/resources/META-INF/services/javax.annotation.processing.Processor similarity index 100% rename from hadoop-hdds/config/src/main/resources/META-INF/services/javax.annotation.processing.Processor rename to hadoop-hdds/common/src/main/resources/META-INF/services/javax.annotation.processing.Processor diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml b/hadoop-hdds/common/src/main/resources/ozone-default.xml index a46ddb16a8950..b0a59fa209ccb 100644 --- a/hadoop-hdds/common/src/main/resources/ozone-default.xml +++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml @@ -104,6 +104,14 @@ Byte limit for ratis leader's log appender queue. + + dfs.container.ratis.log.purge.gap + 1000000 + OZONE, DEBUG, CONTAINER, RATIS + Purge gap between the last purged commit index + and the current index, when the leader decides to purge its log. + + dfs.container.ratis.datanode.storage.dir @@ -178,6 +186,15 @@ taken. + + dfs.container.ratis.statemachine.max.pending.apply-transactions + 10000 + OZONE, RATIS + Maximum number of pending apply transactions in a data + pipeline. The default value is kept same as default snapshot threshold + dfs.ratis.snapshot.threshold. + + dfs.container.ratis.num.write.chunk.threads 60 @@ -186,6 +203,14 @@ will use for writing chunks (60 by default). + + dfs.container.ratis.leader.num.pending.requests + 4096 + OZONE, RATIS, PERFORMANCE + Maximum number of pending requests after which the leader + starts rejecting requests from client. + + dfs.container.ratis.replication.level MAJORITY @@ -205,10 +230,10 @@ dfs.container.ratis.segment.size - 16KB + 1MB OZONE, RATIS, PERFORMANCE The size of the raft segment used by Apache Ratis on datanodes. - (16 KB by default) + (1 MB by default) @@ -262,10 +287,10 @@ dfs.ratis.leader.election.minimum.timeout.duration - 1s + 5s OZONE, RATIS, MANAGEMENT The minimum timeout duration for ratis leader election. - Default is 1s. + Default is 5s. @@ -314,7 +339,7 @@ hdds.prometheus.endpoint.enabled - false + true OZONE, MANAGEMENT Enable prometheus compatible metric page on the HTTP servers. @@ -444,19 +469,6 @@ there is no wait. - - ozone.client.protocol - org.apache.hadoop.ozone.client.rpc.RpcClient - OZONE, CLIENT, MANAGEMENT - Protocol class to be used by the client to connect to ozone - cluster. - The build-in implementation includes: - org.apache.hadoop.ozone.client.rpc.RpcClient for RPC - org.apache.hadoop.ozone.client.rest.RestClient for REST - The default is the RpClient. Please do not change this unless you have a - very good understanding of what you are doing. - - ozone.client.socket.timeout 5000ms @@ -533,21 +545,13 @@ ozone.om.address - + 0.0.0.0:9862 OM, REQUIRED The address of the Ozone OM service. This allows clients to discover the address of the OM. - - ozone.om.group.rights - READ_WRITE - OM, SECURITY - - Default group permissions in Ozone OM. - - ozone.om.handler.count.key 20 @@ -603,7 +607,7 @@ The actual address the OM web server will bind to using HTTPS. If this optional address is set, it overrides only the hostname portion of - ozone.om.http-address. + ozone.om.https-address. @@ -640,14 +644,6 @@ of buckets or keys inside each bucket a user can create. - - ozone.om.user.rights - READ_WRITE - OM, SECURITY - - Default user permissions used in OM. - - ozone.om.db.dirs @@ -658,17 +654,18 @@ exist then the OM will attempt to create it. If undefined, then the OM will log a warning and fallback to - ozone.metadata.dirs. + ozone.metadata.dirs. This fallback approach is not recommended for + production environments. ozone.metadata.dirs - OZONE, OM, SCM, CONTAINER, STORAGE + OZONE, OM, SCM, CONTAINER, STORAGE, REQUIRED - This setting is the fallback location for SCM, OM and DataNodes - to store their metadata. This setting may be used in test/PoC clusters - to simplify configuration. + This setting is the fallback location for SCM, OM, Recon and DataNodes + to store their metadata. This setting may be used only in test/PoC + clusters to simplify configuration. For production clusters or any time you care about performance, it is recommended that ozone.om.db.dirs, ozone.scm.db.dirs and @@ -710,7 +707,8 @@ does not exist then the SCM will attempt to create it. If undefined, then the SCM will log a warning and fallback to - ozone.metadata.dirs. + ozone.metadata.dirs. This fallback approach is not recommended for + production environments. @@ -832,8 +830,10 @@ org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom OZONE, MANAGEMENT - Placement policy class for containers. - Defaults to SCMContainerPlacementRandom.class + + The full name of class which implements org.apache.hadoop.hdds.scm.container.placement.algorithms.ContainerPlacementPolicy. + The class decides which datanode will be used to host the container replica. If not set, + org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom will be used as default value. @@ -1035,7 +1035,7 @@ The actual address the SCM web server will bind to using HTTPS. If this optional address is set, it overrides only the hostname portion of - ozone.scm.http-address. + ozone.scm.https-address. @@ -1069,39 +1069,6 @@ - - - scm.container.client.idle.threshold - 10s - OZONE, PERFORMANCE - - In the standalone pipelines, the SCM clients use netty to - communicate with the container. It also uses connection pooling to - reduce client side overheads. This allows a connection to stay idle for - a while before the connection is closed. - - - - scm.container.client.max.size - 256 - OZONE, PERFORMANCE - - Controls the maximum number of connections that we cached via - clientconnection pooling. If the number of connection - exceed this count then the oldest idle connection is evicted. - - - - - scm.container.client.max.outstanding.requests - 100 - OZONE, PERFORMANCE - - Controls the maximum number of outstanding async requests that can be - handled by the Standalone as well as Ratis client. - - - ozone.scm.container.creation.lease.timeout 60s @@ -1210,13 +1177,14 @@ hadoop.tags.custom - OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM,CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY + OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM, + CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,RECON ozone.tags.system OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM, - CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,TOKEN,TLS + CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,TOKEN,TLS,RECON @@ -1255,7 +1223,7 @@ hdds.datanode.plugins - org.apache.hadoop.ozone.web.OzoneHddsDatanodeService + Comma-separated list of HDDS datanode plug-ins to be activated when HDDS service starts as part of datanode. @@ -1374,7 +1342,7 @@ - hdds.containerscrub.enabled + hdds.container.scrub.enabled false DATANODE @@ -1519,7 +1487,7 @@ 5m OZONE, OM Time interval used to store the omMetrics in to a - file. Background thread perodically stores the OM metrics in to a + file. Background thread periodically stores the OM metrics in to a file. Unit could be defined with postfix (ns,ms,s,m,h,d) @@ -1561,6 +1529,17 @@ + + ozone.om.lock.fair + false + If this is true, the Ozone Manager lock will be used in Fair + mode, which will schedule threads in the order received/queued. If this is + false, uses non-fair ordering. See + java.util.concurrent.locks.ReentrantReadWriteLock + for more information on fair/non-fair locks. + + + ozone.om.ratis.enable false @@ -1596,6 +1575,8 @@ logs. If this is not set then default metadata dirs is used. A warning will be logged if this not set. Ideally, this should be mapped to a fast disk like an SSD. + If undefined, OM ratis storage dir will fallback to ozone.metadata.dirs. + This fallback approach is not recommended for production environments. @@ -1631,6 +1612,14 @@ Byte limit for Raft's Log Worker queue. + + ozone.om.ratis.log.purge.gap + 1000000 + OZONE, OM, RATIS + The minimum gap between log indices for Raft server to purge + its log segments after taking snapshot. + + ozone.om.ratis.snapshot.auto.trigger.threshold @@ -1714,6 +1703,45 @@ . + + ozone.om.ratis.snapshot.dir + + OZONE, OM, STORAGE, MANAGEMENT, RATIS + This directory is used for storing OM's snapshot + related files like the ratisSnapshotIndex and DB checkpoint from leader + OM. + If undefined, OM snapshot dir will fallback to ozone.om.ratis.storage.dir. + This fallback approach is not recommended for production environments. + + + + ozone.om.snapshot.provider.socket.timeout + 5000s + OZONE, OM, HA, MANAGEMENT + + Socket timeout for HTTP call made by OM Snapshot Provider to request + OM snapshot from OM Leader. + + + + ozone.om.snapshot.provider.connection.timeout + 5000s + OZONE, OM, HA, MANAGEMENT + + Connection timeout for HTTP call made by OM Snapshot Provider to request + OM snapshot from OM Leader. + + + + ozone.om.snapshot.provider.request.timeout + 5000ms + OZONE, OM, HA, MANAGEMENT + + Connection request timeout for HTTP call made by OM Snapshot Provider to + request OM snapshot from OM Leader. + + + ozone.acl.authorizer.class org.apache.hadoop.ozone.security.acl.OzoneAccessAuthorizer @@ -1764,7 +1792,7 @@ HTTP/_HOST@EXAMPLE.COM - hdds.scm.http.kerberos.keytab.file + hdds.scm.http.kerberos.keytab /etc/security/keytabs/HTTP.keytab @@ -1776,7 +1804,7 @@ - ozone.om.http.kerberos.keytab.file + ozone.om.http.kerberos.keytab /etc/security/keytabs/HTTP.keytab OzoneManager http server kerberos keytab. @@ -1809,7 +1837,7 @@ assumed. - + hdds.block.token.enabled false @@ -1828,39 +1856,12 @@ OZONE, HDDS, SECURITY, TLS HDDS GRPC server TLS provider. - - hdds.client.cert.chain.file.name - client.crt - OZONE, HDDS, SECURITY - Client certificate file name. It is an optional - field only required when mutual TLS (hdds.grpc.mutual.tls.required) - is set to true . - - - hdds.grpc.mutual.tls.required - false - OZONE, HDDS, SECURITY, TLS - If mutual tls check is enabled for GRPC. - Considered only if hdds.grpc.tls.enabled is set to true. - hdds.grpc.tls.enabled false OZONE, HDDS, SECURITY, TLS If HDDS GRPC server TLS is enabled. - - hdds.server.cert.chain.file.name - server.crt - OZONE, HDDS, SECURITY - Hdds server certificate file name. - - - hdds.trust.cert.collection.file.name - ca.crt - OZONE, HDDS, SECURITY - HDDS Certificate Authority trust store file name. - hdds.x509.default.duration P365D @@ -2291,8 +2292,10 @@ Directory where the Recon Server stores its metadata. This should be specified as a single directory. If the directory does not exist then the Recon will attempt to create it. + If undefined, then the Recon will log a warning and fallback to - ozone.metadata.dirs. + ozone.metadata.dirs. This fallback approach is not recommended for + production environments. @@ -2305,6 +2308,14 @@ awareness document for xml and yaml topology definition samples. + + ozone.network.topology.aware.read + false + OZONE, PERFORMANCE + + Whether to enable topology aware read to improve the read performance. + + ozone.recon.container.db.impl RocksDB @@ -2322,8 +2333,10 @@ Directory where the Recon Server stores its OM snapshot DB. This should be specified as a single directory. If the directory does not exist then the Recon will attempt to create it. + If undefined, then the Recon will log a warning and fallback to - ozone.metadata.dirs. + ozone.metadata.dirs. This fallback approach is not recommended for + production environments. @@ -2353,14 +2366,6 @@ OM snapshot. - - recon.om.socket.timeout - 5s - OZONE, RECON, OM - - Socket timeout for HTTP call made by Recon to request OM snapshot. - - recon.om.snapshot.task.initial.delay 1m @@ -2423,7 +2428,7 @@ OZONE, RECON - Ozone Recon datbase password. + Ozone Recon database password. @@ -2450,7 +2455,7 @@ The max active connections to the SQL database. The default SQLite database only allows single active connection, set this to a - resonable value like 10, for external production database. + reasonable value like 10, for external production database. @@ -2488,4 +2493,12 @@ connections. + + ozone.recon.task.thread.count + 1 + OZONE, RECON + + The number of Recon Tasks that are waiting on updates from OM. + + diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/TestHddsUtils.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/TestHddsUtils.java new file mode 100644 index 0000000000000..75636106498eb --- /dev/null +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/TestHddsUtils.java @@ -0,0 +1,42 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

    + * http://www.apache.org/licenses/LICENSE-2.0 + *

    + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds; + +import java.util.Optional; + +import org.junit.Assert; +import org.junit.Test; + +/** + * Testing HddsUtils. + */ +public class TestHddsUtils { + + @Test + public void testGetHostName() { + Assert.assertEquals(Optional.of("localhost"), + HddsUtils.getHostName("localhost:1234")); + + Assert.assertEquals(Optional.of("localhost"), + HddsUtils.getHostName("localhost")); + + Assert.assertEquals(Optional.empty(), + HddsUtils.getHostName(":1234")); + } + +} \ No newline at end of file diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/ratis/TestContainerCommandRequestMessage.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/ratis/TestContainerCommandRequestMessage.java new file mode 100644 index 0000000000000..bbe6ab7cca7a3 --- /dev/null +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/ratis/TestContainerCommandRequestMessage.java @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.ratis; + +import org.apache.hadoop.hdds.client.BlockID; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.BlockData; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ChunkInfo; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandRequestProto; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.KeyValue; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.PutBlockRequestProto; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.PutSmallFileRequestProto; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.Type; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.WriteChunkRequestProto; +import org.apache.hadoop.ozone.common.Checksum; +import org.apache.hadoop.ozone.common.ChecksumData; +import org.apache.hadoop.ozone.common.OzoneChecksumException; +import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; +import org.junit.Assert; +import org.junit.Test; + +import java.util.Random; +import java.util.UUID; +import java.util.function.BiFunction; + +/** Testing {@link ContainerCommandRequestMessage}. */ +public class TestContainerCommandRequestMessage { + static final Random RANDOM = new Random(); + + static ByteString newData(int length, Random random) { + final ByteString.Output out = ByteString.newOutput(); + for(int i = 0; i < length; i++) { + out.write(random.nextInt()); + } + return out.toByteString(); + } + + static ChecksumData checksum(ByteString data) { + try { + return new Checksum().computeChecksum(data.toByteArray()); + } catch (OzoneChecksumException e) { + throw new IllegalStateException(e); + } + } + + static ContainerCommandRequestProto newPutSmallFile( + BlockID blockID, ByteString data) { + final BlockData.Builder blockData + = BlockData.newBuilder() + .setBlockID(blockID.getDatanodeBlockIDProtobuf()); + final PutBlockRequestProto.Builder putBlockRequest + = PutBlockRequestProto.newBuilder() + .setBlockData(blockData); + final KeyValue keyValue = KeyValue.newBuilder() + .setKey("OverWriteRequested") + .setValue("true") + .build(); + final ChunkInfo chunk = ChunkInfo.newBuilder() + .setChunkName(blockID.getLocalID() + "_chunk") + .setOffset(0) + .setLen(data.size()) + .addMetadata(keyValue) + .setChecksumData(checksum(data).getProtoBufMessage()) + .build(); + final PutSmallFileRequestProto putSmallFileRequest + = PutSmallFileRequestProto.newBuilder() + .setChunkInfo(chunk) + .setBlock(putBlockRequest) + .setData(data) + .build(); + return ContainerCommandRequestProto.newBuilder() + .setCmdType(Type.PutSmallFile) + .setContainerID(blockID.getContainerID()) + .setDatanodeUuid(UUID.randomUUID().toString()) + .setPutSmallFile(putSmallFileRequest) + .build(); + } + + static ContainerCommandRequestProto newWriteChunk( + BlockID blockID, ByteString data) { + final ChunkInfo chunk = ChunkInfo.newBuilder() + .setChunkName(blockID.getLocalID() + "_chunk_" + 1) + .setOffset(0) + .setLen(data.size()) + .setChecksumData(checksum(data).getProtoBufMessage()) + .build(); + + final WriteChunkRequestProto.Builder writeChunkRequest + = WriteChunkRequestProto.newBuilder() + .setBlockID(blockID.getDatanodeBlockIDProtobuf()) + .setChunkData(chunk) + .setData(data); + return ContainerCommandRequestProto.newBuilder() + .setCmdType(Type.WriteChunk) + .setContainerID(blockID.getContainerID()) + .setDatanodeUuid(UUID.randomUUID().toString()) + .setWriteChunk(writeChunkRequest) + .build(); + } + + @Test + public void testPutSmallFile() throws Exception { + runTest(TestContainerCommandRequestMessage::newPutSmallFile); + } + + @Test + public void testWriteChunk() throws Exception { + runTest(TestContainerCommandRequestMessage::newWriteChunk); + } + + static void runTest( + BiFunction method) + throws Exception { + for(int i = 0; i < 2; i++) { + runTest(i, method); + } + for(int i = 2; i < 1 << 10;) { + runTest(i + 1 + RANDOM.nextInt(i - 1), method); + i <<= 1; + runTest(i, method); + } + } + + static void runTest(int length, + BiFunction method) + throws Exception { + System.out.println("length=" + length); + final BlockID blockID = new BlockID(RANDOM.nextLong(), RANDOM.nextLong()); + final ByteString data = newData(length, RANDOM); + + final ContainerCommandRequestProto original = method.apply(blockID, data); + final ContainerCommandRequestMessage message + = ContainerCommandRequestMessage.toMessage(original, null); + final ContainerCommandRequestProto computed + = ContainerCommandRequestMessage.toProto(message.getContent(), null); + Assert.assertEquals(original, computed); + } +} diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/exceptions/TestSCMExceptionResultCodes.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/exceptions/TestSCMExceptionResultCodes.java new file mode 100644 index 0000000000000..b5b4684dda0ca --- /dev/null +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/exceptions/TestSCMExceptionResultCodes.java @@ -0,0 +1,52 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.scm.exceptions; + +import org.apache.hadoop.hdds.scm.exceptions.SCMException.ResultCodes; +import org.apache.hadoop.hdds.protocol.proto. + ScmBlockLocationProtocolProtos.Status; +import org.junit.Assert; +import org.junit.Test; + +/** + * Test Result code mappping between SCMException and the protobuf definitions. + */ +public class TestSCMExceptionResultCodes { + + @Test + public void codeMapping() { + // ResultCode = SCMException definition + // Status = protobuf definition + Assert.assertEquals(ResultCodes.values().length, Status.values().length); + for (int i = 0; i < ResultCodes.values().length; i++) { + ResultCodes codeValue = ResultCodes.values()[i]; + Status protoBufValue = Status.values()[i]; + Assert.assertTrue(String + .format("Protobuf/Enum constant name mismatch %s %s", codeValue, + protoBufValue), sameName(codeValue.name(), protoBufValue.name())); + ResultCodes converted = ResultCodes.values()[protoBufValue.ordinal()]; + Assert.assertEquals(codeValue, converted); + } + } + + private boolean sameName(String codeValue, String protoBufValue) { + return codeValue.equals(protoBufValue); + } + +} + diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNetworkTopologyImpl.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNetworkTopologyImpl.java index 0edfb07fba2bf..b31e4a8e9965c 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNetworkTopologyImpl.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNetworkTopologyImpl.java @@ -357,9 +357,11 @@ public void testChooseRandomSimple() { // test chooseRandom(String scope, String excludedScope) path = dataNodes[random.nextInt(dataNodes.length)].getNetworkFullPath(); - assertNull(cluster.chooseRandom(path, path)); - assertNotNull(cluster.chooseRandom(null, path)); - assertNotNull(cluster.chooseRandom("", path)); + List pathList = new ArrayList<>(); + pathList.add(path); + assertNull(cluster.chooseRandom(path, pathList)); + assertNotNull(cluster.chooseRandom(null, pathList)); + assertNotNull(cluster.chooseRandom("", pathList)); // test chooseRandom(String scope, Collection excludedNodes) assertNull(cluster.chooseRandom("", Arrays.asList(dataNodes))); @@ -399,7 +401,9 @@ public void testChooseRandomExcludedScope() { } // "" excludedScope, no node will ever be chosen - frequency = pickNodes(100, "", null, null, 0); + List pathList = new ArrayList(); + pathList.add(""); + frequency = pickNodes(100, pathList, null, null, 0); for (Node key : dataNodes) { assertTrue(frequency.get(key) == 0); } @@ -411,8 +415,10 @@ public void testChooseRandomExcludedScope() { assertTrue(frequency.get(key) == 0); } // out network topology excluded scope, every node should be chosen - scope = "/city1"; - frequency = pickNodes(cluster.getNumOfLeafNode(null), scope, null, null, 0); + pathList.clear(); + pathList.add("/city1"); + frequency = pickNodes( + cluster.getNumOfLeafNode(null), pathList, null, null, 0); for (Node key : dataNodes) { assertTrue(frequency.get(key) != 0); } @@ -582,19 +588,32 @@ public void testChooseRandomWithAffinityNode() { }}; int[] affinityNodeIndexs = {0, dataNodes.length - 1, random.nextInt(dataNodes.length), random.nextInt(dataNodes.length)}; + Node[][] excludedScopeIndexs = {{dataNodes[0]}, + {dataNodes[dataNodes.length - 1]}, + {dataNodes[random.nextInt(dataNodes.length)]}, + {dataNodes[random.nextInt(dataNodes.length)], + dataNodes[random.nextInt(dataNodes.length)] + }, + {dataNodes[random.nextInt(dataNodes.length)], + dataNodes[random.nextInt(dataNodes.length)], + dataNodes[random.nextInt(dataNodes.length)], + }}; int leafNum = cluster.getNumOfLeafNode(null); Map frequency; - String scope; + List pathList = new ArrayList<>(); for (int k : affinityNodeIndexs) { - for (int i : excludedNodeIndexs) { - String path = dataNodes[i].getNetworkFullPath(); - while (!path.equals(ROOT)) { + for (Node[] excludedScopes : excludedScopeIndexs) { + pathList.clear(); + pathList.addAll(Arrays.stream(excludedScopes) + .map(node -> node.getNetworkFullPath()) + .collect(Collectors.toList())); + while (!pathList.get(0).equals(ROOT)) { int ancestorGen = cluster.getMaxLevel() - 1; while (ancestorGen > 0) { for (Node[] list : excludedNodeLists) { List excludedList = Arrays.asList(list); - frequency = pickNodes(leafNum, path, excludedList, dataNodes[k], - ancestorGen); + frequency = pickNodes(leafNum, pathList, excludedList, + dataNodes[k], ancestorGen); Node affinityAncestor = dataNodes[k].getAncestor(ancestorGen); for (Node key : dataNodes) { if (affinityAncestor != null) { @@ -605,28 +624,33 @@ public void testChooseRandomWithAffinityNode() { } else if (excludedList != null && excludedList.contains(key)) { continue; - } else if (path != null && - key.getNetworkFullPath().startsWith(path)) { + } else if (pathList != null && + pathList.stream().anyMatch(path -> + key.getNetworkFullPath().startsWith(path))) { continue; } else { fail("Node is not picked when sequentially going " + "through ancestor node's leaf nodes. node:" + key.getNetworkFullPath() + ", ancestor node:" + affinityAncestor.getNetworkFullPath() + - ", excludedScope: " + path + ", " + "excludedList:" + - (excludedList == null ? "" : excludedList.toString())); + ", excludedScope: " + pathList.toString() + ", " + + "excludedList:" + (excludedList == null ? "" : + excludedList.toString())); } } } } ancestorGen--; } - path = path.substring(0, path.lastIndexOf(PATH_SEPARATOR_STR)); + pathList = pathList.stream().map(path -> + path.substring(0, path.lastIndexOf(PATH_SEPARATOR_STR))) + .collect(Collectors.toList()); } } } // all nodes excluded, no node will be picked + String scope; List excludedList = Arrays.asList(dataNodes); for (int k : affinityNodeIndexs) { for (int i : excludedNodeIndexs) { @@ -784,15 +808,16 @@ public void testSortByDistanceCost() { for (Node[] nodeList : nodes) { int length = nodeList.length; while (length > 0) { - cluster.sortByDistanceCost(reader, nodeList, length); - for (int i = 0; i < nodeList.length; i++) { - if ((i + 1) < nodeList.length) { - int cost1 = cluster.getDistanceCost(reader, nodeList[i]); - int cost2 = cluster.getDistanceCost(reader, nodeList[i + 1]); + List ret = cluster.sortByDistanceCost(reader, + Arrays.asList(nodeList), length); + for (int i = 0; i < ret.size(); i++) { + if ((i + 1) < ret.size()) { + int cost1 = cluster.getDistanceCost(reader, ret.get(i)); + int cost2 = cluster.getDistanceCost(reader, ret.get(i + 1)); assertTrue("reader:" + (reader != null ? reader.getNetworkFullPath() : "null") + - ",node1:" + nodeList[i].getNetworkFullPath() + - ",node2:" + nodeList[i + 1].getNetworkFullPath() + + ",node1:" + ret.get(i).getNetworkFullPath() + + ",node2:" + ret.get(i + 1).getNetworkFullPath() + ",cost1:" + cost1 + ",cost2:" + cost2, cost1 == Integer.MAX_VALUE || cost1 <= cost2); } @@ -803,20 +828,22 @@ public void testSortByDistanceCost() { } // sort all nodes - Node[] nodeList = dataNodes.clone(); + List nodeList = Arrays.asList(dataNodes.clone()); for (Node reader : readers) { - int length = nodeList.length; + int length = nodeList.size(); while (length >= 0) { - cluster.sortByDistanceCost(reader, nodeList, length); - for (int i = 0; i < nodeList.length; i++) { - if ((i + 1) < nodeList.length) { - int cost1 = cluster.getDistanceCost(reader, nodeList[i]); - int cost2 = cluster.getDistanceCost(reader, nodeList[i + 1]); + List sortedNodeList = + cluster.sortByDistanceCost(reader, nodeList, length); + for (int i = 0; i < sortedNodeList.size(); i++) { + if ((i + 1) < sortedNodeList.size()) { + int cost1 = cluster.getDistanceCost(reader, sortedNodeList.get(i)); + int cost2 = cluster.getDistanceCost( + reader, sortedNodeList.get(i + 1)); // node can be removed when called in testConcurrentAccess assertTrue("reader:" + (reader != null ? reader.getNetworkFullPath() : "null") + - ",node1:" + nodeList[i].getNetworkFullPath() + - ",node2:" + nodeList[i + 1].getNetworkFullPath() + + ",node1:" + sortedNodeList.get(i).getNetworkFullPath() + + ",node2:" + sortedNodeList.get(i + 1).getNetworkFullPath() + ",cost1:" + cost1 + ",cost2:" + cost2, cost1 == Integer.MAX_VALUE || cost1 <= cost2); } @@ -877,9 +904,12 @@ private Map pickNodesAtRandom(int numNodes, frequency.put(dnd, 0); } + List pathList = new ArrayList<>(); + pathList.add(excludedScope.substring(1)); for (int j = 0; j < numNodes; j++) { - Node node = cluster.chooseRandom("", excludedScope.substring(1), - excludedNodes, affinityNode, ancestorGen); + + Node node = cluster.chooseRandom("", pathList, excludedNodes, + affinityNode, ancestorGen); if (node != null) { frequency.put(node, frequency.get(node) + 1); } @@ -892,7 +922,7 @@ private Map pickNodesAtRandom(int numNodes, * This picks a large amount of nodes sequentially. * * @param numNodes the number of nodes - * @param excludedScope the excluded scope, should not start with "~" + * @param excludedScopes the excluded scopes, should not start with "~" * @param excludedNodes the excluded node list * @param affinityNode the chosen node should share the same ancestor at * generation "ancestorGen" with this node @@ -900,8 +930,9 @@ private Map pickNodesAtRandom(int numNodes, * this generation with excludedNodes * @return the frequency that nodes were chosen */ - private Map pickNodes(int numNodes, String excludedScope, - Collection excludedNodes, Node affinityNode, int ancestorGen) { + private Map pickNodes(int numNodes, + List excludedScopes, Collection excludedNodes, + Node affinityNode, int ancestorGen) { Map frequency = new HashMap<>(); for (Node dnd : dataNodes) { frequency.put(dnd, 0); @@ -909,7 +940,7 @@ private Map pickNodes(int numNodes, String excludedScope, excludedNodes = excludedNodes == null ? null : excludedNodes.stream().distinct().collect(Collectors.toList()); for (int j = 0; j < numNodes; j++) { - Node node = cluster.getNode(j, null, excludedScope, excludedNodes, + Node node = cluster.getNode(j, null, excludedScopes, excludedNodes, affinityNode, ancestorGen); if (node != null) { frequency.put(node, frequency.get(node) + 1); diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNodeSchemaLoader.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNodeSchemaLoader.java index 30799b1099b37..0c20353a2ce0c 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNodeSchemaLoader.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNodeSchemaLoader.java @@ -44,7 +44,7 @@ public TestNodeSchemaLoader(String schemaFile, String errMsg) { try { String filePath = classLoader.getResource( "./networkTopologyTestFiles/" + schemaFile).getPath(); - NodeSchemaLoader.getInstance().loadSchemaFromXml(filePath); + NodeSchemaLoader.getInstance().loadSchemaFromFile(filePath); fail("expect exceptions"); } catch (Throwable e) { assertTrue(e.getMessage().contains(errMsg)); @@ -83,7 +83,7 @@ public void testGood() { try { String filePath = classLoader.getResource( "./networkTopologyTestFiles/good.xml").getPath(); - NodeSchemaLoader.getInstance().loadSchemaFromXml(filePath); + NodeSchemaLoader.getInstance().loadSchemaFromFile(filePath); } catch (Throwable e) { fail("should succeed"); } @@ -94,10 +94,10 @@ public void testNotExist() { String filePath = classLoader.getResource( "./networkTopologyTestFiles/good.xml").getPath() + ".backup"; try { - NodeSchemaLoader.getInstance().loadSchemaFromXml(filePath); + NodeSchemaLoader.getInstance().loadSchemaFromFile(filePath); fail("should fail"); } catch (Throwable e) { - assertTrue(e.getMessage().contains("file " + filePath + " is not found")); + assertTrue(e.getMessage().contains("not found")); } } } diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNodeSchemaManager.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNodeSchemaManager.java index 7e304190d6d1a..6698043727649 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNodeSchemaManager.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNodeSchemaManager.java @@ -79,7 +79,7 @@ public void testInitFailure() { manager.init(conf); fail("should fail"); } catch (Throwable e) { - assertTrue(e.getMessage().contains("Fail to load schema file:" + + assertTrue(e.getMessage().contains("Failed to load schema file:" + filePath)); } } diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestYamlSchemaLoader.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestYamlSchemaLoader.java index 580a7fb485e80..c38bf388363cf 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestYamlSchemaLoader.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestYamlSchemaLoader.java @@ -44,7 +44,7 @@ public TestYamlSchemaLoader(String schemaFile, String errMsg) { try { String filePath = classLoader.getResource( "./networkTopologyTestFiles/" + schemaFile).getPath(); - NodeSchemaLoader.getInstance().loadSchemaFromYaml(filePath); + NodeSchemaLoader.getInstance().loadSchemaFromFile(filePath); fail("expect exceptions"); } catch (Throwable e) { assertTrue(e.getMessage().contains(errMsg)); @@ -69,7 +69,7 @@ public void testGood() { try { String filePath = classLoader.getResource( "./networkTopologyTestFiles/good.yaml").getPath(); - NodeSchemaLoader.getInstance().loadSchemaFromYaml(filePath); + NodeSchemaLoader.getInstance().loadSchemaFromFile(filePath); } catch (Throwable e) { fail("should succeed"); } @@ -78,12 +78,12 @@ public void testGood() { @Test public void testNotExist() { String filePath = classLoader.getResource( - "./networkTopologyTestFiles/good.xml").getPath() + ".backup"; + "./networkTopologyTestFiles/good.yaml").getPath() + ".backup"; try { - NodeSchemaLoader.getInstance().loadSchemaFromXml(filePath); + NodeSchemaLoader.getInstance().loadSchemaFromFile(filePath); fail("should fail"); } catch (Throwable e) { - assertTrue(e.getMessage().contains("file " + filePath + " is not found")); + assertTrue(e.getMessage().contains("not found")); } } diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/client/TestCertificateClientInit.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/client/TestCertificateClientInit.java index 61bcf2155491d..dcd9898cbeee0 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/client/TestCertificateClientInit.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/client/TestCertificateClientInit.java @@ -66,8 +66,11 @@ public class TestCertificateClientInit { private HDDSKeyGenerator keyGenerator; private Path metaDirPath; private SecurityConfig securityConfig; - private KeyCodec keyCodec; + private KeyCodec dnKeyCodec; + private KeyCodec omKeyCodec; private X509Certificate x509Certificate; + private final static String DN_COMPONENT = DNCertificateClient.COMPONENT_NAME; + private final static String OM_COMPONENT = OMCertificateClient.COMPONENT_NAME; @Parameter public boolean pvtKeyPresent; @@ -107,9 +110,11 @@ public void setUp() throws Exception { certSerialId); omCertificateClient = new OMCertificateClient(securityConfig, certSerialId); - keyCodec = new KeyCodec(securityConfig); + dnKeyCodec = new KeyCodec(securityConfig, DN_COMPONENT); + omKeyCodec = new KeyCodec(securityConfig, OM_COMPONENT); - Files.createDirectories(securityConfig.getKeyLocation()); + Files.createDirectories(securityConfig.getKeyLocation(DN_COMPONENT)); + Files.createDirectories(securityConfig.getKeyLocation(OM_COMPONENT)); } @After @@ -123,28 +128,32 @@ public void tearDown() { @Test public void testInitDatanode() throws Exception { if (pvtKeyPresent) { - keyCodec.writePrivateKey(keyPair.getPrivate()); + dnKeyCodec.writePrivateKey(keyPair.getPrivate()); } else { - FileUtils.deleteQuietly(Paths.get(securityConfig.getKeyLocation() - .toString(), securityConfig.getPrivateKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + securityConfig.getKeyLocation(DN_COMPONENT).toString(), + securityConfig.getPrivateKeyFileName()).toFile()); } if (pubKeyPresent) { if (dnCertificateClient.getPublicKey() == null) { - keyCodec.writePublicKey(keyPair.getPublic()); + dnKeyCodec.writePublicKey(keyPair.getPublic()); } } else { - FileUtils.deleteQuietly(Paths.get(securityConfig.getKeyLocation() - .toString(), securityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly( + Paths.get(securityConfig.getKeyLocation(DN_COMPONENT).toString(), + securityConfig.getPublicKeyFileName()).toFile()); } if (certPresent) { - CertificateCodec codec = new CertificateCodec(securityConfig); + CertificateCodec codec = new CertificateCodec(securityConfig, + DN_COMPONENT); codec.writeCertificate(new X509CertificateHolder( x509Certificate.getEncoded())); } else { - FileUtils.deleteQuietly(Paths.get(securityConfig.getKeyLocation() - .toString(), securityConfig.getCertificateFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + securityConfig.getKeyLocation(DN_COMPONENT).toString(), + securityConfig.getCertificateFileName()).toFile()); } InitResponse response = dnCertificateClient.init(); @@ -152,10 +161,10 @@ public void testInitDatanode() throws Exception { if (!response.equals(FAILURE)) { assertTrue(OzoneSecurityUtil.checkIfFileExist( - securityConfig.getKeyLocation(), + securityConfig.getKeyLocation(DN_COMPONENT), securityConfig.getPrivateKeyFileName())); assertTrue(OzoneSecurityUtil.checkIfFileExist( - securityConfig.getKeyLocation(), + securityConfig.getKeyLocation(DN_COMPONENT), securityConfig.getPublicKeyFileName())); } } @@ -163,28 +172,32 @@ public void testInitDatanode() throws Exception { @Test public void testInitOzoneManager() throws Exception { if (pvtKeyPresent) { - keyCodec.writePrivateKey(keyPair.getPrivate()); + omKeyCodec.writePrivateKey(keyPair.getPrivate()); } else { - FileUtils.deleteQuietly(Paths.get(securityConfig.getKeyLocation() - .toString(), securityConfig.getPrivateKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + securityConfig.getKeyLocation(OM_COMPONENT).toString(), + securityConfig.getPrivateKeyFileName()).toFile()); } if (pubKeyPresent) { if (omCertificateClient.getPublicKey() == null) { - keyCodec.writePublicKey(keyPair.getPublic()); + omKeyCodec.writePublicKey(keyPair.getPublic()); } } else { - FileUtils.deleteQuietly(Paths.get(securityConfig.getKeyLocation() - .toString(), securityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + securityConfig.getKeyLocation(OM_COMPONENT).toString(), + securityConfig.getPublicKeyFileName()).toFile()); } if (certPresent) { - CertificateCodec codec = new CertificateCodec(securityConfig); + CertificateCodec codec = new CertificateCodec(securityConfig, + OM_COMPONENT); codec.writeCertificate(new X509CertificateHolder( x509Certificate.getEncoded())); } else { - FileUtils.deleteQuietly(Paths.get(securityConfig.getKeyLocation() - .toString(), securityConfig.getCertificateFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + securityConfig.getKeyLocation(OM_COMPONENT).toString(), + securityConfig.getCertificateFileName()).toFile()); } InitResponse response = omCertificateClient.init(); @@ -196,10 +209,10 @@ public void testInitOzoneManager() throws Exception { if (!response.equals(FAILURE)) { assertTrue(OzoneSecurityUtil.checkIfFileExist( - securityConfig.getKeyLocation(), + securityConfig.getKeyLocation(OM_COMPONENT), securityConfig.getPrivateKeyFileName())); assertTrue(OzoneSecurityUtil.checkIfFileExist( - securityConfig.getKeyLocation(), + securityConfig.getKeyLocation(OM_COMPONENT), securityConfig.getPublicKeyFileName())); } } diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/client/TestDefaultCertificateClient.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/client/TestDefaultCertificateClient.java index 11be0ded560e8..f389cdb6d22ea 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/client/TestDefaultCertificateClient.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/client/TestDefaultCertificateClient.java @@ -76,6 +76,8 @@ public class TestDefaultCertificateClient { private SecurityConfig omSecurityConfig; private SecurityConfig dnSecurityConfig; private final static String UTF = "UTF-8"; + private final static String DN_COMPONENT = DNCertificateClient.COMPONENT_NAME; + private final static String OM_COMPONENT = OMCertificateClient.COMPONENT_NAME; private KeyCodec omKeyCodec; private KeyCodec dnKeyCodec; @@ -99,11 +101,11 @@ public void setUp() throws Exception { keyGenerator = new HDDSKeyGenerator(omSecurityConfig); - omKeyCodec = new KeyCodec(omSecurityConfig); - dnKeyCodec = new KeyCodec(dnSecurityConfig); + omKeyCodec = new KeyCodec(omSecurityConfig, OM_COMPONENT); + dnKeyCodec = new KeyCodec(dnSecurityConfig, DN_COMPONENT); - Files.createDirectories(omSecurityConfig.getKeyLocation()); - Files.createDirectories(dnSecurityConfig.getKeyLocation()); + Files.createDirectories(omSecurityConfig.getKeyLocation(OM_COMPONENT)); + Files.createDirectories(dnSecurityConfig.getKeyLocation(DN_COMPONENT)); x509Certificate = generateX509Cert(null); certSerialId = x509Certificate.getSerialNumber().toString(); getCertClient(); @@ -156,14 +158,18 @@ private KeyPair generateKeyPairFiles() throws Exception { } private void cleanupOldKeyPair() { - FileUtils.deleteQuietly(Paths.get(omSecurityConfig.getKeyLocation() - .toString(), omSecurityConfig.getPrivateKeyFileName()).toFile()); - FileUtils.deleteQuietly(Paths.get(omSecurityConfig.getKeyLocation() - .toString(), omSecurityConfig.getPublicKeyFileName()).toFile()); - FileUtils.deleteQuietly(Paths.get(dnSecurityConfig.getKeyLocation() - .toString(), dnSecurityConfig.getPrivateKeyFileName()).toFile()); - FileUtils.deleteQuietly(Paths.get(dnSecurityConfig.getKeyLocation() - .toString(), dnSecurityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + omSecurityConfig.getKeyLocation(OM_COMPONENT).toString(), + omSecurityConfig.getPrivateKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + omSecurityConfig.getKeyLocation(OM_COMPONENT).toString(), + omSecurityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + dnSecurityConfig.getKeyLocation(DN_COMPONENT).toString(), + dnSecurityConfig.getPrivateKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + dnSecurityConfig.getKeyLocation(DN_COMPONENT).toString(), + dnSecurityConfig.getPublicKeyFileName()).toFile()); } /** @@ -196,10 +202,12 @@ private X509Certificate generateX509Cert(KeyPair keyPair) throws Exception { @Test public void testSignDataStream() throws Exception { String data = RandomStringUtils.random(100, UTF); - FileUtils.deleteQuietly(Paths.get(omSecurityConfig.getKeyLocation() - .toString(), omSecurityConfig.getPrivateKeyFileName()).toFile()); - FileUtils.deleteQuietly(Paths.get(omSecurityConfig.getKeyLocation() - .toString(), omSecurityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + omSecurityConfig.getKeyLocation(OM_COMPONENT).toString(), + omSecurityConfig.getPrivateKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + omSecurityConfig.getKeyLocation(OM_COMPONENT).toString(), + omSecurityConfig.getPublicKeyFileName()).toFile()); // Expect error when there is no private key to sign. LambdaTestUtils.intercept(IOException.class, "Error while " + @@ -285,8 +293,9 @@ public void testCertificateLoadingOnInit() throws Exception { X509Certificate cert2 = generateX509Cert(keyPair); X509Certificate cert3 = generateX509Cert(keyPair); - Path certPath = dnSecurityConfig.getCertificateLocation(); - CertificateCodec codec = new CertificateCodec(dnSecurityConfig); + Path certPath = dnSecurityConfig.getCertificateLocation(DN_COMPONENT); + CertificateCodec codec = new CertificateCodec(dnSecurityConfig, + DN_COMPONENT); // Certificate not found. LambdaTestUtils.intercept(CertificateException.class, "Error while" + @@ -308,7 +317,7 @@ public void testCertificateLoadingOnInit() throws Exception { codec.writeCertificate(certPath, "3.crt", getPEMEncodedString(cert3), true); - // Re instentiate DN client which will load certificates from filesystem. + // Re instantiate DN client which will load certificates from filesystem. dnCertClient = new DNCertificateClient(dnSecurityConfig, certSerialId); assertNotNull(dnCertClient.getCertificate(cert1.getSerialNumber() @@ -352,16 +361,20 @@ public void testInitCertAndKeypairValidationFailures() throws Exception { omClientLog.clearOutput(); // Case 1. Expect failure when keypair validation fails. - FileUtils.deleteQuietly(Paths.get(omSecurityConfig.getKeyLocation() - .toString(), omSecurityConfig.getPrivateKeyFileName()).toFile()); - FileUtils.deleteQuietly(Paths.get(omSecurityConfig.getKeyLocation() - .toString(), omSecurityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + omSecurityConfig.getKeyLocation(OM_COMPONENT).toString(), + omSecurityConfig.getPrivateKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + omSecurityConfig.getKeyLocation(OM_COMPONENT).toString(), + omSecurityConfig.getPublicKeyFileName()).toFile()); - FileUtils.deleteQuietly(Paths.get(dnSecurityConfig.getKeyLocation() - .toString(), dnSecurityConfig.getPrivateKeyFileName()).toFile()); - FileUtils.deleteQuietly(Paths.get(dnSecurityConfig.getKeyLocation() - .toString(), dnSecurityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + dnSecurityConfig.getKeyLocation(DN_COMPONENT).toString(), + dnSecurityConfig.getPrivateKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + dnSecurityConfig.getKeyLocation(DN_COMPONENT).toString(), + dnSecurityConfig.getPublicKeyFileName()).toFile()); omKeyCodec.writePrivateKey(keyPair.getPrivate()); omKeyCodec.writePublicKey(keyPair2.getPublic()); @@ -387,16 +400,20 @@ public void testInitCertAndKeypairValidationFailures() throws Exception { // Case 2. Expect failure when certificate is generated from different // private key and keypair validation fails. getCertClient(); - FileUtils.deleteQuietly(Paths.get(omSecurityConfig.getKeyLocation() - .toString(), omSecurityConfig.getCertificateFileName()).toFile()); - FileUtils.deleteQuietly(Paths.get(dnSecurityConfig.getKeyLocation() - .toString(), dnSecurityConfig.getCertificateFileName()).toFile()); - - CertificateCodec omCertCodec = new CertificateCodec(omSecurityConfig); + FileUtils.deleteQuietly(Paths.get( + omSecurityConfig.getKeyLocation(OM_COMPONENT).toString(), + omSecurityConfig.getCertificateFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + dnSecurityConfig.getKeyLocation(DN_COMPONENT).toString(), + dnSecurityConfig.getCertificateFileName()).toFile()); + + CertificateCodec omCertCodec = new CertificateCodec(omSecurityConfig, + OM_COMPONENT); omCertCodec.writeCertificate(new X509CertificateHolder( x509Certificate.getEncoded())); - CertificateCodec dnCertCodec = new CertificateCodec(dnSecurityConfig); + CertificateCodec dnCertCodec = new CertificateCodec(dnSecurityConfig, + DN_COMPONENT); dnCertCodec.writeCertificate(new X509CertificateHolder( x509Certificate.getEncoded())); // Check for DN. @@ -416,10 +433,12 @@ public void testInitCertAndKeypairValidationFailures() throws Exception { // private key and certificate validation fails. // Re write the correct public key. - FileUtils.deleteQuietly(Paths.get(omSecurityConfig.getKeyLocation() - .toString(), omSecurityConfig.getPublicKeyFileName()).toFile()); - FileUtils.deleteQuietly(Paths.get(dnSecurityConfig.getKeyLocation() - .toString(), dnSecurityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + omSecurityConfig.getKeyLocation(OM_COMPONENT).toString(), + omSecurityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + dnSecurityConfig.getKeyLocation(DN_COMPONENT).toString(), + dnSecurityConfig.getPublicKeyFileName()).toFile()); getCertClient(); omKeyCodec.writePublicKey(keyPair.getPublic()); dnKeyCodec.writePublicKey(keyPair.getPublic()); @@ -440,10 +459,12 @@ public void testInitCertAndKeypairValidationFailures() throws Exception { // Case 4. Failure when public key recovery fails. getCertClient(); - FileUtils.deleteQuietly(Paths.get(omSecurityConfig.getKeyLocation() - .toString(), omSecurityConfig.getPublicKeyFileName()).toFile()); - FileUtils.deleteQuietly(Paths.get(dnSecurityConfig.getKeyLocation() - .toString(), dnSecurityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + omSecurityConfig.getKeyLocation(OM_COMPONENT).toString(), + omSecurityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + dnSecurityConfig.getKeyLocation(DN_COMPONENT).toString(), + dnSecurityConfig.getPublicKeyFileName()).toFile()); // Check for DN. assertEquals(dnCertClient.init(), FAILURE); diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/utils/TestCertificateCodec.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/utils/TestCertificateCodec.java index 9ac956ffe82b8..ded52068395ff 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/utils/TestCertificateCodec.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/utils/TestCertificateCodec.java @@ -22,6 +22,7 @@ import org.apache.commons.lang3.RandomStringUtils; import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdds.security.exception.SCMSecurityException; +import org.apache.hadoop.hdds.security.x509.SecurityConfig; import org.apache.hadoop.hdds.security.x509.certificates.utils.SelfSignedCertificate; import org.apache.hadoop.hdds.security.x509.keys.HDDSKeyGenerator; import org.bouncycastle.cert.X509CertificateHolder; @@ -50,12 +51,15 @@ */ public class TestCertificateCodec { private static OzoneConfiguration conf = new OzoneConfiguration(); + private static final String COMPONENT = "test"; + private SecurityConfig securityConfig; @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder(); @Before public void init() throws IOException { conf.set(OZONE_METADATA_DIRS, temporaryFolder.newFolder().toString()); + securityConfig = new SecurityConfig(conf); } /** @@ -88,7 +92,7 @@ public void testGetPEMEncodedString() .setKey(keyGenerator.generateKey()) .makeCA() .build(); - CertificateCodec codec = new CertificateCodec(conf); + CertificateCodec codec = new CertificateCodec(securityConfig, COMPONENT); String pemString = codec.getPEMEncodedString(cert); assertTrue(pemString.startsWith(CertificateCodec.BEGIN_CERT)); assertTrue(pemString.endsWith(CertificateCodec.END_CERT + "\n")); @@ -131,7 +135,7 @@ public void testwriteCertificate() throws NoSuchProviderException, .setKey(keyGenerator.generateKey()) .makeCA() .build(); - CertificateCodec codec = new CertificateCodec(conf); + CertificateCodec codec = new CertificateCodec(securityConfig, COMPONENT); String pemString = codec.getPEMEncodedString(cert); File basePath = temporaryFolder.newFolder(); if (!basePath.exists()) { @@ -172,7 +176,7 @@ public void testwriteCertificateDefault() .setKey(keyGenerator.generateKey()) .makeCA() .build(); - CertificateCodec codec = new CertificateCodec(conf); + CertificateCodec codec = new CertificateCodec(securityConfig, COMPONENT); codec.writeCertificate(cert); X509CertificateHolder certHolder = codec.readCertificate(); assertNotNull(certHolder); diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestKeyCodec.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestKeyCodec.java index d3e13d2383c61..d82b02f43c5dc 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestKeyCodec.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestKeyCodec.java @@ -57,6 +57,8 @@ public class TestKeyCodec { @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder(); private OzoneConfiguration configuration; + private SecurityConfig securityConfig; + private String component; private HDDSKeyGenerator keyGenerator; private String prefix; @@ -66,6 +68,8 @@ public void init() throws IOException { prefix = temporaryFolder.newFolder().toString(); configuration.set(HDDS_METADATA_DIR_NAME, prefix); keyGenerator = new HDDSKeyGenerator(configuration); + securityConfig = new SecurityConfig(configuration); + component = "test_component"; } /** @@ -83,11 +87,11 @@ public void testWriteKey() throws NoSuchProviderException, NoSuchAlgorithmException, IOException, InvalidKeySpecException { KeyPair keys = keyGenerator.generateKey(); - KeyCodec pemWriter = new KeyCodec(configuration); + KeyCodec pemWriter = new KeyCodec(securityConfig, component); pemWriter.writeKey(keys); // Assert that locations have been created. - Path keyLocation = pemWriter.getSecurityConfig().getKeyLocation(); + Path keyLocation = pemWriter.getSecurityConfig().getKeyLocation(component); Assert.assertTrue(keyLocation.toFile().exists()); // Assert that locations are created in the locations that we specified @@ -172,7 +176,7 @@ public void testWriteKey() public void testReWriteKey() throws Exception { KeyPair kp = keyGenerator.generateKey(); - KeyCodec pemWriter = new KeyCodec(configuration); + KeyCodec pemWriter = new KeyCodec(securityConfig, component); SecurityConfig secConfig = pemWriter.getSecurityConfig(); pemWriter.writeKey(kp); @@ -181,13 +185,13 @@ public void testReWriteKey() .intercept(IOException.class, "Private Key file already exists.", () -> pemWriter.writeKey(kp)); FileUtils.deleteQuietly(Paths.get( - secConfig.getKeyLocation().toString() + "/" + secConfig + secConfig.getKeyLocation(component).toString() + "/" + secConfig .getPrivateKeyFileName()).toFile()); LambdaTestUtils .intercept(IOException.class, "Public Key file already exists.", () -> pemWriter.writeKey(kp)); FileUtils.deleteQuietly(Paths.get( - secConfig.getKeyLocation().toString() + "/" + secConfig + secConfig.getKeyLocation(component).toString() + "/" + secConfig .getPublicKeyFileName()).toFile()); // Should succeed now as both public and private key are deleted. @@ -206,7 +210,7 @@ public void testReWriteKey() public void testWriteKeyInNonPosixFS() throws Exception { KeyPair kp = keyGenerator.generateKey(); - KeyCodec pemWriter = new KeyCodec(configuration); + KeyCodec pemWriter = new KeyCodec(securityConfig, component); pemWriter.setIsPosixFileSystem(() -> false); // Assert key rewrite fails in non Posix file system. @@ -221,7 +225,7 @@ public void testReadWritePublicKeywithoutArgs() InvalidKeySpecException { KeyPair kp = keyGenerator.generateKey(); - KeyCodec keycodec = new KeyCodec(configuration); + KeyCodec keycodec = new KeyCodec(securityConfig, component); keycodec.writeKey(kp); PublicKey pubKey = keycodec.readPublicKey(); diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestHddsIdFactory.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestHddsIdFactory.java similarity index 98% rename from hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestHddsIdFactory.java rename to hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestHddsIdFactory.java index 35e1b3ed099ed..11d0fad55eeca 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestHddsIdFactory.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestHddsIdFactory.java @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import java.util.ArrayList; import java.util.List; diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestMetadataStore.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestMetadataStore.java similarity index 98% rename from hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestMetadataStore.java rename to hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestMetadataStore.java index 4b41ceb670af7..d24fcf5c3b8b6 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestMetadataStore.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestMetadataStore.java @@ -14,20 +14,21 @@ * License for the specific language governing permissions and limitations under * the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import com.google.common.collect.Lists; import org.apache.commons.io.FileUtils; import org.apache.commons.lang3.tuple.ImmutablePair; + import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdfs.DFSUtil; import org.apache.hadoop.hdfs.DFSUtilClient; import org.apache.hadoop.ozone.OzoneConfigKeys; import org.apache.hadoop.test.GenericTestUtils; -import org.apache.hadoop.utils.MetadataKeyFilters.KeyPrefixFilter; -import org.apache.hadoop.utils.MetadataKeyFilters.MetadataKeyFilter; -import org.apache.hadoop.utils.MetadataStore.KeyValue; +import org.apache.hadoop.hdds.utils.MetadataKeyFilters.KeyPrefixFilter; +import org.apache.hadoop.hdds.utils.MetadataKeyFilters.MetadataKeyFilter; + import org.junit.After; import org.junit.Before; import org.junit.Rule; @@ -69,13 +70,14 @@ public class TestMetadataStore { public ExpectedException expectedException = ExpectedException.none(); private MetadataStore store; private File testDir; + public TestMetadataStore(String metadataImpl) { this.storeImpl = metadataImpl; } @Parameters public static Collection data() { - return Arrays.asList(new Object[][]{ + return Arrays.asList(new Object[][] { {OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_LEVELDB}, {OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB} }); @@ -122,7 +124,8 @@ public void testIterator() throws Exception { //As database is empty, check whether iterator is working as expected or // not. - MetaStoreIterator metaStoreIterator = dbStore.iterator(); + MetaStoreIterator metaStoreIterator = + dbStore.iterator(); assertFalse(metaStoreIterator.hasNext()); try { metaStoreIterator.next(); @@ -140,7 +143,7 @@ public void testIterator() throws Exception { int i = 0; while (metaStoreIterator.hasNext()) { - KeyValue val = metaStoreIterator.next(); + MetadataStore.KeyValue val = metaStoreIterator.next(); assertEquals("a" + i, getString(val.getKey())); assertEquals("a-value" + i, getString(val.getValue())); i++; @@ -163,7 +166,6 @@ public void testIterator() throws Exception { } - @Test public void testMetaStoreConfigDifferentFromType() throws IOException { @@ -200,7 +202,6 @@ public void testdbTypeNotSet() throws IOException { GenericTestUtils.LogCapturer logCapturer = GenericTestUtils.LogCapturer.captureLogs(MetadataStoreBuilder.LOG); - File dbDir = GenericTestUtils.getTestDir(getClass().getSimpleName() + "-" + storeImpl.toLowerCase() + "-test"); MetadataStore dbStore = MetadataStoreBuilder.newBuilder().setConf(conf) diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestRetriableTask.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestRetriableTask.java similarity index 98% rename from hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestRetriableTask.java rename to hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestRetriableTask.java index e9df3a878b982..148ccf94a1ba7 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestRetriableTask.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestRetriableTask.java @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.hadoop.utils; +package org.apache.hadoop.hdds.utils; import static org.junit.Assert.assertEquals; import static org.junit.jupiter.api.Assertions.assertThrows; diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestRocksDBStoreMBean.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestRocksDBStoreMBean.java new file mode 100644 index 0000000000000..29c780304cbb5 --- /dev/null +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestRocksDBStoreMBean.java @@ -0,0 +1,234 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

    + * http://www.apache.org/licenses/LICENSE-2.0 + *

    + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.utils; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; +import org.apache.hadoop.metrics2.AbstractMetric; +import org.apache.hadoop.metrics2.MetricsCollector; +import org.apache.hadoop.metrics2.MetricsInfo; +import org.apache.hadoop.metrics2.MetricsRecordBuilder; +import org.apache.hadoop.metrics2.MetricsSource; +import org.apache.hadoop.metrics2.MetricsSystem; +import org.apache.hadoop.metrics2.MetricsTag; +import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; +import org.apache.hadoop.ozone.OzoneConfigKeys; +import org.apache.hadoop.test.GenericTestUtils; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import javax.management.MBeanServer; +import java.io.File; +import java.io.IOException; +import java.lang.management.ManagementFactory; +import java.util.HashMap; +import java.util.Map; + +import static java.nio.charset.StandardCharsets.UTF_8; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +/** + * Test the JMX interface for the rocksdb metastore implementation. + */ +public class TestRocksDBStoreMBean { + + private Configuration conf; + + @Before + public void init() throws Exception { + conf = new OzoneConfiguration(); + + conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL, + OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB); + } + + + @Test + public void testJmxBeans() throws Exception { + + RocksDBStore metadataStore = getTestRocksDBStoreWithData(); + + MBeanServer platformMBeanServer = + ManagementFactory.getPlatformMBeanServer(); + Thread.sleep(2000); + + Object keysWritten = platformMBeanServer + .getAttribute(metadataStore.getStatMBeanName(), "NUMBER_KEYS_WRITTEN"); + + assertEquals(10L, keysWritten); + + Object dbWriteAverage = platformMBeanServer + .getAttribute(metadataStore.getStatMBeanName(), "DB_WRITE_AVERAGE"); + assertTrue((double) dbWriteAverage > 0); + + metadataStore.close(); + + } + + @Test() + public void testDisabledStat() throws Exception { + File testDir = GenericTestUtils + .getTestDir(getClass().getSimpleName() + "-withoutstat"); + + conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS, + OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS_OFF); + + RocksDBStore metadataStore = + (RocksDBStore) MetadataStoreBuilder.newBuilder().setConf(conf) + .setCreateIfMissing(true).setDbFile(testDir).build(); + + Assert.assertNull(metadataStore.getStatMBeanName()); + } + + @Test + public void testMetricsSystemIntegration() throws Exception { + + RocksDBStore metadataStore = getTestRocksDBStoreWithData(); + Thread.sleep(2000); + + MetricsSystem ms = DefaultMetricsSystem.instance(); + MetricsSource rdbSource = + ms.getSource("Rocksdb_TestRocksDBStoreMBean-withstat"); + + BufferedMetricsCollector metricsCollector = new BufferedMetricsCollector(); + rdbSource.getMetrics(metricsCollector, true); + + Map metrics = metricsCollector.getMetricsRecordBuilder() + .getMetrics(); + assertTrue(10.0 == metrics.get("NUMBER_KEYS_WRITTEN")); + assertTrue(metrics.get("DB_WRITE_AVERAGE") > 0); + metadataStore.close(); + } + + private RocksDBStore getTestRocksDBStoreWithData() throws IOException { + File testDir = + GenericTestUtils.getTestDir(getClass().getSimpleName() + "-withstat"); + + conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS, "ALL"); + + RocksDBStore metadataStore = + (RocksDBStore) MetadataStoreBuilder.newBuilder().setConf(conf) + .setCreateIfMissing(true).setDbFile(testDir).build(); + + for (int i = 0; i < 10; i++) { + metadataStore.put("key".getBytes(UTF_8), "value".getBytes(UTF_8)); + } + + return metadataStore; + } +} + +/** + * Test class to buffer a single MetricsRecordBuilder instance. + */ +class BufferedMetricsCollector implements MetricsCollector { + + private BufferedMetricsRecordBuilderImpl metricsRecordBuilder; + + BufferedMetricsCollector() { + metricsRecordBuilder = new BufferedMetricsRecordBuilderImpl(); + } + + public BufferedMetricsRecordBuilderImpl getMetricsRecordBuilder() { + return metricsRecordBuilder; + } + + @Override + public MetricsRecordBuilder addRecord(String s) { + metricsRecordBuilder.setContext(s); + return metricsRecordBuilder; + } + + @Override + public MetricsRecordBuilder addRecord(MetricsInfo metricsInfo) { + return metricsRecordBuilder; + } + + /** + * Test class to buffer a single snapshot of metrics. + */ + class BufferedMetricsRecordBuilderImpl extends MetricsRecordBuilder { + + private Map metrics = new HashMap<>(); + private String contextName; + + public Map getMetrics() { + return metrics; + } + + @Override + public MetricsRecordBuilder tag(MetricsInfo metricsInfo, String s) { + return null; + } + + @Override + public MetricsRecordBuilder add(MetricsTag metricsTag) { + return null; + } + + @Override + public MetricsRecordBuilder add(AbstractMetric abstractMetric) { + return null; + } + + @Override + public MetricsRecordBuilder setContext(String s) { + this.contextName = s; + return this; + } + + @Override + public MetricsRecordBuilder addCounter(MetricsInfo metricsInfo, int i) { + return null; + } + + @Override + public MetricsRecordBuilder addCounter(MetricsInfo metricsInfo, long l) { + metrics.put(metricsInfo.name(), (double)l); + return this; + } + + @Override + public MetricsRecordBuilder addGauge(MetricsInfo metricsInfo, int i) { + return null; + } + + @Override + public MetricsRecordBuilder addGauge(MetricsInfo metricsInfo, long l) { + return null; + } + + @Override + public MetricsRecordBuilder addGauge(MetricsInfo metricsInfo, float v) { + return null; + } + + @Override + public MetricsRecordBuilder addGauge(MetricsInfo metricsInfo, double v) { + metrics.put(metricsInfo.name(), v); + return this; + } + + @Override + public MetricsCollector parent() { + return null; + } + } +} \ No newline at end of file diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestDBConfigFromFile.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestDBConfigFromFile.java similarity index 96% rename from hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestDBConfigFromFile.java rename to hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestDBConfigFromFile.java index b20ca70d33b68..4ba54e98fcabb 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestDBConfigFromFile.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestDBConfigFromFile.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import org.apache.commons.io.FileUtils; import org.apache.hadoop.hdfs.DFSUtil; @@ -40,7 +40,7 @@ import java.util.Arrays; import java.util.List; -import static org.apache.hadoop.utils.db.DBConfigFromFile.getOptionsFileNameFromDB; +import static org.apache.hadoop.hdds.utils.db.DBConfigFromFile.getOptionsFileNameFromDB; /** * DBConf tests. diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestDBStoreBuilder.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestDBStoreBuilder.java similarity index 90% rename from hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestDBStoreBuilder.java rename to hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestDBStoreBuilder.java index d89c4ac023db8..d406060165f32 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestDBStoreBuilder.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestDBStoreBuilder.java @@ -17,10 +17,10 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import org.apache.commons.lang3.RandomStringUtils; -import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.junit.Assert; import org.junit.Before; import org.junit.Rule; @@ -50,14 +50,14 @@ public void setUp() throws Exception { @Test public void builderWithoutAnyParams() throws IOException { - Configuration conf = new Configuration(); + OzoneConfiguration conf = new OzoneConfiguration(); thrown.expect(IOException.class); DBStoreBuilder.newBuilder(conf).build(); } @Test public void builderWithOneParamV1() throws IOException { - Configuration conf = new Configuration(); + OzoneConfiguration conf = new OzoneConfiguration(); thrown.expect(IOException.class); DBStoreBuilder.newBuilder(conf) .setName("Test.db") @@ -66,7 +66,7 @@ public void builderWithOneParamV1() throws IOException { @Test public void builderWithOneParamV2() throws IOException { - Configuration conf = new Configuration(); + OzoneConfiguration conf = new OzoneConfiguration(); File newFolder = folder.newFolder(); if(!newFolder.exists()) { Assert.assertTrue(newFolder.mkdirs()); @@ -79,7 +79,7 @@ public void builderWithOneParamV2() throws IOException { @Test public void builderWithOpenClose() throws Exception { - Configuration conf = new Configuration(); + OzoneConfiguration conf = new OzoneConfiguration(); File newFolder = folder.newFolder(); if(!newFolder.exists()) { Assert.assertTrue(newFolder.mkdirs()); @@ -94,7 +94,7 @@ public void builderWithOpenClose() throws Exception { @Test public void builderWithDoubleTableName() throws Exception { - Configuration conf = new Configuration(); + OzoneConfiguration conf = new OzoneConfiguration(); File newFolder = folder.newFolder(); if(!newFolder.exists()) { Assert.assertTrue(newFolder.mkdirs()); @@ -112,7 +112,7 @@ public void builderWithDoubleTableName() throws Exception { @Test public void builderWithDataWrites() throws Exception { - Configuration conf = new Configuration(); + OzoneConfiguration conf = new OzoneConfiguration(); File newFolder = folder.newFolder(); if(!newFolder.exists()) { Assert.assertTrue(newFolder.mkdirs()); @@ -141,7 +141,7 @@ public void builderWithDataWrites() throws Exception { @Test public void builderWithDiskProfileWrites() throws Exception { - Configuration conf = new Configuration(); + OzoneConfiguration conf = new OzoneConfiguration(); File newFolder = folder.newFolder(); if(!newFolder.exists()) { Assert.assertTrue(newFolder.mkdirs()); diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestRDBStore.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStore.java similarity index 83% rename from hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestRDBStore.java rename to hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStore.java index 24a9ee50bf12d..6084ae96cd5cc 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestRDBStore.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStore.java @@ -17,11 +17,10 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import javax.management.MBeanServer; -import java.io.File; import java.io.IOException; import java.lang.management.ManagementFactory; import java.nio.charset.StandardCharsets; @@ -33,6 +32,7 @@ import java.util.Map; import java.util.Set; +import org.apache.commons.codec.binary.StringUtils; import org.apache.hadoop.hdfs.DFSUtil; import org.apache.commons.lang3.RandomStringUtils; @@ -291,45 +291,59 @@ public void testRocksDBCheckpointCleanup() throws Exception { } } + /** + * Not strictly a unit test. Just a confirmation of the expected behavior + * of RocksDB keyMayExist API. + * Expected behavior - On average, keyMayExist latency < key.get() latency + * for invalid keys. + * @throws Exception if unable to read from RocksDB. + */ @Test - public void testReadOnlyRocksDB() throws Exception { - File dbFile = folder.newFolder(); - byte[] key = "Key1".getBytes(); - byte[] value = "Value1".getBytes(); - - //Create Rdb and write some data into it. - RDBStore newStore = new RDBStore(dbFile, options, configSet); - Assert.assertNotNull("DB Store cannot be null", newStore); - Table firstTable = newStore.getTable(families.get(0)); - Assert.assertNotNull("Table cannot be null", firstTable); - firstTable.put(key, value); - - RocksDBCheckpoint checkpoint = (RocksDBCheckpoint) newStore.getCheckpoint( - true); - - //Create Read Only DB from snapshot of first DB. - RDBStore snapshotStore = new RDBStore(checkpoint.getCheckpointLocation() - .toFile(), options, configSet, new CodecRegistry(), true); - - Assert.assertNotNull("DB Store cannot be null", newStore); - - //Verify read is allowed. - firstTable = snapshotStore.getTable(families.get(0)); - Assert.assertNotNull("Table cannot be null", firstTable); - Assert.assertTrue(Arrays.equals(((byte[])firstTable.get(key)), value)); - - //Verify write is not allowed. - byte[] key2 = - RandomStringUtils.random(10).getBytes(StandardCharsets.UTF_8); - byte[] value2 = - RandomStringUtils.random(10).getBytes(StandardCharsets.UTF_8); - try { - firstTable.put(key2, value2); - Assert.fail(); - } catch (IOException e) { - Assert.assertTrue(e.getMessage() - .contains("Not supported operation in read only mode")); + public void testRocksDBKeyMayExistApi() throws Exception { + try (RDBStore newStore = + new RDBStore(folder.newFolder(), options, configSet)) { + RocksDB db = newStore.getDb(); + + //Test with 50 invalid keys. + long start = System.nanoTime(); + for (int i = 0; i < 50; i++) { + Assert.assertTrue(db.get( + StringUtils.getBytesUtf16("key" + i))== null); + } + long end = System.nanoTime(); + long keyGetLatency = end - start; + + start = System.nanoTime(); + for (int i = 0; i < 50; i++) { + Assert.assertFalse(db.keyMayExist( + StringUtils.getBytesUtf16("key" + i), new StringBuilder())); + } + end = System.nanoTime(); + long keyMayExistLatency = end - start; + + Assert.assertTrue(keyMayExistLatency < keyGetLatency); + } + } + + @Test + public void testGetDBUpdatesSince() throws Exception { + + try (RDBStore newStore = + new RDBStore(folder.newFolder(), options, configSet)) { + + try (Table firstTable = newStore.getTable(families.get(1))) { + firstTable.put(StringUtils.getBytesUtf16("Key1"), StringUtils + .getBytesUtf16("Value1")); + firstTable.put(StringUtils.getBytesUtf16("Key2"), StringUtils + .getBytesUtf16("Value2")); + } + Assert.assertTrue( + newStore.getDb().getLatestSequenceNumber() == 2); + + DBUpdatesWrapper dbUpdatesSince = newStore.getUpdatesSince(0); + Assert.assertEquals(2, dbUpdatesSince.getData().size()); } - checkpoint.cleanupCheckpoint(); } + + } \ No newline at end of file diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestRDBTableStore.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBTableStore.java similarity index 83% rename from hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestRDBTableStore.java rename to hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBTableStore.java index 38d30c11b9b76..788883dbbfff6 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestRDBTableStore.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBTableStore.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.IOException; import java.nio.charset.StandardCharsets; @@ -51,7 +51,8 @@ public class TestRDBTableStore { Arrays.asList(DFSUtil.bytes2String(RocksDB.DEFAULT_COLUMN_FAMILY), "First", "Second", "Third", "Fourth", "Fifth", - "Sixth"); + "Sixth", "Seventh", + "Eighth"); @Rule public TemporaryFolder folder = new TemporaryFolder(); private RDBStore rdbStore = null; @@ -228,4 +229,41 @@ public void forEachAndIterator() throws Exception { } } } + + @Test + public void testIsExist() throws Exception { + try (Table testTable = rdbStore.getTable("Seventh")) { + byte[] key = + RandomStringUtils.random(10).getBytes(StandardCharsets.UTF_8); + byte[] value = + RandomStringUtils.random(10).getBytes(StandardCharsets.UTF_8); + testTable.put(key, value); + Assert.assertTrue(testTable.isExist(key)); + + testTable.delete(key); + Assert.assertFalse(testTable.isExist(key)); + + byte[] invalidKey = + RandomStringUtils.random(5).getBytes(StandardCharsets.UTF_8); + Assert.assertFalse(testTable.isExist(invalidKey)); + } + } + + @Test + public void testCountEstimatedRowsInTable() throws Exception { + try (Table testTable = rdbStore.getTable("Eighth")) { + // Add a few keys + final int numKeys = 12345; + for (int i = 0; i < numKeys; i++) { + byte[] key = + RandomStringUtils.random(10).getBytes(StandardCharsets.UTF_8); + byte[] value = + RandomStringUtils.random(10).getBytes(StandardCharsets.UTF_8); + testTable.put(key, value); + } + long keyCount = testTable.getEstimatedKeyCount(); + // The result should be larger than zero but not exceed(?) numKeys + Assert.assertTrue(keyCount > 0 && keyCount <= numKeys); + } + } } diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestTypedRDBTableStore.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestTypedRDBTableStore.java similarity index 81% rename from hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestTypedRDBTableStore.java rename to hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestTypedRDBTableStore.java index adedcaf52c44e..9ee0d19074aa2 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestTypedRDBTableStore.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/TestTypedRDBTableStore.java @@ -17,7 +17,7 @@ * */ -package org.apache.hadoop.utils.db; +package org.apache.hadoop.hdds.utils.db; import java.io.IOException; import java.util.Arrays; @@ -29,11 +29,11 @@ import com.google.common.base.Optional; import org.apache.hadoop.hdfs.DFSUtil; import org.apache.hadoop.test.GenericTestUtils; -import org.apache.hadoop.utils.db.Table.KeyValue; +import org.apache.hadoop.hdds.utils.db.Table.KeyValue; import org.apache.commons.lang3.RandomStringUtils; -import org.apache.hadoop.utils.db.cache.CacheKey; -import org.apache.hadoop.utils.db.cache.CacheValue; +import org.apache.hadoop.hdds.utils.db.cache.CacheKey; +import org.apache.hadoop.hdds.utils.db.cache.CacheValue; import org.junit.After; import org.junit.Assert; import org.junit.Before; @@ -55,7 +55,8 @@ public class TestTypedRDBTableStore { Arrays.asList(DFSUtil.bytes2String(RocksDB.DEFAULT_COLUMN_FAMILY), "First", "Second", "Third", "Fourth", "Fifth", - "Sixth", "Seven"); + "Sixth", "Seven", "Eighth", + "Ninth"); @Rule public TemporaryFolder folder = new TemporaryFolder(); private RDBStore rdbStore = null; @@ -316,4 +317,57 @@ public void testTypedTableWithCacheWithFewDeletedOperationType() } } + + @Test + public void testIsExist() throws Exception { + try (Table testTable = createTypedTable( + "Eighth")) { + String key = + RandomStringUtils.random(10); + String value = RandomStringUtils.random(10); + testTable.put(key, value); + Assert.assertTrue(testTable.isExist(key)); + + String invalidKey = key + RandomStringUtils.random(1); + Assert.assertFalse(testTable.isExist(invalidKey)); + + testTable.delete(key); + Assert.assertFalse(testTable.isExist(key)); + } + } + + @Test + public void testIsExistCache() throws Exception { + try (Table testTable = createTypedTable( + "Eighth")) { + String key = + RandomStringUtils.random(10); + String value = RandomStringUtils.random(10); + testTable.addCacheEntry(new CacheKey<>(key), + new CacheValue<>(Optional.of(value), 1L)); + Assert.assertTrue(testTable.isExist(key)); + + testTable.addCacheEntry(new CacheKey<>(key), + new CacheValue<>(Optional.absent(), 1L)); + Assert.assertFalse(testTable.isExist(key)); + } + } + + @Test + public void testCountEstimatedRowsInTable() throws Exception { + try (Table testTable = createTypedTable( + "Ninth")) { + // Add a few keys + final int numKeys = 12345; + for (int i = 0; i < numKeys; i++) { + String key = + RandomStringUtils.random(10); + String value = RandomStringUtils.random(10); + testTable.put(key, value); + } + long keyCount = testTable.getEstimatedKeyCount(); + // The result should be larger than zero but not exceed(?) numKeys + Assert.assertTrue(keyCount > 0 && keyCount <= numKeys); + } + } } diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/cache/TestTableCacheImpl.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/cache/TestTableCacheImpl.java new file mode 100644 index 0000000000000..42391297a0a61 --- /dev/null +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/cache/TestTableCacheImpl.java @@ -0,0 +1,173 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.hdds.utils.db.cache; + +import java.util.Arrays; +import java.util.Collection; +import java.util.concurrent.CompletableFuture; + +import com.google.common.base.Optional; +import org.apache.hadoop.test.GenericTestUtils; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; + +import static org.junit.Assert.fail; + +/** + * Class tests partial table cache. + */ +@RunWith(value = Parameterized.class) +public class TestTableCacheImpl { + private TableCache, CacheValue> tableCache; + + private final TableCacheImpl.CacheCleanupPolicy cacheCleanupPolicy; + + + @Parameterized.Parameters + public static Collection policy() { + Object[][] params = new Object[][] { + {TableCacheImpl.CacheCleanupPolicy.NEVER}, + {TableCacheImpl.CacheCleanupPolicy.MANUAL} + }; + return Arrays.asList(params); + } + + public TestTableCacheImpl( + TableCacheImpl.CacheCleanupPolicy cacheCleanupPolicy) { + this.cacheCleanupPolicy = cacheCleanupPolicy; + } + + + @Before + public void create() { + tableCache = + new TableCacheImpl<>(cacheCleanupPolicy); + } + @Test + public void testPartialTableCache() { + + + for (int i = 0; i< 10; i++) { + tableCache.put(new CacheKey<>(Integer.toString(i)), + new CacheValue<>(Optional.of(Integer.toString(i)), i)); + } + + + for (int i=0; i < 10; i++) { + Assert.assertEquals(Integer.toString(i), + tableCache.get(new CacheKey<>(Integer.toString(i))).getCacheValue()); + } + + // On a full table cache if some one calls cleanup it is a no-op. + tableCache.cleanup(4); + + for (int i=5; i < 10; i++) { + Assert.assertEquals(Integer.toString(i), + tableCache.get(new CacheKey<>(Integer.toString(i))).getCacheValue()); + } + } + + + @Test + public void testPartialTableCacheParallel() throws Exception { + + int totalCount = 0; + CompletableFuture future = + CompletableFuture.supplyAsync(() -> { + try { + return writeToCache(10, 1, 0); + } catch (InterruptedException ex) { + fail("writeToCache got interrupt exception"); + } + return 0; + }); + int value = future.get(); + Assert.assertEquals(10, value); + + totalCount += value; + + future = + CompletableFuture.supplyAsync(() -> { + try { + return writeToCache(10, 11, 100); + } catch (InterruptedException ex) { + fail("writeToCache got interrupt exception"); + } + return 0; + }); + + // Check we have first 10 entries in cache. + for (int i=1; i <= 10; i++) { + Assert.assertEquals(Integer.toString(i), + tableCache.get(new CacheKey<>(Integer.toString(i))).getCacheValue()); + } + + + value = future.get(); + Assert.assertEquals(10, value); + + totalCount += value; + + if (cacheCleanupPolicy == TableCacheImpl.CacheCleanupPolicy.MANUAL) { + int deleted = 5; + + // cleanup first 5 entires + tableCache.cleanup(deleted); + + // We should totalCount - deleted entries in cache. + final int tc = totalCount; + GenericTestUtils.waitFor(() -> (tc - deleted == tableCache.size()), 100, + 5000); + // Check if we have remaining entries. + for (int i=6; i <= totalCount; i++) { + Assert.assertEquals(Integer.toString(i), tableCache.get( + new CacheKey<>(Integer.toString(i))).getCacheValue()); + } + tableCache.cleanup(10); + + tableCache.cleanup(totalCount); + + // Cleaned up all entries, so cache size should be zero. + GenericTestUtils.waitFor(() -> (0 == tableCache.size()), 100, + 5000); + } else { + tableCache.cleanup(totalCount); + Assert.assertEquals(totalCount, tableCache.size()); + } + + + } + + private int writeToCache(int count, int startVal, long sleep) + throws InterruptedException { + int counter = 1; + while (counter <= count){ + tableCache.put(new CacheKey<>(Integer.toString(startVal)), + new CacheValue<>(Optional.of(Integer.toString(startVal)), startVal)); + startVal++; + counter++; + Thread.sleep(sleep); + } + return count; + } +} diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/cache/package-info.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/cache/package-info.java new file mode 100644 index 0000000000000..f97fda2d81b95 --- /dev/null +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/cache/package-info.java @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +/** + * Tests for the DB Cache Utilities. + */ +package org.apache.hadoop.hdds.utils.db.cache; \ No newline at end of file diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/package-info.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/package-info.java new file mode 100644 index 0000000000000..f1c7ce139a892 --- /dev/null +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/db/package-info.java @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +/** + * Tests for the DB Utilities. + */ +package org.apache.hadoop.hdds.utils.db; \ No newline at end of file diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/package-info.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/package-info.java new file mode 100644 index 0000000000000..f93e3fd68d2f0 --- /dev/null +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/package-info.java @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

    + * http://www.apache.org/licenses/LICENSE-2.0 + *

    + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +/** + * DB test Utils. + */ +package org.apache.hadoop.hdds.utils; \ No newline at end of file diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/TestOzoneAcls.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/TestOzoneAcls.java deleted file mode 100644 index 03c45c501985d..0000000000000 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/TestOzoneAcls.java +++ /dev/null @@ -1,141 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.ozone; - -import org.junit.Test; - -import java.util.HashMap; -import java.util.Set; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.fail; - -/** - * This class is to test acl stoarge and retreival in ozone store. - */ -public class TestOzoneAcls { - - @Test - public void testAclParse() { - HashMap testMatrix; - testMatrix = new HashMap<>(); - - testMatrix.put("user:bilbo:r", Boolean.TRUE); - testMatrix.put("user:bilbo:w", Boolean.TRUE); - testMatrix.put("user:bilbo:rw", Boolean.TRUE); - testMatrix.put("user:bilbo:wr", Boolean.TRUE); - testMatrix.put(" user:bilbo:wr ", Boolean.TRUE); - - - // ACLs makes no judgement on the quality of - // user names. it is for the userAuth interface - // to determine if a user name is really a name - testMatrix.put(" user:*:rw", Boolean.TRUE); - testMatrix.put(" user:~!:rw", Boolean.TRUE); - - - testMatrix.put("", Boolean.FALSE); - testMatrix.put(null, Boolean.FALSE); - testMatrix.put(" user:bilbo:", Boolean.FALSE); - testMatrix.put(" user:bilbo:rx", Boolean.FALSE); - testMatrix.put(" user:bilbo:mk", Boolean.FALSE); - testMatrix.put(" user::rw", Boolean.FALSE); - testMatrix.put("user11:bilbo:rw", Boolean.FALSE); - testMatrix.put(" user:::rw", Boolean.FALSE); - - testMatrix.put(" group:hobbit:r", Boolean.TRUE); - testMatrix.put(" group:hobbit:w", Boolean.TRUE); - testMatrix.put(" group:hobbit:rw", Boolean.TRUE); - testMatrix.put(" group:hobbit:wr", Boolean.TRUE); - testMatrix.put(" group:*:rw", Boolean.TRUE); - testMatrix.put(" group:~!:rw", Boolean.TRUE); - - testMatrix.put(" group:hobbit:", Boolean.FALSE); - testMatrix.put(" group:hobbit:rx", Boolean.FALSE); - testMatrix.put(" group:hobbit:mk", Boolean.FALSE); - testMatrix.put(" group::", Boolean.FALSE); - testMatrix.put(" group::rw", Boolean.FALSE); - testMatrix.put(" group22:hobbit:", Boolean.FALSE); - testMatrix.put(" group:::rw", Boolean.FALSE); - - testMatrix.put("JUNK group:hobbit:r", Boolean.FALSE); - testMatrix.put("JUNK group:hobbit:w", Boolean.FALSE); - testMatrix.put("JUNK group:hobbit:rw", Boolean.FALSE); - testMatrix.put("JUNK group:hobbit:wr", Boolean.FALSE); - testMatrix.put("JUNK group:*:rw", Boolean.FALSE); - testMatrix.put("JUNK group:~!:rw", Boolean.FALSE); - - testMatrix.put(" world::r", Boolean.TRUE); - testMatrix.put(" world::w", Boolean.TRUE); - testMatrix.put(" world::rw", Boolean.TRUE); - testMatrix.put(" world::wr", Boolean.TRUE); - - testMatrix.put(" world:bilbo:w", Boolean.FALSE); - testMatrix.put(" world:bilbo:rw", Boolean.FALSE); - - Set keys = testMatrix.keySet(); - for (String key : keys) { - if (testMatrix.get(key)) { - OzoneAcl.parseAcl(key); - } else { - try { - OzoneAcl.parseAcl(key); - // should never get here since parseAcl will throw - fail("An exception was expected but did not happen."); - } catch (IllegalArgumentException e) { - // nothing to do - } - } - } - } - - @Test - public void testAclValues() { - OzoneAcl acl = OzoneAcl.parseAcl("user:bilbo:rw"); - assertEquals(acl.getName(), "bilbo"); - assertEquals(OzoneAcl.OzoneACLRights.READ_WRITE, acl.getRights()); - assertEquals(OzoneAcl.OzoneACLType.USER, acl.getType()); - - acl = OzoneAcl.parseAcl("user:bilbo:wr"); - assertEquals("bilbo", acl.getName()); - assertEquals(OzoneAcl.OzoneACLRights.READ_WRITE, acl.getRights()); - assertEquals(OzoneAcl.OzoneACLType.USER, acl.getType()); - - acl = OzoneAcl.parseAcl("user:bilbo:r"); - assertEquals("bilbo", acl.getName()); - assertEquals(OzoneAcl.OzoneACLRights.READ, acl.getRights()); - assertEquals(OzoneAcl.OzoneACLType.USER, acl.getType()); - - acl = OzoneAcl.parseAcl("user:bilbo:w"); - assertEquals("bilbo", acl.getName()); - assertEquals(OzoneAcl.OzoneACLRights.WRITE, acl.getRights()); - assertEquals(OzoneAcl.OzoneACLType.USER, acl.getType()); - - acl = OzoneAcl.parseAcl("group:hobbit:wr"); - assertEquals(acl.getName(), "hobbit"); - assertEquals(OzoneAcl.OzoneACLRights.READ_WRITE, acl.getRights()); - assertEquals(OzoneAcl.OzoneACLType.GROUP, acl.getType()); - - acl = OzoneAcl.parseAcl("world::wr"); - assertEquals(acl.getName(), ""); - assertEquals(OzoneAcl.OzoneACLRights.READ_WRITE, acl.getRights()); - assertEquals(OzoneAcl.OzoneACLType.WORLD, acl.getType()); - } - -} diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java index d2da3e632dcfd..789560a2c3a6e 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java @@ -24,7 +24,6 @@ public enum DummyAction implements AuditAction { CREATE_VOLUME, CREATE_BUCKET, - CREATE_KEY, READ_VOLUME, READ_BUCKET, READ_KEY, diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/TestOzoneAuditLogger.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/TestOzoneAuditLogger.java index 77a6c0baddd0b..518ddaedcf75f 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/TestOzoneAuditLogger.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/TestOzoneAuditLogger.java @@ -153,7 +153,7 @@ private void verifyLog(String expected) throws IOException { assertTrue(lines.size() != 0); assertTrue(expected.equalsIgnoreCase(lines.get(0))); //empty the file - lines.remove(0); + lines.clear(); FileUtils.writeLines(file, lines, false); } diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/common/TestChecksumByteBuffer.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/common/TestChecksumByteBuffer.java new file mode 100644 index 0000000000000..2f466377b4b2c --- /dev/null +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/common/TestChecksumByteBuffer.java @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ozone.common; + +import org.apache.hadoop.util.PureJavaCrc32; +import org.apache.hadoop.util.PureJavaCrc32C; +import org.junit.Assert; +import org.junit.Test; + +import java.nio.charset.StandardCharsets; +import java.util.Random; +import java.util.zip.Checksum; + +/** + * Test {@link ChecksumByteBuffer} implementations. + */ +public class TestChecksumByteBuffer { + @Test + public void testPureJavaCrc32ByteBuffer() { + final Checksum expected = new PureJavaCrc32(); + final ChecksumByteBuffer testee = new PureJavaCrc32ByteBuffer(); + new VerifyChecksumByteBuffer(expected, testee).testCorrectness(); + } + + @Test + public void testPureJavaCrc32CByteBuffer() { + final Checksum expected = new PureJavaCrc32C(); + final ChecksumByteBuffer testee = new PureJavaCrc32CByteBuffer(); + new VerifyChecksumByteBuffer(expected, testee).testCorrectness(); + } + + static class VerifyChecksumByteBuffer { + private final Checksum expected; + private final ChecksumByteBuffer testee; + + VerifyChecksumByteBuffer(Checksum expected, ChecksumByteBuffer testee) { + this.expected = expected; + this.testee = testee; + } + + void testCorrectness() { + checkSame(); + + checkBytes("hello world!".getBytes(StandardCharsets.UTF_8)); + + final Random random = new Random(); + final byte[] bytes = new byte[1 << 10]; + for (int i = 0; i < 1000; i++) { + random.nextBytes(bytes); + checkBytes(bytes, random.nextInt(bytes.length)); + } + } + + void checkBytes(byte[] bytes) { + checkBytes(bytes, bytes.length); + } + + void checkBytes(byte[] bytes, int length) { + expected.reset(); + testee.reset(); + checkSame(); + + for (byte b : bytes) { + expected.update(b); + testee.update(b); + checkSame(); + } + + expected.reset(); + testee.reset(); + + for (int i = 0; i < length; i++) { + expected.update(bytes, 0, i); + testee.update(bytes, 0, i); + checkSame(); + } + + expected.reset(); + testee.reset(); + checkSame(); + } + + private void checkSame() { + Assert.assertEquals(expected.getValue(), testee.getValue()); + } + } +} diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/lock/TestLockManager.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/lock/TestLockManager.java index fa3030d0c3519..e88b1bb121b62 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/lock/TestLockManager.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/lock/TestLockManager.java @@ -29,34 +29,143 @@ public class TestLockManager { @Test(timeout = 1000) - public void testWithDifferentResource() { - LockManager manager = new LockManager<>(new OzoneConfiguration()); - manager.lock("/resourceOne"); + public void testWriteLockWithDifferentResource() { + final LockManager manager = + new LockManager<>(new OzoneConfiguration()); + manager.writeLock("/resourceOne"); // This should work, as they are different resource. - manager.lock("/resourceTwo"); - manager.unlock("/resourceOne"); - manager.unlock("/resourceTwo"); + manager.writeLock("/resourceTwo"); + manager.writeUnlock("/resourceOne"); + manager.writeUnlock("/resourceTwo"); Assert.assertTrue(true); } @Test - public void testWithSameResource() throws Exception { - LockManager manager = new LockManager<>(new OzoneConfiguration()); - manager.lock("/resourceOne"); - AtomicBoolean gotLock = new AtomicBoolean(false); + public void testWriteLockWithSameResource() throws Exception { + final LockManager manager = + new LockManager<>(new OzoneConfiguration()); + final AtomicBoolean gotLock = new AtomicBoolean(false); + manager.writeLock("/resourceOne"); new Thread(() -> { - manager.lock("/resourceOne"); + manager.writeLock("/resourceOne"); gotLock.set(true); - manager.unlock("/resourceOne"); + manager.writeUnlock("/resourceOne"); }).start(); - // Let's give some time for the new thread to run + // Let's give some time for the other thread to run Thread.sleep(100); - // Since the new thread is trying to get lock on same object, it will wait. + // Since the other thread is trying to get write lock on same object, + // it will wait. Assert.assertFalse(gotLock.get()); - manager.unlock("/resourceOne"); - // Since we have released the lock, the new thread should have the lock - // now - // Let's give some time for the new thread to run + manager.writeUnlock("/resourceOne"); + // Since we have released the write lock, the other thread should have + // the lock now + // Let's give some time for the other thread to run + Thread.sleep(100); + Assert.assertTrue(gotLock.get()); + } + + @Test(timeout = 1000) + public void testReadLockWithDifferentResource() { + final LockManager manager = + new LockManager<>(new OzoneConfiguration()); + manager.readLock("/resourceOne"); + manager.readLock("/resourceTwo"); + manager.readUnlock("/resourceOne"); + manager.readUnlock("/resourceTwo"); + Assert.assertTrue(true); + } + + @Test + public void testReadLockWithSameResource() throws Exception { + final LockManager manager = + new LockManager<>(new OzoneConfiguration()); + final AtomicBoolean gotLock = new AtomicBoolean(false); + manager.readLock("/resourceOne"); + new Thread(() -> { + manager.readLock("/resourceOne"); + gotLock.set(true); + manager.readUnlock("/resourceOne"); + }).start(); + // Let's give some time for the other thread to run + Thread.sleep(100); + // Since the new thread is trying to get read lock, it should work. + Assert.assertTrue(gotLock.get()); + manager.readUnlock("/resourceOne"); + } + + @Test + public void testWriteReadLockWithSameResource() throws Exception { + final LockManager manager = + new LockManager<>(new OzoneConfiguration()); + final AtomicBoolean gotLock = new AtomicBoolean(false); + manager.writeLock("/resourceOne"); + new Thread(() -> { + manager.readLock("/resourceOne"); + gotLock.set(true); + manager.readUnlock("/resourceOne"); + }).start(); + // Let's give some time for the other thread to run + Thread.sleep(100); + // Since the other thread is trying to get read lock on same object, + // it will wait. + Assert.assertFalse(gotLock.get()); + manager.writeUnlock("/resourceOne"); + // Since we have released the write lock, the other thread should have + // the lock now + // Let's give some time for the other thread to run + Thread.sleep(100); + Assert.assertTrue(gotLock.get()); + } + + @Test + public void testReadWriteLockWithSameResource() throws Exception { + final LockManager manager = + new LockManager<>(new OzoneConfiguration()); + final AtomicBoolean gotLock = new AtomicBoolean(false); + manager.readLock("/resourceOne"); + new Thread(() -> { + manager.writeLock("/resourceOne"); + gotLock.set(true); + manager.writeUnlock("/resourceOne"); + }).start(); + // Let's give some time for the other thread to run + Thread.sleep(100); + // Since the other thread is trying to get write lock on same object, + // it will wait. + Assert.assertFalse(gotLock.get()); + manager.readUnlock("/resourceOne"); + // Since we have released the read lock, the other thread should have + // the lock now + // Let's give some time for the other thread to run + Thread.sleep(100); + Assert.assertTrue(gotLock.get()); + } + + @Test + public void testMultiReadWriteLockWithSameResource() throws Exception { + final LockManager manager = + new LockManager<>(new OzoneConfiguration()); + final AtomicBoolean gotLock = new AtomicBoolean(false); + manager.readLock("/resourceOne"); + manager.readLock("/resourceOne"); + new Thread(() -> { + manager.writeLock("/resourceOne"); + gotLock.set(true); + manager.writeUnlock("/resourceOne"); + }).start(); + // Let's give some time for the other thread to run + Thread.sleep(100); + // Since the other thread is trying to get write lock on same object, + // it will wait. + Assert.assertFalse(gotLock.get()); + manager.readUnlock("/resourceOne"); + //We have only released one read lock, we still hold another read lock. + Thread.sleep(100); + Assert.assertFalse(gotLock.get()); + manager.readUnlock("/resourceOne"); + // Since we have released the read lock, the other thread should have + // the lock now + // Let's give some time for the other thread to run Thread.sleep(100); Assert.assertTrue(gotLock.get()); } diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestRocksDBStoreMBean.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestRocksDBStoreMBean.java deleted file mode 100644 index ccf19b0e1656d..0000000000000 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/TestRocksDBStoreMBean.java +++ /dev/null @@ -1,96 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - *

    - * http://www.apache.org/licenses/LICENSE-2.0 - *

    - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.utils; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hdds.conf.OzoneConfiguration; -import org.apache.hadoop.ozone.OzoneConfigKeys; -import org.apache.hadoop.test.GenericTestUtils; -import org.junit.Assert; -import org.junit.Before; -import org.junit.Test; - -import javax.management.MBeanServer; -import java.io.File; -import java.lang.management.ManagementFactory; - -import static java.nio.charset.StandardCharsets.UTF_8; - -/** - * Test the JMX interface for the rocksdb metastore implementation. - */ -public class TestRocksDBStoreMBean { - - private Configuration conf; - - @Before - public void init() throws Exception { - conf = new OzoneConfiguration(); - - conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL, - OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB); - } - - - @Test - public void testJmxBeans() throws Exception { - File testDir = - GenericTestUtils.getTestDir(getClass().getSimpleName() + "-withstat"); - - conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS, "ALL"); - - RocksDBStore metadataStore = - (RocksDBStore) MetadataStoreBuilder.newBuilder().setConf(conf) - .setCreateIfMissing(true).setDbFile(testDir).build(); - - for (int i = 0; i < 10; i++) { - metadataStore.put("key".getBytes(UTF_8), "value".getBytes(UTF_8)); - } - - MBeanServer platformMBeanServer = - ManagementFactory.getPlatformMBeanServer(); - Thread.sleep(2000); - - Object keysWritten = platformMBeanServer - .getAttribute(metadataStore.getStatMBeanName(), "NUMBER_KEYS_WRITTEN"); - - Assert.assertEquals(10L, keysWritten); - - Object dbWriteAverage = platformMBeanServer - .getAttribute(metadataStore.getStatMBeanName(), "DB_WRITE_AVERAGE"); - Assert.assertTrue((double) dbWriteAverage > 0); - - metadataStore.close(); - - } - - @Test() - public void testDisabledStat() throws Exception { - File testDir = GenericTestUtils - .getTestDir(getClass().getSimpleName() + "-withoutstat"); - - conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS, - OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS_OFF); - - RocksDBStore metadataStore = - (RocksDBStore) MetadataStoreBuilder.newBuilder().setConf(conf) - .setCreateIfMissing(true).setDbFile(testDir).build(); - - Assert.assertNull(metadataStore.getStatMBeanName()); - } -} \ No newline at end of file diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/cache/TestPartialTableCache.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/cache/TestPartialTableCache.java deleted file mode 100644 index f70665960e2e2..0000000000000 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/cache/TestPartialTableCache.java +++ /dev/null @@ -1,142 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package org.apache.hadoop.utils.db.cache; - -import java.util.concurrent.CompletableFuture; - -import com.google.common.base.Optional; -import org.apache.hadoop.test.GenericTestUtils; -import org.junit.Assert; -import org.junit.Before; -import org.junit.Test; - -import static org.junit.Assert.fail; - -/** - * Class tests partial table cache. - */ -public class TestPartialTableCache { - private TableCache, CacheValue> tableCache; - - @Before - public void create() { - tableCache = new PartialTableCache<>(); - } - @Test - public void testPartialTableCache() { - - - for (int i = 0; i< 10; i++) { - tableCache.put(new CacheKey<>(Integer.toString(i)), - new CacheValue<>(Optional.of(Integer.toString(i)), i)); - } - - - for (int i=0; i < 10; i++) { - Assert.assertEquals(Integer.toString(i), - tableCache.get(new CacheKey<>(Integer.toString(i))).getValue()); - } - - // On a full table cache if some one calls cleanup it is a no-op. - tableCache.cleanup(4); - - for (int i=5; i < 10; i++) { - Assert.assertEquals(Integer.toString(i), - tableCache.get(new CacheKey<>(Integer.toString(i))).getValue()); - } - } - - - @Test - public void testPartialTableCacheParallel() throws Exception { - - int totalCount = 0; - CompletableFuture future = - CompletableFuture.supplyAsync(() -> { - try { - return writeToCache(10, 1, 0); - } catch (InterruptedException ex) { - fail("writeToCache got interrupt exception"); - } - return 0; - }); - int value = future.get(); - Assert.assertEquals(10, value); - - totalCount += value; - - future = - CompletableFuture.supplyAsync(() -> { - try { - return writeToCache(10, 11, 100); - } catch (InterruptedException ex) { - fail("writeToCache got interrupt exception"); - } - return 0; - }); - - // Check we have first 10 entries in cache. - for (int i=1; i <= 10; i++) { - Assert.assertEquals(Integer.toString(i), - tableCache.get(new CacheKey<>(Integer.toString(i))).getValue()); - } - - int deleted = 5; - // cleanup first 5 entires - tableCache.cleanup(deleted); - - value = future.get(); - Assert.assertEquals(10, value); - - totalCount += value; - - // We should totalCount - deleted entries in cache. - final int tc = totalCount; - GenericTestUtils.waitFor(() -> (tc - deleted == tableCache.size()), 100, - 5000); - - // Check if we have remaining entries. - for (int i=6; i <= totalCount; i++) { - Assert.assertEquals(Integer.toString(i), - tableCache.get(new CacheKey<>(Integer.toString(i))).getValue()); - } - - tableCache.cleanup(10); - - tableCache.cleanup(totalCount); - - // Cleaned up all entries, so cache size should be zero. - GenericTestUtils.waitFor(() -> (0 == tableCache.size()), 100, - 5000); - } - - private int writeToCache(int count, int startVal, long sleep) - throws InterruptedException { - int counter = 1; - while (counter <= count){ - tableCache.put(new CacheKey<>(Integer.toString(startVal)), - new CacheValue<>(Optional.of(Integer.toString(startVal)), startVal)); - startVal++; - counter++; - Thread.sleep(sleep); - } - return count; - } -} diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/cache/package-info.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/cache/package-info.java deleted file mode 100644 index b46cf614e8a88..0000000000000 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/cache/package-info.java +++ /dev/null @@ -1,22 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -/** - * Tests for the DB Cache Utilities. - */ -package org.apache.hadoop.utils.db.cache; \ No newline at end of file diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/package-info.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/package-info.java deleted file mode 100644 index f06855e038a33..0000000000000 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/package-info.java +++ /dev/null @@ -1,22 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -/** - * Tests for the DB Utilities. - */ -package org.apache.hadoop.utils.db; \ No newline at end of file diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/package-info.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/package-info.java deleted file mode 100644 index 1fafbd3749dbf..0000000000000 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/package-info.java +++ /dev/null @@ -1,22 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - *

    - * http://www.apache.org/licenses/LICENSE-2.0 - *

    - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -/** - * DB test Utils. - */ -package org.apache.hadoop.utils; \ No newline at end of file diff --git a/hadoop-hdds/config/pom.xml b/hadoop-hdds/config/pom.xml index 075f587a35046..a5955498fff15 100644 --- a/hadoop-hdds/config/pom.xml +++ b/hadoop-hdds/config/pom.xml @@ -15,7 +15,7 @@ +https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -42,25 +42,4 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> - - - - maven-compiler-plugin - 3.1 - - - default-compile - compile - - compile - - - - -proc:none - - - - - - diff --git a/hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileGenerator.java b/hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileGenerator.java index e9e88a0898805..471b679f8452e 100644 --- a/hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileGenerator.java +++ b/hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileGenerator.java @@ -33,6 +33,7 @@ import java.io.OutputStreamWriter; import java.io.Writer; import java.nio.charset.StandardCharsets; +import java.nio.file.NoSuchFileException; import java.util.Set; /** @@ -60,7 +61,7 @@ public boolean process(Set annotations, .getResource(StandardLocation.CLASS_OUTPUT, "", OUTPUT_FILE_NAME).openInputStream()) { appender.load(input); - } catch (FileNotFoundException ex) { + } catch (FileNotFoundException | NoSuchFileException ex) { appender.init(); } @@ -93,21 +94,21 @@ public boolean process(Set annotations, } } - FileObject resource = filer - .createResource(StandardLocation.CLASS_OUTPUT, "", - OUTPUT_FILE_NAME); + } + FileObject resource = filer + .createResource(StandardLocation.CLASS_OUTPUT, "", + OUTPUT_FILE_NAME); - try (Writer writer = new OutputStreamWriter( - resource.openOutputStream(), StandardCharsets.UTF_8)) { - appender.write(writer); - } + try (Writer writer = new OutputStreamWriter( + resource.openOutputStream(), StandardCharsets.UTF_8)) { + appender.write(writer); } + } catch (IOException e) { processingEnv.getMessager().printMessage(Kind.ERROR, - "Can't generate the config file from annotation: " + e.getMessage()); + "Can't generate the config file from annotation: " + e); } return false; } - } diff --git a/hadoop-hdds/config/src/test/resources/META-INF/services/javax.annotation.processing.Processor b/hadoop-hdds/config/src/test/resources/META-INF/services/javax.annotation.processing.Processor new file mode 100644 index 0000000000000..f29efdab384d1 --- /dev/null +++ b/hadoop-hdds/config/src/test/resources/META-INF/services/javax.annotation.processing.Processor @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +org.apache.hadoop.hdds.conf.ConfigFileGenerator diff --git a/hadoop-hdds/container-service/pom.xml b/hadoop-hdds/container-service/pom.xml index c74d68690e360..0eef961733dd7 100644 --- a/hadoop-hdds/container-service/pom.xml +++ b/hadoop-hdds/container-service/pom.xml @@ -15,7 +15,7 @@ +https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -37,6 +37,10 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> org.apache.hadoop hadoop-hdds-server-framework + + io.dropwizard.metrics + metrics-core + org.mockito @@ -51,17 +55,10 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> 1.16 - com.google.code.findbugs - findbugs - 3.0.1 + com.github.spotbugs + spotbugs provided - - - io.dropwizard.metrics - metrics-core - test - @@ -95,8 +92,8 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> - org.codehaus.mojo - findbugs-maven-plugin + com.github.spotbugs + spotbugs-maven-plugin ${basedir}/dev-support/findbugsExcludeFile.xml diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java index 7cd4fd82c574a..b13c37dd45342 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java @@ -26,7 +26,8 @@ import org.apache.hadoop.hdds.cli.HddsVersionProvider; import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdds.protocol.DatanodeDetails; -import org.apache.hadoop.hdds.protocol.SCMSecurityProtocol; +import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto; +import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB; import org.apache.hadoop.hdds.scm.HddsServerUtil; import org.apache.hadoop.hdds.scm.ScmConfigKeys; import org.apache.hadoop.hdds.security.x509.SecurityConfig; @@ -35,7 +36,6 @@ import org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest; import org.apache.hadoop.hdds.tracing.TracingUtil; import org.apache.hadoop.hdfs.DFSConfigKeys; -import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; import org.apache.hadoop.ozone.container.common.helpers.ContainerUtils; import org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine; import org.apache.hadoop.security.SecurityUtil; @@ -169,8 +169,8 @@ public void start(OzoneConfiguration configuration) { } public void start() { - DefaultMetricsSystem.initialize("HddsDatanode"); OzoneConfiguration.activate(); + HddsUtils.initializeMetrics(conf, "HddsDatanode"); if (HddsUtils.isHddsEnabled(conf)) { try { String hostname = HddsUtils.getHostName(conf); @@ -271,16 +271,24 @@ private void getSCMSignedCert(OzoneConfiguration config) { try { PKCS10CertificationRequest csr = getCSR(config); // TODO: For SCM CA we should fetch certificate from multiple SCMs. - SCMSecurityProtocol secureScmClient = - HddsUtils.getScmSecurityClient(config, - HddsUtils.getScmAddressForSecurityProtocol(config)); - - String pemEncodedCert = secureScmClient.getDataNodeCertificate( - datanodeDetails.getProtoBufMessage(), getEncodedString(csr)); - dnCertClient.storeCertificate(pemEncodedCert, true); - datanodeDetails.setCertSerialId(getX509Certificate(pemEncodedCert). - getSerialNumber().toString()); - persistDatanodeDetails(datanodeDetails); + SCMSecurityProtocolClientSideTranslatorPB secureScmClient = + HddsUtils.getScmSecurityClient(config); + SCMGetCertResponseProto response = secureScmClient. + getDataNodeCertificateChain(datanodeDetails.getProtoBufMessage(), + getEncodedString(csr)); + // Persist certificates. + if(response.hasX509CACertificate()) { + String pemEncodedCert = response.getX509Certificate(); + dnCertClient.storeCertificate(pemEncodedCert, true); + dnCertClient.storeCertificate(response.getX509CACertificate(), true, + true); + datanodeDetails.setCertSerialId(getX509Certificate(pemEncodedCert). + getSerialNumber().toString()); + persistDatanodeDetails(datanodeDetails); + } else { + throw new RuntimeException("Unable to retrieve datanode certificate " + + "chain"); + } } catch (IOException | CertificateException e) { LOG.error("Error while storing SCM signed certificate.", e); throw new RuntimeException(e); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerMetrics.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerMetrics.java index 2879001c28b82..9ea4adf8a7250 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerMetrics.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerMetrics.java @@ -46,6 +46,8 @@ @InterfaceAudience.Private @Metrics(about="Storage Container DataNode Metrics", context="dfs") public class ContainerMetrics { + public static final String STORAGE_CONTAINER_METRICS = + "StorageContainerMetrics"; @Metric private MutableCounterLong numOps; private MutableCounterLong[] numOpsArray; private MutableCounterLong[] opsBytesArray; @@ -89,11 +91,16 @@ public static ContainerMetrics create(Configuration conf) { // Percentile measurement is off by default, by watching no intervals int[] intervals = conf.getInts(DFSConfigKeys.DFS_METRICS_PERCENTILES_INTERVALS_KEY); - return ms.register("StorageContainerMetrics", + return ms.register(STORAGE_CONTAINER_METRICS, "Storage Container Node Metrics", new ContainerMetrics(intervals)); } + public static void remove() { + MetricsSystem ms = DefaultMetricsSystem.instance(); + ms.unregisterSource(STORAGE_CONTAINER_METRICS); + } + public void incContainerOpsMetrics(ContainerProtos.Type type) { numOps.incr(); numOpsArray[type.ordinal()].incr(); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java index ec70dbd96e600..85738e240974a 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java @@ -284,6 +284,14 @@ public synchronized boolean isQuasiClosed() { return ContainerDataProto.State.QUASI_CLOSED == state; } + /** + * checks if the container is unhealthy. + * @return - boolean + */ + public synchronized boolean isUnhealthy() { + return ContainerDataProto.State.UNHEALTHY == state; + } + /** * Marks this container as quasi closed. */ @@ -544,4 +552,9 @@ public void computeAndSetChecksum(Yaml yaml) throws IOException { * @return Protocol Buffer Message */ public abstract ContainerProtos.ContainerDataProto getProtoBufMessage(); + + /** + * Returns the blockCommitSequenceId. + */ + public abstract long getBlockCommitSequenceId(); } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerDataYaml.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerDataYaml.java index 15719440f39ef..1f9966c1a76c4 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerDataYaml.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerDataYaml.java @@ -80,6 +80,7 @@ private ContainerDataYaml() { public static void createContainerFile(ContainerType containerType, ContainerData containerData, File containerFile) throws IOException { Writer writer = null; + FileOutputStream out = null; try { // Create Yaml for given container type Yaml yaml = getYamlForContainerType(containerType); @@ -87,13 +88,17 @@ public static void createContainerFile(ContainerType containerType, containerData.computeAndSetChecksum(yaml); // Write the ContainerData with checksum to Yaml file. - writer = new OutputStreamWriter(new FileOutputStream( - containerFile), "UTF-8"); + out = new FileOutputStream( + containerFile); + writer = new OutputStreamWriter(out, "UTF-8"); yaml.dump(containerData, writer); } finally { try { if (writer != null) { + writer.flush(); + // make sure the container metadata is synced to disk. + out.getFD().sync(); writer.close(); } } catch (IOException ex) { diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerSet.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerSet.java index 7dbcbef596415..41415ebe0ac3c 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerSet.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerSet.java @@ -27,20 +27,20 @@ import org.apache.hadoop.hdds.scm.container.common.helpers .StorageContainerException; import org.apache.hadoop.ozone.container.common.interfaces.Container; -import org.apache.hadoop.ozone.container.common - .interfaces.ContainerDeletionChoosingPolicy; +import org.apache.hadoop.ozone.container.common.volume.HddsVolume; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.io.IOException; +import java.util.ArrayList; import java.util.Iterator; -import java.util.List; import java.util.Set; +import java.util.List; +import java.util.Collections; import java.util.Map; import java.util.concurrent.ConcurrentNavigableMap; import java.util.concurrent.ConcurrentSkipListMap; import java.util.concurrent.ConcurrentSkipListSet; -import java.util.stream.Collectors; /** @@ -50,24 +50,26 @@ public class ContainerSet { private static final Logger LOG = LoggerFactory.getLogger(ContainerSet.class); - private final ConcurrentSkipListMap containerMap = new + private final ConcurrentSkipListMap> containerMap = new ConcurrentSkipListMap<>(); private final ConcurrentSkipListSet missingContainerSet = new ConcurrentSkipListSet<>(); /** * Add Container to container map. - * @param container + * @param container container to be added * @return If container is added to containerMap returns true, otherwise * false */ - public boolean addContainer(Container container) throws + public boolean addContainer(Container container) throws StorageContainerException { Preconditions.checkNotNull(container, "container cannot be null"); long containerId = container.getContainerData().getContainerID(); - if(containerMap.putIfAbsent(containerId, container) == null) { - LOG.debug("Container with container Id {} is added to containerMap", - containerId); + if (containerMap.putIfAbsent(containerId, container) == null) { + if (LOG.isDebugEnabled()) { + LOG.debug("Container with container Id {} is added to containerMap", + containerId); + } // wish we could have done this from ContainerData.setState container.getContainerData().commitSpace(); return true; @@ -81,10 +83,10 @@ public boolean addContainer(Container container) throws /** * Returns the Container with specified containerId. - * @param containerId + * @param containerId ID of the container to get * @return Container */ - public Container getContainer(long containerId) { + public Container getContainer(long containerId) { Preconditions.checkState(containerId >= 0, "Container Id cannot be negative."); return containerMap.get(containerId); @@ -92,15 +94,15 @@ public Container getContainer(long containerId) { /** * Removes the Container matching with specified containerId. - * @param containerId + * @param containerId ID of the container to remove * @return If container is removed from containerMap returns true, otherwise * false */ public boolean removeContainer(long containerId) { Preconditions.checkState(containerId >= 0, "Container Id cannot be negative."); - Container removed = containerMap.remove(containerId); - if(removed == null) { + Container removed = containerMap.remove(containerId); + if (removed == null) { LOG.debug("Container with containerId {} is not present in " + "containerMap", containerId); return false; @@ -122,18 +124,33 @@ public int containerCount() { /** * Return an container Iterator over {@link ContainerSet#containerMap}. - * @return {@literal Iterator} + * @return {@literal Iterator>} */ - public Iterator getContainerIterator() { + public Iterator> getContainerIterator() { return containerMap.values().iterator(); } + /** + * Return an iterator of containers associated with the specified volume. + * + * @param volume the HDDS volume which should be used to filter containers + * @return {@literal Iterator>} + */ + public Iterator> getContainerIterator(HddsVolume volume) { + Preconditions.checkNotNull(volume); + Preconditions.checkNotNull(volume.getStorageID()); + String volumeUuid = volume.getStorageID(); + return containerMap.values().stream() + .filter(x -> volumeUuid.equals(x.getContainerData().getVolume() + .getStorageID())) + .iterator(); + } + /** * Return an containerMap iterator over {@link ContainerSet#containerMap}. * @return containerMap Iterator */ - public Iterator> getContainerMapIterator() { - containerMap.keySet().stream().collect(Collectors.toSet()); + public Iterator>> getContainerMapIterator() { return containerMap.entrySet().iterator(); } @@ -142,10 +159,14 @@ public Iterator> getContainerMapIterator() { * @return containerMap */ @VisibleForTesting - public Map getContainerMapCopy() { + public Map> getContainerMapCopy() { return ImmutableMap.copyOf(containerMap); } + public Map> getContainerMap() { + return Collections.unmodifiableMap(containerMap); + } + /** * A simple interface for container Iterations. *

    @@ -157,7 +178,6 @@ public Map getContainerMapCopy() { * @param startContainerId - Return containers with Id >= startContainerId. * @param count - how many to return * @param data - Actual containerData - * @throws StorageContainerException */ public void listContainer(long startContainerId, long count, List data) throws @@ -171,14 +191,14 @@ public void listContainer(long startContainerId, long count, "must be positive"); LOG.debug("listContainer returns containerData starting from {} of count " + "{}", startContainerId, count); - ConcurrentNavigableMap map; + ConcurrentNavigableMap> map; if (startContainerId == 0) { map = containerMap.tailMap(containerMap.firstKey(), true); } else { map = containerMap.tailMap(startContainerId, true); } int currentCount = 0; - for (Container entry : map.values()) { + for (Container entry : map.values()) { if (currentCount < count) { data.add(entry.getContainerData()); currentCount++; @@ -192,7 +212,6 @@ public void listContainer(long startContainerId, long count, * Get container report. * * @return The container report. - * @throws IOException */ public ContainerReportsProto getContainerReport() throws IOException { LOG.debug("Starting container report iteration."); @@ -200,44 +219,63 @@ public ContainerReportsProto getContainerReport() throws IOException { // No need for locking since containerMap is a ConcurrentSkipListMap // And we can never get the exact state since close might happen // after we iterate a point. - List containers = containerMap.values().stream().collect( - Collectors.toList()); + List> containers = new ArrayList<>(containerMap.values()); ContainerReportsProto.Builder crBuilder = ContainerReportsProto.newBuilder(); - for (Container container: containers) { + for (Container container: containers) { crBuilder.addReports(container.getContainerReport()); } return crBuilder.build(); } - public List chooseContainerForBlockDeletion(int count, - ContainerDeletionChoosingPolicy deletionPolicy) - throws StorageContainerException { - Map containerDataMap = containerMap.entrySet().stream() - .filter(e -> deletionPolicy.isValidContainerType( - e.getValue().getContainerType())) - .collect(Collectors.toMap(Map.Entry::getKey, - e -> e.getValue().getContainerData())); - return deletionPolicy - .chooseContainerForBlockDeletion(count, containerDataMap); - } - public Set getMissingContainerSet() { return missingContainerSet; } /** - * Builds the missing container set by taking a diff total no containers - * actually found and number of containers which actually got created. + * Builds the missing container set by taking a diff between total no + * containers actually found and number of containers which actually + * got created. It also validates the BCSID stored in the snapshot file + * for each container as against what is reported in containerScan. * This will only be called during the initialization of Datanode Service * when it still not a part of any write Pipeline. - * @param createdContainerSet ContainerId set persisted in the Ratis snapshot + * @param container2BCSIDMap Map of containerId to BCSID persisted in the + * Ratis snapshot */ - public void buildMissingContainerSet(Set createdContainerSet) { - missingContainerSet.addAll(createdContainerSet); - missingContainerSet.removeAll(containerMap.keySet()); + public void buildMissingContainerSetAndValidate( + Map container2BCSIDMap) { + container2BCSIDMap.entrySet().parallelStream().forEach((mapEntry) -> { + long id = mapEntry.getKey(); + if (!containerMap.containsKey(id)) { + LOG.warn("Adding container {} to missing container set.", id); + missingContainerSet.add(id); + } else { + Container container = containerMap.get(id); + long containerBCSID = container.getBlockCommitSequenceId(); + long snapshotBCSID = mapEntry.getValue(); + if (containerBCSID < snapshotBCSID) { + LOG.warn( + "Marking container {} unhealthy as reported BCSID {} is smaller" + + " than ratis snapshot recorded value {}", id, + containerBCSID, snapshotBCSID); + // just mark the container unhealthy. Once the DatanodeStateMachine + // thread starts it will send container report to SCM where these + // unhealthy containers would be detected + try { + container.markContainerUnhealthy(); + } catch (StorageContainerException sce) { + // The container will still be marked unhealthy in memory even if + // exception occurs. It won't accept any new transactions and will + // be handled by SCM. Eve if dn restarts, it will still be detected + // as unheathy as its BCSID won't change. + LOG.error("Unable to persist unhealthy state for container {}", id); + } + } + } + }); + } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java index 4e8d5b914e3fa..76f6b3cd2f183 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java @@ -67,6 +67,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.io.IOException; import java.util.Map; import java.util.Optional; import java.util.Set; @@ -113,8 +114,6 @@ public void init() { @Override public void shutdown() { - // Shutdown the volumes - volumeSet.shutdown(); } /** @@ -136,8 +135,10 @@ private boolean canIgnoreException(Result result) { } @Override - public void buildMissingContainerSet(Set createdContainerSet) { - containerSet.buildMissingContainerSet(createdContainerSet); + public void buildMissingContainerSetAndValidate( + Map container2BCSIDMap) { + containerSet + .buildMissingContainerSetAndValidate(container2BCSIDMap); } @Override @@ -154,8 +155,10 @@ public ContainerCommandResponseProto dispatch( private ContainerCommandResponseProto dispatchRequest( ContainerCommandRequestProto msg, DispatcherContext dispatcherContext) { Preconditions.checkNotNull(msg); - LOG.trace("Command {}, trace ID: {} ", msg.getCmdType().toString(), - msg.getTraceID()); + if (LOG.isTraceEnabled()) { + LOG.trace("Command {}, trace ID: {} ", msg.getCmdType().toString(), + msg.getTraceID()); + } AuditAction action = ContainerCommandRequestPBHelper.getAuditAction( msg.getCmdType()); @@ -186,9 +189,9 @@ private ContainerCommandResponseProto dispatchRequest( cmdType == ContainerProtos.Type.WriteChunk && (dispatcherContext == null || dispatcherContext.getStage() == DispatcherContext.WriteChunkStage.COMBINED); - Set containerIdSet = null; + Map container2BCSIDMap = null; if (dispatcherContext != null) { - containerIdSet = dispatcherContext.getCreateContainerSet(); + container2BCSIDMap = dispatcherContext.getContainer2BCSIDMap(); } if (isWriteCommitStage) { // check if the container Id exist in the loaded snapshot file. if @@ -197,9 +200,10 @@ private ContainerCommandResponseProto dispatchRequest( // snapshot. // just add it to the list, and remove it from missing container set // as it might have been added in the list during "init". - Preconditions.checkNotNull(containerIdSet); - if (!containerIdSet.contains(containerID)) { - containerIdSet.add(containerID); + Preconditions.checkNotNull(container2BCSIDMap); + if (container2BCSIDMap.get(containerID) == null) { + container2BCSIDMap + .put(containerID, container.getBlockCommitSequenceId()); containerSet.getMissingContainerSet().remove(containerID); } } @@ -229,11 +233,12 @@ private ContainerCommandResponseProto dispatchRequest( audit(action, eventType, params, AuditEventStatus.FAILURE, sce); return ContainerUtils.logAndReturnError(LOG, sce, msg); } - Preconditions.checkArgument(isWriteStage && containerIdSet != null + Preconditions.checkArgument(isWriteStage && container2BCSIDMap != null || dispatcherContext == null); - if (containerIdSet != null) { + if (container2BCSIDMap != null) { // adds this container to list of containers created in the pipeline - containerIdSet.add(containerID); + // with initial BCSID recorded as 0. + container2BCSIDMap.putIfAbsent(containerID, Long.valueOf(0)); } container = getContainer(containerID); } @@ -285,6 +290,11 @@ private ContainerCommandResponseProto dispatchRequest( // state here. Result result = responseProto.getResult(); + if (cmdType == ContainerProtos.Type.CreateContainer + && result == Result.SUCCESS && dispatcherContext != null) { + Preconditions.checkNotNull(dispatcherContext.getContainer2BCSIDMap()); + container2BCSIDMap.putIfAbsent(containerID, Long.valueOf(0)); + } if (!HddsUtils.isReadOnly(msg) && !canIgnoreException(result)) { // If the container is open/closing and the container operation // has failed, it should be first marked unhealthy and the initiate the @@ -299,12 +309,23 @@ private ContainerCommandResponseProto dispatchRequest( State containerState = container.getContainerData().getState(); Preconditions.checkState( containerState == State.OPEN || containerState == State.CLOSING); - container.getContainerData() - .setState(ContainerDataProto.State.UNHEALTHY); + // mark and persist the container state to be unhealthy + try { + handler.markContainerUnhealthy(container); + } catch (IOException ioe) { + // just log the error here in case marking the container fails, + // Return the actual failure response to the client + LOG.error("Failed to mark container " + containerID + " UNHEALTHY. ", + ioe); + } + // in any case, the in memory state of the container should be unhealthy + Preconditions.checkArgument( + container.getContainerData().getState() == State.UNHEALTHY); sendCloseContainerActionIfNeeded(container); } - if(result == Result.SUCCESS) { + if (result == Result.SUCCESS) { + updateBCSID(container, dispatcherContext, cmdType); audit(action, eventType, params, AuditEventStatus.SUCCESS, null); } else { audit(action, eventType, params, AuditEventStatus.FAILURE, @@ -320,6 +341,22 @@ private ContainerCommandResponseProto dispatchRequest( } } + private void updateBCSID(Container container, + DispatcherContext dispatcherContext, ContainerProtos.Type cmdType) { + if (dispatcherContext != null && (cmdType == ContainerProtos.Type.PutBlock + || cmdType == ContainerProtos.Type.PutSmallFile)) { + Preconditions.checkNotNull(container); + long bcsID = container.getBlockCommitSequenceId(); + long containerId = container.getContainerData().getContainerID(); + Map container2BCSIDMap; + container2BCSIDMap = dispatcherContext.getContainer2BCSIDMap(); + Preconditions.checkNotNull(container2BCSIDMap); + Preconditions.checkArgument(container2BCSIDMap.containsKey(containerId)); + // updates the latest BCSID on every putBlock or putSmallFile + // transaction over Ratis. + container2BCSIDMap.computeIfPresent(containerId, (u, v) -> v = bcsID); + } + } /** * Create a container using the input container request. * @param containerRequest - the container request which requires container @@ -518,7 +555,10 @@ private void audit(AuditAction action, EventType eventType, } break; - default: LOG.debug("Invalid audit event status - " + result); + default: + if (LOG.isDebugEnabled()) { + LOG.debug("Invalid audit event status - " + result); + } } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/RandomContainerDeletionChoosingPolicy.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/RandomContainerDeletionChoosingPolicy.java index 5c6c319600e5b..4dde3d6cb7190 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/RandomContainerDeletionChoosingPolicy.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/RandomContainerDeletionChoosingPolicy.java @@ -55,11 +55,12 @@ public List chooseContainerForBlockDeletion(int count, if (currentCount < count) { result.add(entry); currentCount++; - - LOG.debug("Select container {} for block deletion, " - + "pending deletion blocks num: {}.", - entry.getContainerID(), - ((KeyValueContainerData)entry).getNumPendingDeletionBlocks()); + if (LOG.isDebugEnabled()) { + LOG.debug("Select container {} for block deletion, " + + "pending deletion blocks num: {}.", + entry.getContainerID(), + ((KeyValueContainerData) entry).getNumPendingDeletionBlocks()); + } } else { break; } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/TopNOrderedContainerDeletionChoosingPolicy.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/TopNOrderedContainerDeletionChoosingPolicy.java index b17680c419831..41fc26716c190 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/TopNOrderedContainerDeletionChoosingPolicy.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/TopNOrderedContainerDeletionChoosingPolicy.java @@ -69,12 +69,13 @@ public List chooseContainerForBlockDeletion(int count, if (entry.getNumPendingDeletionBlocks() > 0) { result.add(entry); currentCount++; - - LOG.debug( - "Select container {} for block deletion, " - + "pending deletion blocks num: {}.", - entry.getContainerID(), - entry.getNumPendingDeletionBlocks()); + if (LOG.isDebugEnabled()) { + LOG.debug( + "Select container {} for block deletion, " + + "pending deletion blocks num: {}.", + entry.getContainerID(), + entry.getNumPendingDeletionBlocks()); + } } else { LOG.debug("Stop looking for next container, there is no" + " pending deletion block contained in remaining containers."); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Container.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Container.java index 1fcaaf5182345..7f7deaf92063a 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Container.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Container.java @@ -30,6 +30,8 @@ import org.apache.hadoop.hdds.scm.container.common.helpers .StorageContainerException; +import org.apache.hadoop.hdfs.util.Canceler; +import org.apache.hadoop.hdfs.util.DataTransferThrottler; import org.apache.hadoop.hdfs.util.RwLock; import org.apache.hadoop.ozone.container.common.impl.ContainerData; import org.apache.hadoop.ozone.container.common.volume.VolumeSet; @@ -152,8 +154,35 @@ ContainerReplicaProto getContainerReport() */ void updateBlockCommitSequenceId(long blockCommitSequenceId); + /** + * Returns the blockCommitSequenceId. + */ + long getBlockCommitSequenceId(); + /** * check and report the structural integrity of the container. + * @return true if the integrity checks pass + * Scan the container metadata to detect corruption. */ - void check() throws StorageContainerException; + boolean scanMetaData(); + + /** + * Return if the container data should be checksum verified to detect + * corruption. The result depends upon the current state of the container + * (e.g. if a container is accepting writes, it may not be a good idea to + * perform checksum verification to avoid concurrency issues). + */ + boolean shouldScanData(); + + /** + * Perform checksum verification for the container data. + * + * @param throttler A reference of {@link DataTransferThrottler} used to + * perform I/O bandwidth throttling + * @param canceler A reference of {@link Canceler} used to cancel the + * I/O bandwidth throttling (e.g. for shutdown purpose). + * @return true if the checksum verification succeeds + * false otherwise + */ + boolean scanData(DataTransferThrottler throttler, Canceler canceler); } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/ContainerDispatcher.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/ContainerDispatcher.java index e5a74cb10ec99..ee0b6bcb20001 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/ContainerDispatcher.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/ContainerDispatcher.java @@ -26,7 +26,7 @@ import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException; import org.apache.hadoop.ozone.container.common.transport.server.ratis.DispatcherContext; -import java.util.Set; +import java.util.Map; /** * Dispatcher acts as the bridge between the transport layer and @@ -62,9 +62,9 @@ void validateContainerCommand( /** * finds and builds the missing containers in case of a lost disk etc - * in the ContainerSet. + * in the ContainerSet. It also validates the BCSID of the containers found. */ - void buildMissingContainerSet(Set createdContainers); + void buildMissingContainerSetAndValidate(Map container2BCSIDMap); /** * Shutdown Dispatcher services. diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Handler.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Handler.java index a3bb34b565291..8c3b981a09385 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Handler.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Handler.java @@ -19,8 +19,9 @@ package org.apache.hadoop.ozone.container.common.interfaces; -import java.io.FileInputStream; import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdds.protocol.DatanodeDetails; @@ -109,17 +110,31 @@ public abstract ContainerCommandResponseProto handle( DispatcherContext dispatcherContext); /** - * Import container data from a raw input stream. + * Imports container from a raw input stream. */ public abstract Container importContainer( long containerID, long maxSize, String originPipelineId, String originNodeId, - FileInputStream rawContainerStream, + InputStream rawContainerStream, TarContainerPacker packer) throws IOException; + /** + * Exports container to the output stream. + */ + public abstract void exportContainer( + Container container, + OutputStream outputStream, + TarContainerPacker packer) + throws IOException; + + /** + * Stop the Handler. + */ + public abstract void stop(); + /** * Marks the container for closing. Moves the container to CLOSING state. * @@ -129,6 +144,15 @@ public abstract Container importContainer( public abstract void markContainerForClose(Container container) throws IOException; + /** + * Marks the container Unhealthy. Moves the container to UHEALTHY state. + * + * @param container container to update + * @throws IOException in case of exception + */ + public abstract void markContainerUnhealthy(Container container) + throws IOException; + /** * Moves the Container to QUASI_CLOSED state. * diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/report/ReportManager.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/report/ReportManager.java index 8097cd6364eab..536d4cc06b327 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/report/ReportManager.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/report/ReportManager.java @@ -23,10 +23,13 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.ozone.container.common.statemachine.StateContext; import org.apache.hadoop.util.concurrent.HadoopExecutors; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import java.util.ArrayList; import java.util.List; import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; /** * ReportManager is responsible for managing all the {@link ReportPublisher} @@ -34,6 +37,8 @@ * which should be used for scheduling the reports. */ public final class ReportManager { + private static final Logger LOG = + LoggerFactory.getLogger(ReportManager.class); private final StateContext context; private final List publishers; @@ -71,6 +76,11 @@ public void init() { */ public void shutdown() { executorService.shutdown(); + try { + executorService.awaitTermination(5, TimeUnit.SECONDS); + } catch (Exception e) { + LOG.error("Failed to shutdown Report Manager", e); + } } /** diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java index 0119d23fe69c0..c9eb7024eaf18 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java @@ -481,4 +481,9 @@ public long getCommandHandled() { public CommandDispatcher getCommandDispatcher() { return commandDispatcher; } + + @VisibleForTesting + public ReplicationSupervisor getSupervisor() { + return supervisor; + } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/EndpointStateMachine.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/EndpointStateMachine.java index a243c93ed3627..f0064ec5d740e 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/EndpointStateMachine.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/EndpointStateMachine.java @@ -207,7 +207,9 @@ public void logIfNeeded(Exception ex) { TimeUnit.MILLISECONDS.toSeconds( this.getMissedCount() * getScmHeartbeatInterval(this.conf)), ex); } - LOG.trace("Incrementing the Missed count. Ex : {}", ex); + if (LOG.isTraceEnabled()) { + LOG.trace("Incrementing the Missed count. Ex : {}", ex); + } this.incMissed(); } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/SCMConnectionManager.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/SCMConnectionManager.java index f6f64a46119e8..ce31ebdf4d6e2 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/SCMConnectionManager.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/SCMConnectionManager.java @@ -18,6 +18,8 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.IOUtils; +import org.apache.hadoop.io.retry.RetryPolicies; +import org.apache.hadoop.io.retry.RetryPolicy; import org.apache.hadoop.ipc.ProtobufRpcEngine; import org.apache.hadoop.ipc.RPC; import org.apache.hadoop.metrics2.util.MBeans; @@ -38,6 +40,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.concurrent.TimeUnit; import java.util.concurrent.locks.ReadWriteLock; import java.util.concurrent.locks.ReentrantReadWriteLock; @@ -139,10 +142,14 @@ public void addSCMServer(InetSocketAddress address) throws IOException { long version = RPC.getProtocolVersion(StorageContainerDatanodeProtocolPB.class); - StorageContainerDatanodeProtocolPB rpcProxy = RPC.getProxy( + RetryPolicy retryPolicy = + RetryPolicies.retryForeverWithFixedSleep( + 1000, TimeUnit.MILLISECONDS); + StorageContainerDatanodeProtocolPB rpcProxy = RPC.getProtocolProxy( StorageContainerDatanodeProtocolPB.class, version, address, UserGroupInformation.getCurrentUser(), conf, - NetUtils.getDefaultSocketFactory(conf), getRpcTimeout()); + NetUtils.getDefaultSocketFactory(conf), getRpcTimeout(), + retryPolicy).getProxy(); StorageContainerDatanodeProtocolClientSideTranslatorPB rpcClient = new StorageContainerDatanodeProtocolClientSideTranslatorPB(rpcProxy); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CloseContainerCommandHandler.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CloseContainerCommandHandler.java index d4c3ff722adcf..2dec08fe83c4a 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CloseContainerCommandHandler.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CloseContainerCommandHandler.java @@ -86,37 +86,40 @@ public void handle(SCMCommand command, OzoneContainer ozoneContainer, return; } - if (container.getContainerState() == - ContainerProtos.ContainerDataProto.State.CLOSED) { - // Closing a container is an idempotent operation. - return; - } - - // Move the container to CLOSING state + // move the container to CLOSING if in OPEN state controller.markContainerForClose(containerId); - // If the container is part of open pipeline, close it via write channel - if (ozoneContainer.getWriteChannel() - .isExist(closeCommand.getPipelineID())) { + switch (container.getContainerState()) { + case OPEN: + case CLOSING: + // If the container is part of open pipeline, close it via write channel + if (ozoneContainer.getWriteChannel() + .isExist(closeCommand.getPipelineID())) { + ContainerCommandRequestProto request = + getContainerCommandRequestProto(datanodeDetails, + closeCommand.getContainerID()); + ozoneContainer.getWriteChannel() + .submitRequest(request, closeCommand.getPipelineID()); + } else { + // Container should not exist in CLOSING state without a pipeline + controller.markContainerUnhealthy(containerId); + } + break; + case QUASI_CLOSED: if (closeCommand.getForce()) { - LOG.warn("Cannot force close a container when the container is" + - " part of an active pipeline."); - return; + controller.closeContainer(containerId); + break; } - ContainerCommandRequestProto request = - getContainerCommandRequestProto(datanodeDetails, - closeCommand.getContainerID()); - ozoneContainer.getWriteChannel().submitRequest( - request, closeCommand.getPipelineID()); - return; - } - // If we reach here, there is no active pipeline for this container. - if (!closeCommand.getForce()) { - // QUASI_CLOSE the container. - controller.quasiCloseContainer(containerId); - } else { - // SCM told us to force close the container. - controller.closeContainer(containerId); + case CLOSED: + break; + case UNHEALTHY: + case INVALID: + if (LOG.isDebugEnabled()) { + LOG.debug("Cannot close the container #{}, the container is" + + " in {} state.", containerId, container.getContainerState()); + } + default: + break; } } catch (NotLeaderException e) { LOG.debug("Follower cannot close container #{}.", containerId); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CommandDispatcher.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CommandDispatcher.java index 5163d9851a2e8..af854ec3d61a1 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CommandDispatcher.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CommandDispatcher.java @@ -17,6 +17,7 @@ package org.apache.hadoop.ozone.container.common.statemachine.commandhandler; +import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; import org.apache.hadoop.hdds.protocol.proto .StorageContainerDatanodeProtocolProtos.SCMCommandProto.Type; @@ -81,6 +82,11 @@ public CommandHandler getCloseContainerHandler() { return handlerMap.get(Type.closeContainerCommand); } + @VisibleForTesting + public CommandHandler getDeleteBlocksCommandHandler() { + return handlerMap.get(Type.deleteBlocksCommand); + } + /** * Dispatch the command to the correct handler. * diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/DeleteBlocksCommandHandler.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/DeleteBlocksCommandHandler.java index aa63fb48f4de2..cdecf5d7ed470 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/DeleteBlocksCommandHandler.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/DeleteBlocksCommandHandler.java @@ -47,8 +47,8 @@ import org.apache.hadoop.ozone.protocol.commands.DeleteBlocksCommand; import org.apache.hadoop.ozone.protocol.commands.SCMCommand; import org.apache.hadoop.util.Time; -import org.apache.hadoop.utils.BatchOperation; -import org.apache.hadoop.utils.MetadataStore; +import org.apache.hadoop.hdds.utils.BatchOperation; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -127,7 +127,12 @@ public void handle(SCMCommand command, OzoneContainer container, case KeyValueContainer: KeyValueContainerData containerData = (KeyValueContainerData) cont.getContainerData(); - deleteKeyValueContainerBlocks(containerData, entry); + cont.writeLock(); + try { + deleteKeyValueContainerBlocks(containerData, entry); + } finally { + cont.writeUnlock(); + } txResultBuilder.setContainerID(containerId) .setSuccess(true); break; @@ -191,59 +196,69 @@ private void deleteKeyValueContainerBlocks( } if (delTX.getTxID() < containerData.getDeleteTransactionId()) { - LOG.debug(String.format("Ignoring delete blocks for containerId: %d." - + " Outdated delete transactionId %d < %d", containerId, - delTX.getTxID(), containerData.getDeleteTransactionId())); + if (LOG.isDebugEnabled()) { + LOG.debug(String.format("Ignoring delete blocks for containerId: %d." + + " Outdated delete transactionId %d < %d", containerId, + delTX.getTxID(), containerData.getDeleteTransactionId())); + } return; } int newDeletionBlocks = 0; - MetadataStore containerDB = BlockUtils.getDB(containerData, conf); - for (Long blk : delTX.getLocalIDList()) { - BatchOperation batch = new BatchOperation(); - byte[] blkBytes = Longs.toByteArray(blk); - byte[] blkInfo = containerDB.get(blkBytes); - if (blkInfo != null) { - byte[] deletingKeyBytes = - DFSUtil.string2Bytes(OzoneConsts.DELETING_KEY_PREFIX + blk); - byte[] deletedKeyBytes = - DFSUtil.string2Bytes(OzoneConsts.DELETED_KEY_PREFIX + blk); - if (containerDB.get(deletingKeyBytes) != null - || containerDB.get(deletedKeyBytes) != null) { - LOG.debug(String.format( - "Ignoring delete for block %d in container %d." - + " Entry already added.", blk, containerId)); - continue; - } - // Found the block in container db, - // use an atomic update to change its state to deleting. - batch.put(deletingKeyBytes, blkInfo); - batch.delete(blkBytes); - try { - containerDB.writeBatch(batch); - newDeletionBlocks++; - LOG.debug("Transited Block {} to DELETING state in container {}", - blk, containerId); - } catch (IOException e) { - // if some blocks failed to delete, we fail this TX, - // without sending this ACK to SCM, SCM will resend the TX - // with a certain number of retries. - throw new IOException( - "Failed to delete blocks for TXID = " + delTX.getTxID(), e); + try(ReferenceCountedDB containerDB = + BlockUtils.getDB(containerData, conf)) { + for (Long blk : delTX.getLocalIDList()) { + BatchOperation batch = new BatchOperation(); + byte[] blkBytes = Longs.toByteArray(blk); + byte[] blkInfo = containerDB.getStore().get(blkBytes); + if (blkInfo != null) { + byte[] deletingKeyBytes = + DFSUtil.string2Bytes(OzoneConsts.DELETING_KEY_PREFIX + blk); + byte[] deletedKeyBytes = + DFSUtil.string2Bytes(OzoneConsts.DELETED_KEY_PREFIX + blk); + if (containerDB.getStore().get(deletingKeyBytes) != null + || containerDB.getStore().get(deletedKeyBytes) != null) { + if (LOG.isDebugEnabled()) { + LOG.debug(String.format( + "Ignoring delete for block %d in container %d." + + " Entry already added.", blk, containerId)); + } + continue; + } + // Found the block in container db, + // use an atomic update to change its state to deleting. + batch.put(deletingKeyBytes, blkInfo); + batch.delete(blkBytes); + try { + containerDB.getStore().writeBatch(batch); + newDeletionBlocks++; + if (LOG.isDebugEnabled()) { + LOG.debug("Transited Block {} to DELETING state in container {}", + blk, containerId); + } + } catch (IOException e) { + // if some blocks failed to delete, we fail this TX, + // without sending this ACK to SCM, SCM will resend the TX + // with a certain number of retries. + throw new IOException( + "Failed to delete blocks for TXID = " + delTX.getTxID(), e); + } + } else { + if (LOG.isDebugEnabled()) { + LOG.debug("Block {} not found or already under deletion in" + + " container {}, skip deleting it.", blk, containerId); + } } - } else { - LOG.debug("Block {} not found or already under deletion in" - + " container {}, skip deleting it.", blk, containerId); } - } - containerDB - .put(DFSUtil.string2Bytes(OzoneConsts.DELETE_TRANSACTION_KEY_PREFIX), - Longs.toByteArray(delTX.getTxID())); - containerData - .updateDeleteTransactionId(delTX.getTxID()); - // update pending deletion blocks count in in-memory container status - containerData.incrPendingDeletionBlocks(newDeletionBlocks); + containerDB.getStore() + .put(DFSUtil.string2Bytes(OzoneConsts.DELETE_TRANSACTION_KEY_PREFIX), + Longs.toByteArray(delTX.getTxID())); + containerData + .updateDeleteTransactionId(delTX.getTxID()); + // update pending deletion blocks count in in-memory container status + containerData.incrPendingDeletionBlocks(newDeletionBlocks); + } } @Override diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java index 513043f6fdbd6..c50f4573d07f1 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java @@ -135,8 +135,12 @@ public EndpointStateMachine.EndPointStates call() throws Exception { addReports(requestBuilder); addContainerActions(requestBuilder); addPipelineActions(requestBuilder); + SCMHeartbeatRequestProto request = requestBuilder.build(); + if (LOG.isDebugEnabled()) { + LOG.debug("Sending heartbeat message :: {}", request.toString()); + } SCMHeartbeatResponseProto reponse = rpcEndpoint.getEndPoint() - .sendHeartbeat(requestBuilder.build()); + .sendHeartbeat(request); processResponse(reponse, datanodeDetailsProto); rpcEndpoint.setLastSuccessfulHeartbeat(ZonedDateTime.now()); rpcEndpoint.zeroMissedCount(); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/RegisterEndpointTask.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/RegisterEndpointTask.java index 9918f9dfdaba5..b94b1cfc85d8a 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/RegisterEndpointTask.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/RegisterEndpointTask.java @@ -128,6 +128,10 @@ public EndpointStateMachine.EndPointStates call() throws Exception { datanodeDetails.setHostName(response.getHostname()); datanodeDetails.setIpAddress(response.getIpAddress()); } + if (response.hasNetworkName() && response.hasNetworkLocation()) { + datanodeDetails.setNetworkName(response.getNetworkName()); + datanodeDetails.setNetworkLocation(response.getNetworkLocation()); + } EndpointStateMachine.EndPointStates nextState = rpcEndPoint.getState().getNextState(); rpcEndPoint.setState(nextState); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java index 6fe8fd4c2e4bf..bb352ea5165c6 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java @@ -21,6 +21,7 @@ import com.google.common.base.Preconditions; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdds.protocol.DatanodeDetails; +import org.apache.hadoop.hdds.protocol.DatanodeDetails.Port.Name; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos .ContainerCommandRequestProto; @@ -44,19 +45,15 @@ import org.apache.ratis.thirdparty.io.grpc.ServerInterceptors; import org.apache.ratis.thirdparty.io.grpc.netty.GrpcSslContexts; import org.apache.ratis.thirdparty.io.grpc.netty.NettyServerBuilder; -import org.apache.ratis.thirdparty.io.netty.handler.ssl.ClientAuth; import org.apache.ratis.thirdparty.io.netty.handler.ssl.SslContextBuilder; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import java.io.File; import java.io.IOException; -import java.net.InetSocketAddress; -import java.net.ServerSocket; -import java.net.SocketAddress; import java.util.Collections; import java.util.List; import java.util.UUID; +import java.util.concurrent.TimeUnit; /** * Creates a Grpc server endpoint that acts as the communication layer for @@ -65,11 +62,14 @@ public final class XceiverServerGrpc extends XceiverServer { private static final Logger LOG = LoggerFactory.getLogger(XceiverServerGrpc.class); + private static final String COMPONENT = "dn"; private int port; private UUID id; private Server server; private final ContainerDispatcher storageContainer; private boolean isStarted; + private DatanodeDetails datanodeDetails; + /** * Constructs a Grpc server class. @@ -83,25 +83,15 @@ public XceiverServerGrpc(DatanodeDetails datanodeDetails, Configuration conf, Preconditions.checkNotNull(conf); this.id = datanodeDetails.getUuid(); + this.datanodeDetails = datanodeDetails; this.port = conf.getInt(OzoneConfigKeys.DFS_CONTAINER_IPC_PORT, OzoneConfigKeys.DFS_CONTAINER_IPC_PORT_DEFAULT); - // Get an available port on current node and - // use that as the container port + if (conf.getBoolean(OzoneConfigKeys.DFS_CONTAINER_IPC_RANDOM_PORT, OzoneConfigKeys.DFS_CONTAINER_IPC_RANDOM_PORT_DEFAULT)) { - try (ServerSocket socket = new ServerSocket()) { - socket.setReuseAddress(true); - SocketAddress address = new InetSocketAddress(0); - socket.bind(address); - this.port = socket.getLocalPort(); - LOG.info("Found a free port for the server : {}", this.port); - } catch (IOException e) { - LOG.error("Unable find a random free port for the server, " - + "fallback to use default port {}", this.port, e); - } + this.port = 0; } - datanodeDetails.setPort( - DatanodeDetails.newPort(DatanodeDetails.Port.Name.STANDALONE, port)); + NettyServerBuilder nettyServerBuilder = ((NettyServerBuilder) ServerBuilder.forPort(port)) .maxInboundMessageSize(OzoneConsts.OZONE_SCM_CHUNK_MAX_SIZE); @@ -120,20 +110,9 @@ public XceiverServerGrpc(DatanodeDetails datanodeDetails, Configuration conf, } if (getSecConfig().isGrpcTlsEnabled()) { - File privateKeyFilePath = getSecurityConfig().getServerPrivateKeyFile(); - File serverCertChainFilePath = - getSecurityConfig().getServerCertChainFile(); - File clientCertChainFilePath = - getSecurityConfig().getClientCertChainFile(); try { SslContextBuilder sslClientContextBuilder = SslContextBuilder.forServer( - serverCertChainFilePath, privateKeyFilePath); - if (getSecurityConfig().isGrpcMutualTlsRequired() && - clientCertChainFilePath != null) { - // Only needed for mutual TLS - sslClientContextBuilder.clientAuth(ClientAuth.REQUIRE); - sslClientContextBuilder.trustManager(clientCertChainFilePath); - } + caClient.getPrivateKey(), caClient.getCertificate()); SslContextBuilder sslContextBuilder = GrpcSslContexts.configure( sslClientContextBuilder, getSecurityConfig().getGrpcSslProvider()); nettyServerBuilder.sslContext(sslContextBuilder.build()); @@ -164,6 +143,19 @@ public HddsProtos.ReplicationType getServerType() { public void start() throws IOException { if (!isStarted) { server.start(); + int realPort = server.getPort(); + + if (port == 0) { + LOG.info("{} {} is started using port {}", getClass().getSimpleName(), + this.id, realPort); + port = realPort; + } + + //register the real port to the datanode details. + datanodeDetails.setPort(DatanodeDetails + .newPort(Name.STANDALONE, + realPort)); + isStarted = true; } } @@ -172,6 +164,11 @@ public void start() throws IOException { public void stop() { if (isStarted) { server.shutdown(); + try { + server.awaitTermination(5, TimeUnit.SECONDS); + } catch (Exception e) { + LOG.error("failed to shutdown XceiverServerGrpc", e); + } isStarted = false; } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/CSMMetrics.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/CSMMetrics.java index 9ccf88ac77763..9893ae48347f9 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/CSMMetrics.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/CSMMetrics.java @@ -19,11 +19,14 @@ import com.google.common.annotations.VisibleForTesting; import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; import org.apache.hadoop.metrics2.MetricsSystem; import org.apache.hadoop.metrics2.annotation.Metric; import org.apache.hadoop.metrics2.annotation.Metrics; import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; import org.apache.hadoop.metrics2.lib.MutableCounterLong; +import org.apache.hadoop.metrics2.lib.MutableRate; +import org.apache.hadoop.metrics2.lib.MetricsRegistry; import org.apache.ratis.protocol.RaftGroupId; /** @@ -37,15 +40,38 @@ public class CSMMetrics { // ratis op metrics metrics private @Metric MutableCounterLong numWriteStateMachineOps; - private @Metric MutableCounterLong numReadStateMachineOps; + private @Metric MutableCounterLong numQueryStateMachineOps; private @Metric MutableCounterLong numApplyTransactionOps; + private @Metric MutableCounterLong numReadStateMachineOps; + private @Metric MutableCounterLong numBytesWrittenCount; + private @Metric MutableCounterLong numBytesCommittedCount; + + private @Metric MutableRate transactionLatency; + private MutableRate[] opsLatency; + private MetricsRegistry registry = null; // Failure Metrics private @Metric MutableCounterLong numWriteStateMachineFails; - private @Metric MutableCounterLong numReadStateMachineFails; + private @Metric MutableCounterLong numWriteDataFails; + private @Metric MutableCounterLong numQueryStateMachineFails; private @Metric MutableCounterLong numApplyTransactionFails; + private @Metric MutableCounterLong numReadStateMachineFails; + private @Metric MutableCounterLong numReadStateMachineMissCount; + private @Metric MutableCounterLong numStartTransactionVerifyFailures; + private @Metric MutableCounterLong numContainerNotOpenVerifyFailures; + + private @Metric MutableRate applyTransaction; + private @Metric MutableRate writeStateMachineData; public CSMMetrics() { + int numCmdTypes = ContainerProtos.Type.values().length; + this.opsLatency = new MutableRate[numCmdTypes]; + this.registry = new MetricsRegistry(CSMMetrics.class.getSimpleName()); + for (int i = 0; i < numCmdTypes; i++) { + opsLatency[i] = registry.newRate( + ContainerProtos.Type.forNumber(i + 1).toString(), + ContainerProtos.Type.forNumber(i + 1) + " op"); + } } public static CSMMetrics create(RaftGroupId gid) { @@ -59,6 +85,10 @@ public void incNumWriteStateMachineOps() { numWriteStateMachineOps.incr(); } + public void incNumQueryStateMachineOps() { + numQueryStateMachineOps.incr(); + } + public void incNumReadStateMachineOps() { numReadStateMachineOps.incr(); } @@ -71,10 +101,30 @@ public void incNumWriteStateMachineFails() { numWriteStateMachineFails.incr(); } + public void incNumWriteDataFails() { + numWriteDataFails.incr(); + } + + public void incNumQueryStateMachineFails() { + numQueryStateMachineFails.incr(); + } + + public void incNumBytesWrittenCount(long value) { + numBytesWrittenCount.incr(value); + } + + public void incNumBytesCommittedCount(long value) { + numBytesCommittedCount.incr(value); + } + public void incNumReadStateMachineFails() { numReadStateMachineFails.incr(); } + public void incNumReadStateMachineMissCount() { + numReadStateMachineMissCount.incr(); + } + public void incNumApplyTransactionsFails() { numApplyTransactionFails.incr(); } @@ -85,8 +135,8 @@ public long getNumWriteStateMachineOps() { } @VisibleForTesting - public long getNumReadStateMachineOps() { - return numReadStateMachineOps.value(); + public long getNumQueryStateMachineOps() { + return numQueryStateMachineOps.value(); } @VisibleForTesting @@ -100,8 +150,13 @@ public long getNumWriteStateMachineFails() { } @VisibleForTesting - public long getNumReadStateMachineFails() { - return numReadStateMachineFails.value(); + public long getNumWriteDataFails() { + return numWriteDataFails.value(); + } + + @VisibleForTesting + public long getNumQueryStateMachineFails() { + return numQueryStateMachineFails.value(); } @VisibleForTesting @@ -109,6 +164,56 @@ public long getNumApplyTransactionsFails() { return numApplyTransactionFails.value(); } + @VisibleForTesting + public long getNumReadStateMachineFails() { + return numReadStateMachineFails.value(); + } + + @VisibleForTesting + public long getNumReadStateMachineMissCount() { + return numReadStateMachineMissCount.value(); + } + + @VisibleForTesting + public long getNumReadStateMachineOps() { + return numReadStateMachineOps.value(); + } + + @VisibleForTesting + public long getNumBytesWrittenCount() { + return numBytesWrittenCount.value(); + } + + @VisibleForTesting + public long getNumBytesCommittedCount() { + return numBytesCommittedCount.value(); + } + + public MutableRate getApplyTransactionLatency() { + return applyTransaction; + } + + public void incPipelineLatency(ContainerProtos.Type type, long latencyNanos) { + opsLatency[type.ordinal()].add(latencyNanos); + transactionLatency.add(latencyNanos); + } + + public void incNumStartTransactionVerifyFailures() { + numStartTransactionVerifyFailures.incr(); + } + + public void incNumContainerNotOpenVerifyFailures() { + numContainerNotOpenVerifyFailures.incr(); + } + + public void recordApplyTransactionCompletion(long latencyNanos) { + applyTransaction.add(latencyNanos); + } + + public void recordWriteStateMachineCompletion(long latencyNanos) { + writeStateMachineData.add(latencyNanos); + } + public void unRegister() { MetricsSystem ms = DefaultMetricsSystem.instance(); ms.unregisterSource(SOURCE_NAME); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java index 42a4d997a8f6c..b89ec730f7c36 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java @@ -18,28 +18,36 @@ package org.apache.hadoop.ozone.container.common.transport.server.ratis; +import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; import com.google.common.cache.Cache; import com.google.common.cache.CacheBuilder; import org.apache.commons.io.IOUtils; +import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdds.HddsUtils; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; +import org.apache.hadoop.hdds.ratis.ContainerCommandRequestMessage; +import org.apache.hadoop.hdds.scm.ScmConfigKeys; +import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException; import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException; +import org.apache.hadoop.ozone.OzoneConfigKeys; import org.apache.hadoop.ozone.container.common.helpers.ContainerUtils; +import org.apache.hadoop.ozone.container.ozoneimpl.ContainerController; +import org.apache.hadoop.util.Time; import org.apache.ratis.proto.RaftProtos.RaftPeerRole; -import org.apache.ratis.protocol.RaftGroup; import org.apache.ratis.protocol.RaftGroupId; +import org.apache.ratis.protocol.StateMachineException; import org.apache.ratis.server.RaftServer; -import org.apache.ratis.server.impl.RaftServerConstants; import org.apache.ratis.server.impl.RaftServerProxy; import org.apache.ratis.server.protocol.TermIndex; +import org.apache.ratis.server.raftlog.RaftLog; import org.apache.ratis.statemachine.impl.SingleFileSnapshotInfo; import org.apache.ratis.thirdparty.com.google.protobuf .InvalidProtocolBufferException; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.Type; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos. - ContainerIdSetProto; + Container2BCSIDMapProto; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos .ContainerCommandRequestProto; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos @@ -76,16 +84,16 @@ import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.ExecutorService; +import java.util.concurrent.Semaphore; import java.util.concurrent.TimeUnit; import java.util.concurrent.ExecutionException; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.stream.Collectors; -import java.util.Set; -import java.util.concurrent.ConcurrentSkipListSet; +import java.util.concurrent.Executors; import java.io.FileOutputStream; import java.io.FileInputStream; import java.io.OutputStream; - /** A {@link org.apache.ratis.statemachine.StateMachine} for containers. * * The stateMachine is responsible for handling different types of container @@ -130,19 +138,22 @@ public class ContainerStateMachine extends BaseStateMachine { new SimpleStateMachineStorage(); private final RaftGroupId gid; private final ContainerDispatcher dispatcher; + private final ContainerController containerController; private ThreadPoolExecutor chunkExecutor; private final XceiverServerRatis ratisServer; - private final ConcurrentHashMap> - writeChunkFutureMap; + private final ConcurrentHashMap> writeChunkFutureMap; // keeps track of the containers created per pipeline - private final Set createContainerSet; + private final Map container2BCSIDMap; private ExecutorService[] executors; - private final int numExecutors; private final Map applyTransactionCompletionMap; private final Cache stateMachineDataCache; private final boolean isBlockTokenEnabled; private final TokenVerifier tokenVerifier; + private final AtomicBoolean stateMachineHealthy; + + private final Semaphore applyTransactionSemaphore; /** * CSM metrics. */ @@ -150,16 +161,16 @@ public class ContainerStateMachine extends BaseStateMachine { @SuppressWarnings("parameternumber") public ContainerStateMachine(RaftGroupId gid, ContainerDispatcher dispatcher, - ThreadPoolExecutor chunkExecutor, XceiverServerRatis ratisServer, - List executors, long expiryInterval, - boolean isBlockTokenEnabled, TokenVerifier tokenVerifier) { + ContainerController containerController, ThreadPoolExecutor chunkExecutor, + XceiverServerRatis ratisServer, long expiryInterval, + boolean isBlockTokenEnabled, TokenVerifier tokenVerifier, + Configuration conf) { this.gid = gid; this.dispatcher = dispatcher; + this.containerController = containerController; this.chunkExecutor = chunkExecutor; this.ratisServer = ratisServer; metrics = CSMMetrics.create(gid); - this.numExecutors = executors.size(); - this.executors = executors.toArray(new ExecutorService[numExecutors]); this.writeChunkFutureMap = new ConcurrentHashMap<>(); applyTransactionCompletionMap = new ConcurrentHashMap<>(); stateMachineDataCache = CacheBuilder.newBuilder() @@ -169,7 +180,27 @@ public ContainerStateMachine(RaftGroupId gid, ContainerDispatcher dispatcher, .maximumSize(chunkExecutor.getCorePoolSize()).build(); this.isBlockTokenEnabled = isBlockTokenEnabled; this.tokenVerifier = tokenVerifier; - this.createContainerSet = new ConcurrentSkipListSet<>(); + this.container2BCSIDMap = new ConcurrentHashMap<>(); + + final int numContainerOpExecutors = conf.getInt( + OzoneConfigKeys.DFS_CONTAINER_RATIS_NUM_CONTAINER_OP_EXECUTORS_KEY, + OzoneConfigKeys.DFS_CONTAINER_RATIS_NUM_CONTAINER_OP_EXECUTORS_DEFAULT); + int maxPendingApplyTransactions = conf.getInt( + ScmConfigKeys. + DFS_CONTAINER_RATIS_STATEMACHINE_MAX_PENDING_APPLY_TXNS, + ScmConfigKeys. + DFS_CONTAINER_RATIS_STATEMACHINE_MAX_PENDING_APPLY_TXNS_DEFAULT); + applyTransactionSemaphore = new Semaphore(maxPendingApplyTransactions); + stateMachineHealthy = new AtomicBoolean(true); + this.executors = new ExecutorService[numContainerOpExecutors]; + for (int i = 0; i < numContainerOpExecutors; i++) { + final int index = i; + this.executors[index] = Executors.newSingleThreadExecutor(r -> { + Thread t = new Thread(r); + t.setName("RatisApplyTransactionExecutor " + index); + return t; + }); + } } @Override @@ -187,6 +218,7 @@ public void initialize( throws IOException { super.initialize(server, id, raftStorage); storage.init(raftStorage); + ratisServer.notifyGroupAdd(gid); loadSnapshot(storage.getLatestSnapshot()); } @@ -195,31 +227,31 @@ private long loadSnapshot(SingleFileSnapshotInfo snapshot) throws IOException { if (snapshot == null) { TermIndex empty = - TermIndex.newTermIndex(0, RaftServerConstants.INVALID_LOG_INDEX); - LOG.info( - "The snapshot info is null." + "Setting the last applied index to:" - + empty); + TermIndex.newTermIndex(0, RaftLog.INVALID_LOG_INDEX); + LOG.info("{}: The snapshot info is null. Setting the last applied index" + + "to:{}", gid, empty); setLastAppliedTermIndex(empty); - return RaftServerConstants.INVALID_LOG_INDEX; + return empty.getIndex(); } final File snapshotFile = snapshot.getFile().getPath().toFile(); final TermIndex last = SimpleStateMachineStorage.getTermIndexFromSnapshotFile(snapshotFile); - LOG.info("Setting the last applied index to " + last); + LOG.info("{}: Setting the last applied index to {}", gid, last); setLastAppliedTermIndex(last); // initialize the dispatcher with snapshot so that it build the missing // container list try (FileInputStream fin = new FileInputStream(snapshotFile)) { - byte[] containerIds = IOUtils.toByteArray(fin); - ContainerProtos.ContainerIdSetProto proto = - ContainerProtos.ContainerIdSetProto.parseFrom(containerIds); + byte[] container2BCSIDData = IOUtils.toByteArray(fin); + ContainerProtos.Container2BCSIDMapProto proto = + ContainerProtos.Container2BCSIDMapProto + .parseFrom(container2BCSIDData); // read the created containers list from the snapshot file and add it to - // the createContainerSet here. - // createContainerSet will further grow as and when containers get created - createContainerSet.addAll(proto.getContainerIdList()); - dispatcher.buildMissingContainerSet(createContainerSet); + // the container2BCSIDMap here. + // container2BCSIDMap will further grow as and when containers get created + container2BCSIDMap.putAll(proto.getContainer2BCSIDMap()); + dispatcher.buildMissingContainerSetAndValidate(container2BCSIDMap); } return last.getIndex(); } @@ -231,29 +263,47 @@ private long loadSnapshot(SingleFileSnapshotInfo snapshot) * @throws IOException */ public void persistContainerSet(OutputStream out) throws IOException { - ContainerIdSetProto.Builder builder = ContainerIdSetProto.newBuilder(); - builder.addAllContainerId(createContainerSet); + Container2BCSIDMapProto.Builder builder = + Container2BCSIDMapProto.newBuilder(); + builder.putAllContainer2BCSID(container2BCSIDMap); // TODO : while snapshot is being taken, deleteContainer call should not // should not happen. Lock protection will be required if delete // container happens outside of Ratis. IOUtils.write(builder.build().toByteArray(), out); } + public boolean isStateMachineHealthy() { + return stateMachineHealthy.get(); + } + @Override public long takeSnapshot() throws IOException { TermIndex ti = getLastAppliedTermIndex(); - LOG.info("Taking snapshot at termIndex:" + ti); - if (ti != null && ti.getIndex() != RaftServerConstants.INVALID_LOG_INDEX) { + long startTime = Time.monotonicNow(); + if (!isStateMachineHealthy()) { + String msg = + "Failed to take snapshot " + " for " + gid + " as the stateMachine" + + " is unhealthy. The last applied index is at " + ti; + StateMachineException sme = new StateMachineException(msg); + LOG.error(msg); + throw sme; + } + if (ti != null && ti.getIndex() != RaftLog.INVALID_LOG_INDEX) { final File snapshotFile = storage.getSnapshotFile(ti.getTerm(), ti.getIndex()); - LOG.info("Taking a snapshot to file {}", snapshotFile); + LOG.info("{}: Taking a snapshot at:{} file {}", gid, ti, snapshotFile); try (FileOutputStream fos = new FileOutputStream(snapshotFile)) { persistContainerSet(fos); + fos.flush(); + // make sure the snapshot file is synced + fos.getFD().sync(); } catch (IOException ioe) { - LOG.warn("Failed to write snapshot file \"" + snapshotFile - + "\", last applied index=" + ti); + LOG.error("{}: Failed to write snapshot at:{} file {}", gid, ti, + snapshotFile); throw ioe; } + LOG.info("{}: Finished taking a snapshot at:{} file:{} time:{}", gid, ti, + snapshotFile, (Time.monotonicNow() - startTime)); return ti.getIndex(); } return -1; @@ -262,12 +312,19 @@ public long takeSnapshot() throws IOException { @Override public TransactionContext startTransaction(RaftClientRequest request) throws IOException { + long startTime = Time.monotonicNowNanos(); final ContainerCommandRequestProto proto = - getContainerCommandRequestProto(request.getMessage().getContent()); + message2ContainerCommandRequestProto(request.getMessage()); Preconditions.checkArgument(request.getRaftGroupId().equals(gid)); try { dispatcher.validateContainerCommand(proto); } catch (IOException ioe) { + if (ioe instanceof ContainerNotOpenException) { + metrics.incNumContainerNotOpenVerifyFailures(); + } else { + metrics.incNumStartTransactionVerifyFailures(); + LOG.error("startTransaction validation failed on leader", ioe); + } TransactionContext ctxt = TransactionContext.newBuilder() .setClientRequest(request) .setStateMachine(this) @@ -297,6 +354,7 @@ public TransactionContext startTransaction(RaftClientRequest request) .setClientRequest(request) .setStateMachine(this) .setServerRole(RaftPeerRole.LEADER) + .setStateMachineContext(startTime) .setStateMachineData(write.getData()) .setLogData(commitContainerCommandProto.toByteString()) .build(); @@ -305,7 +363,8 @@ public TransactionContext startTransaction(RaftClientRequest request) .setClientRequest(request) .setStateMachine(this) .setServerRole(RaftPeerRole.LEADER) - .setLogData(request.getMessage().getContent()) + .setStateMachineContext(startTime) + .setLogData(proto.toByteString()) .build(); } @@ -325,11 +384,18 @@ private ContainerCommandRequestProto getContainerCommandRequestProto( .setPipelineID(gid.getUuid().toString()).build(); } + private ContainerCommandRequestProto message2ContainerCommandRequestProto( + Message message) throws InvalidProtocolBufferException { + return ContainerCommandRequestMessage.toProto(message.getContent(), gid); + } + private ContainerCommandResponseProto dispatchCommand( ContainerCommandRequestProto requestProto, DispatcherContext context) { - LOG.trace("dispatch {} containerID={} pipelineID={} traceID={}", - requestProto.getCmdType(), requestProto.getContainerID(), - requestProto.getPipelineID(), requestProto.getTraceID()); + if (LOG.isTraceEnabled()) { + LOG.trace("{}: dispatch {} containerID={} pipelineID={} traceID={}", gid, + requestProto.getCmdType(), requestProto.getContainerID(), + requestProto.getPipelineID(), requestProto.getTraceID()); + } if (isBlockTokenEnabled) { try { // ServerInterceptors intercepts incoming request and creates ugi. @@ -345,23 +411,27 @@ private ContainerCommandResponseProto dispatchCommand( } ContainerCommandResponseProto response = dispatcher.dispatch(requestProto, context); - LOG.trace("response {}", response); + if (LOG.isTraceEnabled()) { + LOG.trace("{}: response {}", gid, response); + } return response; } - private Message runCommand(ContainerCommandRequestProto requestProto, + private ContainerCommandResponseProto runCommand( + ContainerCommandRequestProto requestProto, DispatcherContext context) { - return dispatchCommand(requestProto, context)::toByteString; + return dispatchCommand(requestProto, context); } private ExecutorService getCommandExecutor( ContainerCommandRequestProto requestProto) { - int executorId = (int)(requestProto.getContainerID() % numExecutors); + int executorId = (int)(requestProto.getContainerID() % executors.length); return executors[executorId]; } private CompletableFuture handleWriteChunk( - ContainerCommandRequestProto requestProto, long entryIndex, long term) { + ContainerCommandRequestProto requestProto, long entryIndex, long term, + long startTime) { final WriteChunkRequestProto write = requestProto.getWriteChunk(); RaftServer server = ratisServer.getServer(); Preconditions.checkState(server instanceof RaftServerProxy); @@ -377,27 +447,60 @@ private CompletableFuture handleWriteChunk( .setTerm(term) .setLogIndex(entryIndex) .setStage(DispatcherContext.WriteChunkStage.WRITE_DATA) - .setCreateContainerSet(createContainerSet) + .setContainer2BCSIDMap(container2BCSIDMap) .build(); + CompletableFuture raftFuture = new CompletableFuture<>(); // ensure the write chunk happens asynchronously in writeChunkExecutor pool // thread. - CompletableFuture writeChunkFuture = CompletableFuture - .supplyAsync(() -> runCommand(requestProto, context), chunkExecutor); + CompletableFuture writeChunkFuture = + CompletableFuture.supplyAsync(() -> { + try { + return runCommand(requestProto, context); + } catch (Exception e) { + LOG.error(gid + ": writeChunk writeStateMachineData failed: blockId" + + write.getBlockID() + " logIndex " + entryIndex + " chunkName " + + write.getChunkData().getChunkName() + e); + raftFuture.completeExceptionally(e); + throw e; + } + }, chunkExecutor); writeChunkFutureMap.put(entryIndex, writeChunkFuture); - LOG.debug("writeChunk writeStateMachineData : blockId " + write.getBlockID() - + " logIndex " + entryIndex + " chunkName " + write.getChunkData() - .getChunkName()); + if (LOG.isDebugEnabled()) { + LOG.debug(gid + ": writeChunk writeStateMachineData : blockId " + + write.getBlockID() + " logIndex " + entryIndex + " chunkName " + + write.getChunkData().getChunkName()); + } // Remove the future once it finishes execution from the // writeChunkFutureMap. writeChunkFuture.thenApply(r -> { + if (r.getResult() != ContainerProtos.Result.SUCCESS) { + StorageContainerException sce = + new StorageContainerException(r.getMessage(), r.getResult()); + LOG.error(gid + ": writeChunk writeStateMachineData failed: blockId" + + write.getBlockID() + " logIndex " + entryIndex + " chunkName " + + write.getChunkData().getChunkName() + " Error message: " + + r.getMessage() + " Container Result: " + r.getResult()); + metrics.incNumWriteDataFails(); + raftFuture.completeExceptionally(sce); + } else { + metrics.incNumBytesWrittenCount( + requestProto.getWriteChunk().getChunkData().getLen()); + if (LOG.isDebugEnabled()) { + LOG.debug(gid + + ": writeChunk writeStateMachineData completed: blockId" + + write.getBlockID() + " logIndex " + entryIndex + " chunkName " + + write.getChunkData().getChunkName()); + } + raftFuture.complete(r::toByteString); + metrics.recordWriteStateMachineCompletion( + Time.monotonicNowNanos() - startTime); + } + writeChunkFutureMap.remove(entryIndex); - LOG.debug("writeChunk writeStateMachineData completed: blockId " + write - .getBlockID() + " logIndex " + entryIndex + " chunkName " + write - .getChunkData().getChunkName()); return r; }); - return writeChunkFuture; + return raftFuture; } /* @@ -408,6 +511,7 @@ private CompletableFuture handleWriteChunk( public CompletableFuture writeStateMachineData(LogEntryProto entry) { try { metrics.incNumWriteStateMachineOps(); + long writeStateMachineStartTime = Time.monotonicNowNanos(); ContainerCommandRequestProto requestProto = getContainerCommandRequestProto( entry.getStateMachineLogEntry().getLogData()); @@ -424,7 +528,7 @@ public CompletableFuture writeStateMachineData(LogEntryProto entry) { switch (cmdType) { case WriteChunk: return handleWriteChunk(requestProto, entry.getIndex(), - entry.getTerm()); + entry.getTerm(), writeStateMachineStartTime); default: throw new IllegalStateException("Cmd Type:" + cmdType + " should not have state machine data"); @@ -438,12 +542,13 @@ public CompletableFuture writeStateMachineData(LogEntryProto entry) { @Override public CompletableFuture query(Message request) { try { - metrics.incNumReadStateMachineOps(); + metrics.incNumQueryStateMachineOps(); final ContainerCommandRequestProto requestProto = - getContainerCommandRequestProto(request.getContent()); - return CompletableFuture.completedFuture(runCommand(requestProto, null)); + message2ContainerCommandRequestProto(request); + return CompletableFuture + .completedFuture(runCommand(requestProto, null)::toByteString); } catch (IOException e) { - metrics.incNumReadStateMachineFails(); + metrics.incNumQueryStateMachineFails(); return completeExceptionally(e); } } @@ -451,6 +556,9 @@ public CompletableFuture query(Message request) { private ByteString readStateMachineData( ContainerCommandRequestProto requestProto, long term, long index) throws IOException { + // the stateMachine data is not present in the stateMachine cache, + // increment the stateMachine cache miss count + metrics.incNumReadStateMachineMissCount(); WriteChunkRequestProto writeChunkRequestProto = requestProto.getWriteChunk(); ContainerProtos.ChunkInfo chunkInfo = writeChunkRequestProto.getChunkData(); @@ -461,18 +569,24 @@ private ByteString readStateMachineData( .setChunkData(chunkInfo); ContainerCommandRequestProto dataContainerCommandProto = ContainerCommandRequestProto.newBuilder(requestProto) - .setCmdType(Type.ReadChunk) - .setReadChunk(readChunkRequestProto) + .setCmdType(Type.ReadChunk).setReadChunk(readChunkRequestProto) .build(); DispatcherContext context = - new DispatcherContext.Builder() - .setTerm(term) - .setLogIndex(index) - .setReadFromTmpFile(true) - .build(); + new DispatcherContext.Builder().setTerm(term).setLogIndex(index) + .setReadFromTmpFile(true).build(); // read the chunk ContainerCommandResponseProto response = dispatchCommand(dataContainerCommandProto, context); + if (response.getResult() != ContainerProtos.Result.SUCCESS) { + StorageContainerException sce = + new StorageContainerException(response.getMessage(), + response.getResult()); + LOG.error("gid {} : ReadStateMachine failed. cmd {} logIndex {} msg : " + + "{} Container Result: {}", gid, response.getCmdType(), index, + response.getMessage(), response.getResult()); + throw sce; + } + ReadChunkResponseProto responseProto = response.getReadChunk(); ByteString data = responseProto.getData(); @@ -504,7 +618,7 @@ private ByteString getCachedStateMachineData(Long logIndex, long term, */ @Override public CompletableFuture flushStateMachineData(long index) { - List> futureList = + List> futureList = writeChunkFutureMap.entrySet().stream().filter(x -> x.getKey() <= index) .map(Map.Entry::getValue).collect(Collectors.toList()); return CompletableFuture.allOf( @@ -520,6 +634,7 @@ public CompletableFuture flushStateMachineData(long index) { public CompletableFuture readStateMachineData( LogEntryProto entry) { StateMachineLogEntryProto smLogEntryProto = entry.getStateMachineLogEntry(); + metrics.incNumReadStateMachineOps(); if (!getStateMachineData(smLogEntryProto).isEmpty()) { return CompletableFuture.completedFuture(ByteString.EMPTY); } @@ -537,6 +652,7 @@ public CompletableFuture readStateMachineData( getCachedStateMachineData(entry.getIndex(), entry.getTerm(), requestProto)); } catch (ExecutionException e) { + metrics.incNumReadStateMachineFails(); future.completeExceptionally(e); } return future; @@ -547,12 +663,13 @@ public CompletableFuture readStateMachineData( + " cannot have state machine data"); } } catch (Exception e) { - LOG.error("unable to read stateMachineData:" + e); + metrics.incNumReadStateMachineFails(); + LOG.error("{} unable to read stateMachineData:", gid, e); return completeExceptionally(e); } } - private void updateLastApplied() { + private synchronized void updateLastApplied() { Long appliedTerm = null; long appliedIndex = -1; for(long i = getLastAppliedTermIndex().getIndex() + 1;; i++) { @@ -591,7 +708,9 @@ public CompletableFuture applyTransaction(TransactionContext trx) { .setTerm(trx.getLogEntry().getTerm()) .setLogIndex(index); + long applyTxnStartTime = Time.monotonicNowNanos(); try { + applyTransactionSemaphore.acquire(); metrics.incNumApplyTransactionsOps(); ContainerCommandRequestProto requestProto = getContainerCommandRequestProto( @@ -604,24 +723,81 @@ public CompletableFuture applyTransaction(TransactionContext trx) { builder .setStage(DispatcherContext.WriteChunkStage.COMMIT_DATA); } - if (cmdType == Type.WriteChunk || cmdType ==Type.PutSmallFile) { - builder.setCreateContainerSet(createContainerSet); + if (cmdType == Type.WriteChunk || cmdType == Type.PutSmallFile + || cmdType == Type.PutBlock || cmdType == Type.CreateContainer) { + builder.setContainer2BCSIDMap(container2BCSIDMap); } + CompletableFuture applyTransactionFuture = + new CompletableFuture<>(); // Ensure the command gets executed in a separate thread than // stateMachineUpdater thread which is calling applyTransaction here. - CompletableFuture future = CompletableFuture - .supplyAsync(() -> runCommand(requestProto, builder.build()), - getCommandExecutor(requestProto)); - - future.thenAccept(m -> { - final Long previous = - applyTransactionCompletionMap + CompletableFuture future = + CompletableFuture.supplyAsync(() -> { + try { + return runCommand(requestProto, builder.build()); + } catch (Exception e) { + LOG.error("gid {} : ApplyTransaction failed. cmd {} logIndex " + + "{} exception {}", gid, requestProto.getCmdType(), + index, e); + applyTransactionFuture.completeExceptionally(e); + throw e; + } + }, getCommandExecutor(requestProto)); + future.thenApply(r -> { + if (trx.getServerRole() == RaftPeerRole.LEADER) { + long startTime = (long) trx.getStateMachineContext(); + metrics.incPipelineLatency(cmdType, + Time.monotonicNowNanos() - startTime); + } + // ignore close container exception while marking the stateMachine + // unhealthy + if (r.getResult() != ContainerProtos.Result.SUCCESS + && r.getResult() != ContainerProtos.Result.CONTAINER_NOT_OPEN + && r.getResult() != ContainerProtos.Result.CLOSED_CONTAINER_IO) { + StorageContainerException sce = + new StorageContainerException(r.getMessage(), r.getResult()); + LOG.error( + "gid {} : ApplyTransaction failed. cmd {} logIndex {} msg : " + + "{} Container Result: {}", gid, r.getCmdType(), index, + r.getMessage(), r.getResult()); + metrics.incNumApplyTransactionsFails(); + // Since the applyTransaction now is completed exceptionally, + // before any further snapshot is taken , the exception will be + // caught in stateMachineUpdater in Ratis and ratis server will + // shutdown. + applyTransactionFuture.completeExceptionally(sce); + stateMachineHealthy.compareAndSet(true, false); + ratisServer.handleApplyTransactionFailure(gid, trx.getServerRole()); + } else { + if (LOG.isDebugEnabled()) { + LOG.debug( + "gid {} : ApplyTransaction completed. cmd {} logIndex {} msg : " + + "{} Container Result: {}", gid, r.getCmdType(), index, + r.getMessage(), r.getResult()); + } + applyTransactionFuture.complete(r::toByteString); + if (cmdType == Type.WriteChunk || cmdType == Type.PutSmallFile) { + metrics.incNumBytesCommittedCount( + requestProto.getWriteChunk().getChunkData().getLen()); + } + // add the entry to the applyTransactionCompletionMap only if the + // stateMachine is healthy i.e, there has been no applyTransaction + // failures before. + if (isStateMachineHealthy()) { + final Long previous = applyTransactionCompletionMap .put(index, trx.getLogEntry().getTerm()); - Preconditions.checkState(previous == null); - updateLastApplied(); + Preconditions.checkState(previous == null); + updateLastApplied(); + } + } + return applyTransactionFuture; + }).whenComplete((r, t) -> { + applyTransactionSemaphore.release(); + metrics.recordApplyTransactionCompletion( + Time.monotonicNowNanos() - applyTxnStartTime); }); - return future; - } catch (IOException e) { + return applyTransactionFuture; + } catch (IOException | InterruptedException e) { metrics.incNumApplyTransactionsFails(); return completeExceptionally(e); } @@ -633,20 +809,20 @@ private static CompletableFuture completeExceptionally(Exception e) { return future; } - private void evictStateMachineCache() { + @VisibleForTesting + public void evictStateMachineCache() { stateMachineDataCache.invalidateAll(); stateMachineDataCache.cleanUp(); } @Override - public void notifySlowness(RaftGroup group, RoleInfoProto roleInfoProto) { - ratisServer.handleNodeSlowness(group, roleInfoProto); + public void notifySlowness(RoleInfoProto roleInfoProto) { + ratisServer.handleNodeSlowness(gid, roleInfoProto); } @Override - public void notifyExtendedNoLeader(RaftGroup group, - RoleInfoProto roleInfoProto) { - ratisServer.handleNoLeader(group, roleInfoProto); + public void notifyExtendedNoLeader(RoleInfoProto roleInfoProto) { + ratisServer.handleNoLeader(gid, roleInfoProto); } @Override @@ -655,8 +831,41 @@ public void notifyNotLeader(Collection pendingEntries) evictStateMachineCache(); } + @Override + public void notifyLogFailed(Throwable t, LogEntryProto failedEntry) { + ratisServer.handleNodeLogFailure(gid, t); + } + + @Override + public CompletableFuture notifyInstallSnapshotFromLeader( + RoleInfoProto roleInfoProto, TermIndex firstTermIndexInLog) { + ratisServer.handleInstallSnapshotFromLeader(gid, roleInfoProto, + firstTermIndexInLog); + final CompletableFuture future = new CompletableFuture<>(); + future.complete(firstTermIndexInLog); + return future; + } + + @Override + public void notifyGroupRemove() { + ratisServer.notifyGroupRemove(gid); + // Make best effort to quasi-close all the containers on group removal. + // Containers already in terminal state like CLOSED or UNHEALTHY will not + // be affected. + for (Long cid : container2BCSIDMap.keySet()) { + try { + containerController.markContainerForClose(cid); + containerController.quasiCloseContainer(cid); + } catch (IOException e) { + } + } + } + @Override public void close() throws IOException { evictStateMachineCache(); + for (ExecutorService executor : executors) { + executor.shutdown(); + } } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext.java index 446f19f05ea38..7d46910164eea 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext.java @@ -20,7 +20,7 @@ import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; -import java.util.Set; +import java.util.Map; /** * DispatcherContext class holds transport protocol specific context info @@ -45,15 +45,15 @@ public enum WriteChunkStage { // the log index in Ratis log to which the request belongs to private final long logIndex; - private final Set createContainerSet; + private final Map container2BCSIDMap; private DispatcherContext(long term, long index, WriteChunkStage stage, - boolean readFromTmpFile, Set containerSet) { + boolean readFromTmpFile, Map container2BCSIDMap) { this.term = term; this.logIndex = index; this.stage = stage; this.readFromTmpFile = readFromTmpFile; - this.createContainerSet = containerSet; + this.container2BCSIDMap = container2BCSIDMap; } public long getLogIndex() { @@ -72,8 +72,8 @@ public WriteChunkStage getStage() { return stage; } - public Set getCreateContainerSet() { - return createContainerSet; + public Map getContainer2BCSIDMap() { + return container2BCSIDMap; } /** @@ -84,7 +84,7 @@ public static final class Builder { private boolean readFromTmpFile = false; private long term; private long logIndex; - private Set createContainerSet; + private Map container2BCSIDMap; /** * Sets the WriteChunkStage. @@ -131,13 +131,13 @@ public Builder setLogIndex(long index) { } /** - * Sets the createContainerSet to contain all the containerIds per + * Sets the container2BCSIDMap to contain all the containerIds per * RaftGroup. - * @param set createContainerSet + * @param map container2BCSIDMap * @return Builder */ - public Builder setCreateContainerSet(Set set) { - this.createContainerSet = set; + public Builder setContainer2BCSIDMap(Map map) { + this.container2BCSIDMap = map; return this; } /** @@ -147,7 +147,7 @@ public Builder setCreateContainerSet(Set set) { */ public DispatcherContext build() { return new DispatcherContext(term, logIndex, stage, readFromTmpFile, - createContainerSet); + container2BCSIDMap); } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/RatisServerConfiguration.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/RatisServerConfiguration.java new file mode 100644 index 0000000000000..7f112eacd81cf --- /dev/null +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/RatisServerConfiguration.java @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.container.common.transport.server.ratis; + +import org.apache.hadoop.hdds.conf.Config; +import org.apache.hadoop.hdds.conf.ConfigGroup; +import org.apache.hadoop.hdds.conf.ConfigTag; +import org.apache.hadoop.hdds.conf.ConfigType; + +/** + * Holds configuration items for Ratis/Raft server. + */ +@ConfigGroup(prefix = "hdds.ratis.server") +public class RatisServerConfiguration { + + private int numSnapshotsRetained; + + @Config(key = "num.snapshots.retained", + type = ConfigType.INT, + defaultValue = "5", + tags = {ConfigTag.STORAGE}, + description = "Config parameter to specify number of old snapshots " + + "retained at the Ratis leader.") + public void setNumSnapshotsRetained(int numSnapshotsRetained) { + this.numSnapshotsRetained = numSnapshotsRetained; + } + + public int getNumSnapshotsRetained() { + return numSnapshotsRetained; + } + +} diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java index 424281891b680..80e91cdf55de0 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java @@ -19,46 +19,37 @@ package org.apache.hadoop.ozone.container.common.transport.server.ratis; import com.google.common.annotations.VisibleForTesting; -import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.StorageUnit; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdds.protocol.DatanodeDetails; -import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos - .ContainerCommandRequestProto; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandRequestProto; import org.apache.hadoop.hdds.protocol.proto.HddsProtos; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.PipelineReport; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.ClosePipelineInfo; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.PipelineAction; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.PipelineReport; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.ClosePipelineInfo; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.PipelineAction; +import org.apache.hadoop.hdds.ratis.ContainerCommandRequestMessage; import org.apache.hadoop.hdds.scm.HddsServerUtil; import org.apache.hadoop.hdds.scm.pipeline.PipelineID; import org.apache.hadoop.hdds.security.x509.SecurityConfig; import org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient; import org.apache.hadoop.hdds.tracing.TracingUtil; import org.apache.hadoop.ozone.OzoneConfigKeys; + import org.apache.hadoop.ozone.OzoneConsts; import org.apache.hadoop.ozone.container.common.interfaces.ContainerDispatcher; import org.apache.hadoop.ozone.container.common.statemachine.StateContext; import org.apache.hadoop.ozone.container.common.transport.server.XceiverServer; import io.opentracing.Scope; +import org.apache.hadoop.ozone.container.ozoneimpl.ContainerController; import org.apache.ratis.RaftConfigKeys; -import org.apache.ratis.RatisHelper; +import org.apache.hadoop.hdds.ratis.RatisHelper; import org.apache.ratis.conf.RaftProperties; import org.apache.ratis.grpc.GrpcConfigKeys; import org.apache.ratis.grpc.GrpcFactory; import org.apache.ratis.grpc.GrpcTlsConfig; import org.apache.ratis.netty.NettyConfigKeys; -import org.apache.ratis.protocol.RaftClientRequest; -import org.apache.ratis.protocol.Message; -import org.apache.ratis.protocol.RaftClientReply; -import org.apache.ratis.protocol.ClientId; -import org.apache.ratis.protocol.NotLeaderException; -import org.apache.ratis.protocol.StateMachineException; -import org.apache.ratis.protocol.RaftPeerId; -import org.apache.ratis.protocol.RaftGroup; -import org.apache.ratis.protocol.RaftGroupId; +import org.apache.ratis.protocol.*; import org.apache.ratis.rpc.RpcType; import org.apache.ratis.rpc.SupportedRpcType; import org.apache.ratis.server.RaftServer; @@ -66,6 +57,8 @@ import org.apache.ratis.proto.RaftProtos; import org.apache.ratis.proto.RaftProtos.RoleInfoProto; import org.apache.ratis.proto.RaftProtos.ReplicationLevel; +import org.apache.ratis.server.protocol.TermIndex; +import org.apache.ratis.server.impl.RaftServerProxy; import org.apache.ratis.util.SizeInBytes; import org.apache.ratis.util.TimeDuration; import org.slf4j.Logger; @@ -73,18 +66,15 @@ import java.io.File; import java.io.IOException; -import java.net.InetSocketAddress; -import java.net.ServerSocket; -import java.net.SocketAddress; -import java.util.ArrayList; -import java.util.Collections; +import java.util.HashSet; import java.util.List; import java.util.Objects; +import java.util.Collections; +import java.util.Set; import java.util.UUID; +import java.util.ArrayList; import java.util.concurrent.ArrayBlockingQueue; import java.util.concurrent.ThreadPoolExecutor; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicLong; @@ -101,26 +91,35 @@ private static long nextCallId() { return CALL_ID_COUNTER.getAndIncrement() & Long.MAX_VALUE; } - private final int port; + private int port; private final RaftServer server; private ThreadPoolExecutor chunkExecutor; - private final List executors; private final ContainerDispatcher dispatcher; + private final ContainerController containerController; private ClientId clientId = ClientId.randomId(); private final StateContext context; private final ReplicationLevel replicationLevel; private long nodeFailureTimeoutMs; private final long cacheEntryExpiryInteval; private boolean isStarted = false; + private DatanodeDetails datanodeDetails; + private final OzoneConfiguration conf; + // TODO: Remove the gids set when Ratis supports an api to query active + // pipelines + private final Set raftGids = new HashSet<>(); + @SuppressWarnings("parameternumber") private XceiverServerRatis(DatanodeDetails dd, int port, - ContainerDispatcher dispatcher, Configuration conf, StateContext - context, GrpcTlsConfig tlsConfig, CertificateClient caClient) + ContainerDispatcher dispatcher, ContainerController containerController, + StateContext context, GrpcTlsConfig tlsConfig, CertificateClient caClient, + OzoneConfiguration conf) throws IOException { super(conf, caClient); + this.conf = conf; Objects.requireNonNull(dd, "id == null"); + datanodeDetails = dd; this.port = port; - RaftProperties serverProperties = newRaftProperties(conf); + RaftProperties serverProperties = newRaftProperties(); final int numWriteChunkThreads = conf.getInt( OzoneConfigKeys.DFS_CONTAINER_RATIS_NUM_WRITE_CHUNK_THREADS_KEY, OzoneConfigKeys.DFS_CONTAINER_RATIS_NUM_WRITE_CHUNK_THREADS_DEFAULT); @@ -129,28 +128,22 @@ private XceiverServerRatis(DatanodeDetails dd, int port, 100, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024), new ThreadPoolExecutor.CallerRunsPolicy()); - final int numContainerOpExecutors = conf.getInt( - OzoneConfigKeys.DFS_CONTAINER_RATIS_NUM_CONTAINER_OP_EXECUTORS_KEY, - OzoneConfigKeys.DFS_CONTAINER_RATIS_NUM_CONTAINER_OP_EXECUTORS_DEFAULT); this.context = context; this.replicationLevel = conf.getEnum(OzoneConfigKeys.DFS_CONTAINER_RATIS_REPLICATION_LEVEL_KEY, OzoneConfigKeys.DFS_CONTAINER_RATIS_REPLICATION_LEVEL_DEFAULT); - this.executors = new ArrayList<>(); cacheEntryExpiryInteval = conf.getTimeDuration(OzoneConfigKeys. DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL, OzoneConfigKeys. DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL_DEFAULT, TimeUnit.MILLISECONDS); this.dispatcher = dispatcher; - for (int i = 0; i < numContainerOpExecutors; i++) { - executors.add(Executors.newSingleThreadExecutor()); - } + this.containerController = containerController; - RaftServer.Builder builder = RaftServer.newBuilder() - .setServerId(RatisHelper.toRaftPeerId(dd)) - .setProperties(serverProperties) - .setStateMachineRegistry(this::getStateMachine); + RaftServer.Builder builder = + RaftServer.newBuilder().setServerId(RatisHelper.toRaftPeerId(dd)) + .setProperties(serverProperties) + .setStateMachineRegistry(this::getStateMachine); if (tlsConfig != null) { builder.setParameters(GrpcFactory.newRaftParameters(tlsConfig)); } @@ -158,23 +151,24 @@ private XceiverServerRatis(DatanodeDetails dd, int port, } private ContainerStateMachine getStateMachine(RaftGroupId gid) { - return new ContainerStateMachine(gid, dispatcher, chunkExecutor, this, - Collections.unmodifiableList(executors), cacheEntryExpiryInteval, - getSecurityConfig().isBlockTokenEnabled(), getBlockTokenVerifier()); + return new ContainerStateMachine(gid, dispatcher, containerController, + chunkExecutor, this, cacheEntryExpiryInteval, + getSecurityConfig().isBlockTokenEnabled(), getBlockTokenVerifier(), + conf); } - private RaftProperties newRaftProperties(Configuration conf) { + private RaftProperties newRaftProperties() { final RaftProperties properties = new RaftProperties(); // Set rpc type - final RpcType rpc = setRpcType(conf, properties); + final RpcType rpc = setRpcType(properties); // set raft segment size - setRaftSegmentSize(conf, properties); + setRaftSegmentSize(properties); // set raft segment pre-allocated size final int raftSegmentPreallocatedSize = - setRaftSegmentPreallocatedSize(conf, properties); + setRaftSegmentPreallocatedSize(properties); // Set max write buffer size, which is the scm chunk size final int maxChunkSize = setMaxWriteBuffer(properties); @@ -196,19 +190,19 @@ private RaftProperties newRaftProperties(Configuration conf) { .setSyncTimeout(properties, dataSyncTimeout); // Set the server Request timeout - setServerRequestTimeout(conf, properties); + setServerRequestTimeout(properties); // set timeout for a retry cache entry - setTimeoutForRetryCache(conf, properties); + setTimeoutForRetryCache(properties); // Set the ratis leader election timeout - setRatisLeaderElectionTimeout(conf, properties); + setRatisLeaderElectionTimeout(properties); // Set the maximum cache segments RaftServerConfigKeys.Log.setMaxCachedSegmentNum(properties, 2); // set the node failure timeout - setNodeFailureTimeout(conf, properties); + setNodeFailureTimeout(properties); // Set the ratis storage directory String storageDir = HddsServerUtil.getOzoneDatanodeRatisDirectory(conf); @@ -233,6 +227,11 @@ private RaftProperties newRaftProperties(Configuration conf) { setAutoTriggerEnabled(properties, true); RaftServerConfigKeys.Snapshot. setAutoTriggerThreshold(properties, snapshotThreshold); + int maxPendingRequets = conf.getInt( + OzoneConfigKeys.DFS_CONTAINER_RATIS_LEADER_NUM_PENDING_REQUESTS, + OzoneConfigKeys.DFS_CONTAINER_RATIS_LEADER_NUM_PENDING_REQUESTS_DEFAULT + ); + RaftServerConfigKeys.Write.setElementLimit(properties, maxPendingRequets); int logQueueNumElements = conf.getInt(OzoneConfigKeys.DFS_CONTAINER_RATIS_LOG_QUEUE_NUM_ELEMENTS, OzoneConfigKeys.DFS_CONTAINER_RATIS_LOG_QUEUE_NUM_ELEMENTS_DEFAULT); @@ -240,8 +239,9 @@ private RaftProperties newRaftProperties(Configuration conf) { OzoneConfigKeys.DFS_CONTAINER_RATIS_LOG_QUEUE_BYTE_LIMIT, OzoneConfigKeys.DFS_CONTAINER_RATIS_LOG_QUEUE_BYTE_LIMIT_DEFAULT, StorageUnit.BYTES); - RaftServerConfigKeys.Log.setElementLimit(properties, logQueueNumElements); - RaftServerConfigKeys.Log.setByteLimit(properties, logQueueByteLimit); + RaftServerConfigKeys.Log.setQueueElementLimit( + properties, logQueueNumElements); + RaftServerConfigKeys.Log.setQueueByteLimit(properties, logQueueByteLimit); int numSyncRetries = conf.getInt( OzoneConfigKeys.DFS_CONTAINER_RATIS_STATEMACHINEDATA_SYNC_RETRIES, @@ -251,13 +251,28 @@ private RaftProperties newRaftProperties(Configuration conf) { numSyncRetries); // Enable the StateMachineCaching - RaftServerConfigKeys.Log.StateMachineData - .setCachingEnabled(properties, true); + RaftServerConfigKeys.Log.StateMachineData.setCachingEnabled( + properties, true); + + RaftServerConfigKeys.Log.Appender.setInstallSnapshotEnabled(properties, + false); + + int purgeGap = conf.getInt( + OzoneConfigKeys.DFS_CONTAINER_RATIS_LOG_PURGE_GAP, + OzoneConfigKeys.DFS_CONTAINER_RATIS_LOG_PURGE_GAP_DEFAULT); + RaftServerConfigKeys.Log.setPurgeGap(properties, purgeGap); + + //Set the number of Snapshots Retained. + RatisServerConfiguration ratisServerConfiguration = + conf.getObject(RatisServerConfiguration.class); + int numSnapshotsRetained = + ratisServerConfiguration.getNumSnapshotsRetained(); + RaftServerConfigKeys.Snapshot.setRetentionFileNum(properties, + numSnapshotsRetained); return properties; } - private void setNodeFailureTimeout(Configuration conf, - RaftProperties properties) { + private void setNodeFailureTimeout(RaftProperties properties) { TimeUnit timeUnit; long duration; timeUnit = OzoneConfigKeys.DFS_RATIS_SERVER_FAILURE_DURATION_DEFAULT @@ -268,15 +283,14 @@ private void setNodeFailureTimeout(Configuration conf, .getDuration(), timeUnit); final TimeDuration nodeFailureTimeout = TimeDuration.valueOf(duration, timeUnit); - RaftServerConfigKeys.setLeaderElectionTimeout(properties, + RaftServerConfigKeys.Notification.setNoLeaderTimeout(properties, nodeFailureTimeout); RaftServerConfigKeys.Rpc.setSlownessTimeout(properties, nodeFailureTimeout); nodeFailureTimeoutMs = nodeFailureTimeout.toLong(TimeUnit.MILLISECONDS); } - private void setRatisLeaderElectionTimeout(Configuration conf, - RaftProperties properties) { + private void setRatisLeaderElectionTimeout(RaftProperties properties) { long duration; TimeUnit leaderElectionMinTimeoutUnit = OzoneConfigKeys. @@ -297,8 +311,7 @@ private void setRatisLeaderElectionTimeout(Configuration conf, TimeDuration.valueOf(leaderElectionMaxTimeout, TimeUnit.MILLISECONDS)); } - private void setTimeoutForRetryCache(Configuration conf, - RaftProperties properties) { + private void setTimeoutForRetryCache(RaftProperties properties) { TimeUnit timeUnit; long duration; timeUnit = @@ -314,8 +327,7 @@ private void setTimeoutForRetryCache(Configuration conf, .setExpiryTime(properties, retryCacheTimeout); } - private void setServerRequestTimeout(Configuration conf, - RaftProperties properties) { + private void setServerRequestTimeout(RaftProperties properties) { TimeUnit timeUnit; long duration; timeUnit = OzoneConfigKeys.DFS_RATIS_SERVER_REQUEST_TIMEOUT_DURATION_DEFAULT @@ -337,8 +349,7 @@ private int setMaxWriteBuffer(RaftProperties properties) { return maxChunkSize; } - private int setRaftSegmentPreallocatedSize(Configuration conf, - RaftProperties properties) { + private int setRaftSegmentPreallocatedSize(RaftProperties properties) { final int raftSegmentPreallocatedSize = (int) conf.getStorageSize( OzoneConfigKeys.DFS_CONTAINER_RATIS_SEGMENT_PREALLOCATED_SIZE_KEY, OzoneConfigKeys.DFS_CONTAINER_RATIS_SEGMENT_PREALLOCATED_SIZE_DEFAULT, @@ -361,8 +372,7 @@ private int setRaftSegmentPreallocatedSize(Configuration conf, return raftSegmentPreallocatedSize; } - private void setRaftSegmentSize(Configuration conf, - RaftProperties properties) { + private void setRaftSegmentSize(RaftProperties properties) { final int raftSegmentSize = (int)conf.getStorageSize( OzoneConfigKeys.DFS_CONTAINER_RATIS_SEGMENT_SIZE_KEY, OzoneConfigKeys.DFS_CONTAINER_RATIS_SEGMENT_SIZE_DEFAULT, @@ -371,7 +381,7 @@ private void setRaftSegmentSize(Configuration conf, SizeInBytes.valueOf(raftSegmentSize)); } - private RpcType setRpcType(Configuration conf, RaftProperties properties) { + private RpcType setRpcType(RaftProperties properties) { final String rpcType = conf.get( OzoneConfigKeys.DFS_CONTAINER_RATIS_RPC_TYPE_KEY, OzoneConfigKeys.DFS_CONTAINER_RATIS_RPC_TYPE_DEFAULT); @@ -381,9 +391,9 @@ private RpcType setRpcType(Configuration conf, RaftProperties properties) { } public static XceiverServerRatis newXceiverServerRatis( - DatanodeDetails datanodeDetails, Configuration ozoneConf, - ContainerDispatcher dispatcher, StateContext context, - CertificateClient caClient) throws IOException { + DatanodeDetails datanodeDetails, OzoneConfiguration ozoneConf, + ContainerDispatcher dispatcher, ContainerController containerController, + CertificateClient caClient, StateContext context) throws IOException { int localPort = ozoneConf.getInt( OzoneConfigKeys.DFS_CONTAINER_RATIS_IPC_PORT, OzoneConfigKeys.DFS_CONTAINER_RATIS_IPC_PORT_DEFAULT); @@ -393,23 +403,13 @@ public static XceiverServerRatis newXceiverServerRatis( if (ozoneConf.getBoolean(OzoneConfigKeys .DFS_CONTAINER_RATIS_IPC_RANDOM_PORT, OzoneConfigKeys.DFS_CONTAINER_RATIS_IPC_RANDOM_PORT_DEFAULT)) { - try (ServerSocket socket = new ServerSocket()) { - socket.setReuseAddress(true); - SocketAddress address = new InetSocketAddress(0); - socket.bind(address); - localPort = socket.getLocalPort(); - LOG.info("Found a free port for the server : {}", localPort); - } catch (IOException e) { - LOG.error("Unable find a random free port for the server, " - + "fallback to use default port {}", localPort, e); - } + localPort = 0; } - GrpcTlsConfig tlsConfig = RatisHelper.createTlsServerConfig( - new SecurityConfig(ozoneConf)); - datanodeDetails.setPort( - DatanodeDetails.newPort(DatanodeDetails.Port.Name.RATIS, localPort)); - return new XceiverServerRatis(datanodeDetails, localPort, - dispatcher, ozoneConf, context, tlsConfig, caClient); + GrpcTlsConfig tlsConfig = RatisHelper.createTlsServerConfigForDN( + new SecurityConfig(ozoneConf), caClient); + + return new XceiverServerRatis(datanodeDetails, localPort, dispatcher, + containerController, context, tlsConfig, caClient, ozoneConf); } @Override @@ -419,6 +419,22 @@ public void start() throws IOException { server.getId(), getIPCPort()); chunkExecutor.prestartAllCoreThreads(); server.start(); + + int realPort = + ((RaftServerProxy) server).getServerRpc().getInetSocketAddress() + .getPort(); + + if (port == 0) { + LOG.info("{} {} is started using port {}", getClass().getSimpleName(), + server.getId(), realPort); + port = realPort; + } + + //register the real port to the datanode details. + datanodeDetails.setPort(DatanodeDetails + .newPort(DatanodeDetails.Port.Name.RATIS, + realPort)); + isStarted = true; } } @@ -431,7 +447,6 @@ public void stop() { // some of the tasks would be executed using the executors. server.close(); chunkExecutor.shutdown(); - executors.forEach(ExecutorService::shutdown); isStarted = false; } catch (IOException e) { throw new RuntimeException(e); @@ -502,8 +517,15 @@ private RaftClientRequest createRaftClientRequest( RaftClientRequest.Type type) { return new RaftClientRequest(clientId, server.getId(), RaftGroupId.valueOf(PipelineID.getFromProtobuf(pipelineID).getId()), - nextCallId(), Message.valueOf(request.toByteString()), type, - null); + nextCallId(), ContainerCommandRequestMessage.toMessage(request, null), + type, null); + } + + private GroupInfoRequest createGroupInfoRequest( + HddsProtos.PipelineID pipelineID) { + return new GroupInfoRequest(clientId, server.getId(), + RaftGroupId.valueOf(PipelineID.getFromProtobuf(pipelineID).getId()), + nextCallId()); } private void handlePipelineFailure(RaftGroupId groupId, @@ -535,32 +557,37 @@ private void handlePipelineFailure(RaftGroupId groupId, + roleInfoProto.getRole()); } + triggerPipelineClose(groupId, msg, + ClosePipelineInfo.Reason.PIPELINE_FAILED, false); + } + + private void triggerPipelineClose(RaftGroupId groupId, String detail, + ClosePipelineInfo.Reason reasonCode, boolean triggerHB) { PipelineID pipelineID = PipelineID.valueOf(groupId.getUuid()); ClosePipelineInfo.Builder closePipelineInfo = ClosePipelineInfo.newBuilder() .setPipelineID(pipelineID.getProtobuf()) - .setReason(ClosePipelineInfo.Reason.PIPELINE_FAILED) - .setDetailedReason(msg); + .setReason(reasonCode) + .setDetailedReason(detail); PipelineAction action = PipelineAction.newBuilder() .setClosePipeline(closePipelineInfo) .setAction(PipelineAction.Action.CLOSE) .build(); context.addPipelineActionIfAbsent(action); - LOG.debug( + // wait for the next HB timeout or right away? + if (triggerHB) { + context.getParent().triggerHeartbeat(); + } + LOG.error( "pipeline Action " + action.getAction() + " on pipeline " + pipelineID + ".Reason : " + action.getClosePipeline().getDetailedReason()); } @Override public boolean isExist(HddsProtos.PipelineID pipelineId) { - for (RaftGroupId groupId : server.getGroupIds()) { - if (PipelineID.valueOf(groupId.getUuid()).getProtobuf() - .equals(pipelineId)) { - return true; - } - } - return false; + return raftGids.contains( + RaftGroupId.valueOf(PipelineID.getFromProtobuf(pipelineId).getId())); } @Override @@ -590,11 +617,73 @@ public List getPipelineIds() { return pipelineIDs; } - void handleNodeSlowness(RaftGroup group, RoleInfoProto roleInfoProto) { - handlePipelineFailure(group.getGroupId(), roleInfoProto); + void handleNodeSlowness(RaftGroupId groupId, RoleInfoProto roleInfoProto) { + handlePipelineFailure(groupId, roleInfoProto); + } + + void handleNoLeader(RaftGroupId groupId, RoleInfoProto roleInfoProto) { + handlePipelineFailure(groupId, roleInfoProto); + } + + void handleApplyTransactionFailure(RaftGroupId groupId, + RaftProtos.RaftPeerRole role) { + UUID dnId = RatisHelper.toDatanodeId(getServer().getId()); + String msg = + "Ratis Transaction failure in datanode " + dnId + " with role " + role + + " .Triggering pipeline close action."; + triggerPipelineClose(groupId, msg, + ClosePipelineInfo.Reason.STATEMACHINE_TRANSACTION_FAILED, true); + } + /** + * The fact that the snapshot contents cannot be used to actually catch up + * the follower, it is the reason to initiate close pipeline and + * not install the snapshot. The follower will basically never be able to + * catch up. + * + * @param groupId raft group information + * @param roleInfoProto information about the current node role and + * rpc delay information. + * @param firstTermIndexInLog After the snapshot installation is complete, + * return the last included term index in the snapshot. + */ + void handleInstallSnapshotFromLeader(RaftGroupId groupId, + RoleInfoProto roleInfoProto, + TermIndex firstTermIndexInLog) { + LOG.warn("Install snapshot notification received from Leader with " + + "termIndex: {}, terminating pipeline: {}", + firstTermIndexInLog, groupId); + handlePipelineFailure(groupId, roleInfoProto); + } + + /** + * Notify the Datanode Ratis endpoint of Ratis log failure. + * Expected to be invoked from the Container StateMachine + * @param groupId the Ratis group/pipeline for which log has failed + * @param t exception encountered at the time of the failure + * + */ + @VisibleForTesting + public void handleNodeLogFailure(RaftGroupId groupId, Throwable t) { + String msg = (t == null) ? "Unspecified failure reported in Ratis log" + : t.getMessage(); + + triggerPipelineClose(groupId, msg, + ClosePipelineInfo.Reason.PIPELINE_LOG_FAILED, true); + } + + public long getMinReplicatedIndex(PipelineID pipelineID) throws IOException { + Long minIndex; + GroupInfoReply reply = getServer() + .getGroupInfo(createGroupInfoRequest(pipelineID.getProtobuf())); + minIndex = RatisHelper.getMinReplicatedIndex(reply.getCommitInfos()); + return minIndex == null ? -1 : minIndex.longValue(); + } + + void notifyGroupRemove(RaftGroupId gid) { + raftGids.remove(gid); } - void handleNoLeader(RaftGroup group, RoleInfoProto roleInfoProto) { - handlePipelineFailure(group.getGroupId(), roleInfoProto); + void notifyGroupAdd(RaftGroupId gid) { + raftGids.add(gid); } -} \ No newline at end of file +} diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerCache.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerCache.java index 25d1bdf291817..4ddb4e48792fb 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerCache.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerCache.java @@ -23,8 +23,8 @@ import org.apache.commons.collections.map.LRUMap; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.ozone.OzoneConfigKeys; -import org.apache.hadoop.utils.MetadataStore; -import org.apache.hadoop.utils.MetadataStoreBuilder; +import org.apache.hadoop.hdds.utils.MetadataStore; +import org.apache.hadoop.hdds.utils.MetadataStoreBuilder; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -66,22 +66,6 @@ public synchronized static ContainerCache getInstance(Configuration conf) { return cache; } - /** - * Closes a db instance. - * - * @param containerPath - path of the container db to be closed. - * @param db - db instance to close. - */ - private void closeDB(String containerPath, MetadataStore db) { - if (db != null) { - try { - db.close(); - } catch (Exception e) { - LOG.error("Error closing DB. Container: " + containerPath, e); - } - } - } - /** * Closes all the db instances and resets the cache. */ @@ -92,8 +76,9 @@ public void shutdownCache() { MapIterator iterator = cache.mapIterator(); while (iterator.hasNext()) { iterator.next(); - MetadataStore db = (MetadataStore) iterator.getValue(); - closeDB((String)iterator.getKey(), db); + ReferenceCountedDB db = (ReferenceCountedDB) iterator.getValue(); + Preconditions.checkArgument(db.cleanup(), "refCount:", + db.getReferenceCount()); } // reset the cache cache.clear(); @@ -107,15 +92,10 @@ public void shutdownCache() { */ @Override protected boolean removeLRU(LinkEntry entry) { - MetadataStore db = (MetadataStore) entry.getValue(); - String dbFile = (String)entry.getKey(); + ReferenceCountedDB db = (ReferenceCountedDB) entry.getValue(); lock.lock(); try { - closeDB(dbFile, db); - return true; - } catch (Exception e) { - LOG.error("Eviction for db:{} failed", dbFile, e); - return false; + return db.cleanup(); } finally { lock.unlock(); } @@ -128,26 +108,30 @@ protected boolean removeLRU(LinkEntry entry) { * @param containerDBType - DB type of the container. * @param containerDBPath - DB path of the container. * @param conf - Hadoop Configuration. - * @return MetadataStore. + * @return ReferenceCountedDB. */ - public MetadataStore getDB(long containerID, String containerDBType, + public ReferenceCountedDB getDB(long containerID, String containerDBType, String containerDBPath, Configuration conf) throws IOException { Preconditions.checkState(containerID >= 0, "Container ID cannot be negative."); lock.lock(); try { - MetadataStore db = (MetadataStore) this.get(containerDBPath); + ReferenceCountedDB db = (ReferenceCountedDB) this.get(containerDBPath); if (db == null) { - db = MetadataStoreBuilder.newBuilder() + MetadataStore metadataStore = + MetadataStoreBuilder.newBuilder() .setDbFile(new File(containerDBPath)) .setCreateIfMissing(false) .setConf(conf) .setDBType(containerDBType) .build(); + db = new ReferenceCountedDB(metadataStore, containerDBPath); this.put(containerDBPath, db); } + // increment the reference before returning the object + db.incrementReference(); return db; } catch (Exception e) { LOG.error("Error opening DB. Container:{} ContainerPath:{}", @@ -161,14 +145,17 @@ public MetadataStore getDB(long containerID, String containerDBType, /** * Remove a DB handler from cache. * - * @param containerPath - path of the container db file. + * @param containerDBPath - path of the container db file. */ - public void removeDB(String containerPath) { + public void removeDB(String containerDBPath) { lock.lock(); try { - MetadataStore db = (MetadataStore)this.get(containerPath); - closeDB(containerPath, db); - this.remove(containerPath); + ReferenceCountedDB db = (ReferenceCountedDB)this.get(containerDBPath); + if (db != null) { + Preconditions.checkArgument(db.cleanup(), "refCount:", + db.getReferenceCount()); + } + this.remove(containerDBPath); } finally { lock.unlock(); } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ReferenceCountedDB.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ReferenceCountedDB.java new file mode 100644 index 0000000000000..fb143a407f7f4 --- /dev/null +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ReferenceCountedDB.java @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.container.common.utils; + +import com.google.common.base.Preconditions; + +import org.apache.commons.lang.exception.ExceptionUtils; +import org.apache.hadoop.hdds.utils.MetadataStore; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.Closeable; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Class to implement reference counting over instances handed by Container + * Cache. + * Enable DEBUG log below will enable us quickly locate the leaked reference + * from caller stack. When JDK9 StackWalker is available, we can switch to + * StackWalker instead of new Exception().printStackTrace(). + */ +public class ReferenceCountedDB implements Closeable { + private static final Logger LOG = + LoggerFactory.getLogger(ReferenceCountedDB.class); + private final AtomicInteger referenceCount; + private final MetadataStore store; + private final String containerDBPath; + + public ReferenceCountedDB(MetadataStore store, String containerDBPath) { + this.referenceCount = new AtomicInteger(0); + this.store = store; + this.containerDBPath = containerDBPath; + } + + public long getReferenceCount() { + return referenceCount.get(); + } + + public void incrementReference() { + this.referenceCount.incrementAndGet(); + if (LOG.isTraceEnabled()) { + LOG.trace("IncRef {} to refCnt {}, stackTrace: {}", containerDBPath, + referenceCount.get(), ExceptionUtils.getStackTrace(new Throwable())); + } + } + + public void decrementReference() { + int refCount = this.referenceCount.decrementAndGet(); + Preconditions.checkArgument(refCount >= 0, "refCount:", refCount); + if (LOG.isTraceEnabled()) { + LOG.trace("DecRef {} to refCnt {}, stackTrace: {}", containerDBPath, + referenceCount.get(), ExceptionUtils.getStackTrace(new Throwable())); + } + } + + public boolean cleanup() { + if (referenceCount.get() == 0 && store != null) { + if (LOG.isDebugEnabled()) { + LOG.debug("Close {} refCnt {}", containerDBPath, + referenceCount.get()); + } + try { + store.close(); + return true; + } catch (Exception e) { + LOG.error("Error closing DB. Container: " + containerDBPath, e); + return false; + } + } else { + return false; + } + } + + public MetadataStore getStore() { + return store; + } + + public void close() { + decrementReference(); + } +} \ No newline at end of file diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/AsyncChecker.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/AsyncChecker.java new file mode 100644 index 0000000000000..f7391e3cca00d --- /dev/null +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/AsyncChecker.java @@ -0,0 +1,65 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

    + * http://www.apache.org/licenses/LICENSE-2.0 + *

    + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.container.common.volume; + +import com.google.common.util.concurrent.ListenableFuture; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.hdfs.server.datanode.checker.Checkable; + +import java.util.Optional; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.TimeUnit; + +/** + * A class that can be used to schedule an asynchronous check on a given + * {@link Checkable}. If the check is successfully scheduled then a + * {@link ListenableFuture} is returned. + * + */ +@InterfaceAudience.Private +@InterfaceStability.Unstable +public interface AsyncChecker { + + /** + * Schedule an asynchronous check for the given object. + * + * @param target object to be checked. + * + * @param context the interpretation of the context depends on the + * target. + * + * @return returns a {@link Optional of ListenableFuture} that can be used to + * retrieve the result of the asynchronous check. + */ + Optional> schedule(Checkable target, K context); + + /** + * Cancel all executing checks and wait for them to complete. + * First attempts a graceful cancellation, then cancels forcefully. + * Waits for the supplied timeout after both attempts. + * + * See {@link ExecutorService#awaitTermination} for a description of + * the parameters. + * + * @throws InterruptedException + */ + void shutdownAndWait(long timeout, TimeUnit timeUnit) + throws InterruptedException; +} diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java index 3a2034580ad0d..3e89f9031389c 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java @@ -18,9 +18,10 @@ package org.apache.hadoop.ozone.container.common.volume; +import javax.annotation.Nullable; + import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; -import com.sun.istack.Nullable; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.GetSpaceUsed; import org.apache.hadoop.fs.StorageType; @@ -203,7 +204,7 @@ private void initialize() throws IOException { switch (intialVolumeState) { case NON_EXISTENT: // Root directory does not exist. Create it. - if (!hddsRootDir.mkdir()) { + if (!hddsRootDir.mkdirs()) { throw new IOException("Cannot create directory " + hddsRootDir); } setState(VolumeState.NOT_FORMATTED); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolumeChecker.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolumeChecker.java index cd08cd2bf497c..800789f6e0e73 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolumeChecker.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolumeChecker.java @@ -19,7 +19,6 @@ package org.apache.hadoop.ozone.container.common.volume; import com.google.common.annotations.VisibleForTesting; -import com.google.common.base.Optional; import com.google.common.collect.Sets; import com.google.common.util.concurrent.FutureCallback; import com.google.common.util.concurrent.Futures; @@ -28,7 +27,6 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdfs.DFSConfigKeys; import org.apache.hadoop.hdfs.server.datanode.DataNode; -import org.apache.hadoop.hdfs.server.datanode.checker.AsyncChecker; import org.apache.hadoop.hdfs.server.datanode.checker.VolumeCheckResult; import org.apache.hadoop.util.DiskChecker.DiskErrorException; import org.apache.hadoop.util.Timer; @@ -43,6 +41,7 @@ import java.util.Collections; import java.util.HashSet; import java.util.Set; +import java.util.Optional; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutionException; import java.util.concurrent.ExecutorService; @@ -182,10 +181,12 @@ public Set checkAllVolumes(Collection volumes) final long gap = timer.monotonicNow() - lastAllVolumesCheck; if (gap < minDiskCheckGapMs) { numSkippedChecks.incrementAndGet(); - LOG.trace( - "Skipped checking all volumes, time since last check {} is less " + - "than the minimum gap between checks ({} ms).", - gap, minDiskCheckGapMs); + if (LOG.isTraceEnabled()) { + LOG.trace( + "Skipped checking all volumes, time since last check {} is less " + + "than the minimum gap between checks ({} ms).", + gap, minDiskCheckGapMs); + } return Collections.emptySet(); } @@ -315,7 +316,9 @@ public void onSuccess(@Nonnull VolumeCheckResult result) { switch (result) { case HEALTHY: case DEGRADED: - LOG.debug("Volume {} is {}.", volume, result); + if (LOG.isDebugEnabled()) { + LOG.debug("Volume {} is {}.", volume, result); + } markHealthy(); break; case FAILED: diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/RoundRobinVolumeChoosingPolicy.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/RoundRobinVolumeChoosingPolicy.java index 75c92ec024bb4..f503149aca438 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/RoundRobinVolumeChoosingPolicy.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/RoundRobinVolumeChoosingPolicy.java @@ -58,7 +58,9 @@ public HddsVolume chooseVolume(List volumes, while (true) { final HddsVolume volume = volumes.get(currentVolumeIndex); - long availableVolumeSize = volume.getAvailable(); + // adjust for remaining capacity in Open containers + long availableVolumeSize = volume.getAvailable() + - volume.getCommittedBytes(); currentVolumeIndex = (currentVolumeIndex + 1) % volumes.size(); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/ThrottledAsyncChecker.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/ThrottledAsyncChecker.java index d1b256947d76c..836fdf3e39518 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/ThrottledAsyncChecker.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/ThrottledAsyncChecker.java @@ -18,7 +18,6 @@ package org.apache.hadoop.ozone.container.common.volume; -import com.google.common.base.Optional; import com.google.common.util.concurrent.FutureCallback; import com.google.common.util.concurrent.Futures; import com.google.common.util.concurrent.ListenableFuture; @@ -26,7 +25,6 @@ import com.google.common.util.concurrent.MoreExecutors; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; -import org.apache.hadoop.hdfs.server.datanode.checker.AsyncChecker; import org.apache.hadoop.hdfs.server.datanode.checker.Checkable; import org.apache.hadoop.util.Timer; @@ -37,6 +35,7 @@ import javax.annotation.Nullable; import java.util.HashMap; import java.util.Map; +import java.util.Optional; import java.util.WeakHashMap; import java.util.concurrent.ExecutorService; import java.util.concurrent.ScheduledExecutorService; @@ -122,7 +121,7 @@ public ThrottledAsyncChecker(final Timer timer, public Optional> schedule( Checkable target, K context) { if (checksInProgress.containsKey(target)) { - return Optional.absent(); + return Optional.empty(); } if (completedChecks.containsKey(target)) { @@ -130,10 +129,12 @@ public Optional> schedule( completedChecks.get(target); final long msSinceLastCheck = timer.monotonicNow() - result.completedAt; if (msSinceLastCheck < minMsBetweenChecks) { - LOG.debug("Skipped checking {}. Time since last check {}ms " + - "is less than the min gap {}ms.", - target, msSinceLastCheck, minMsBetweenChecks); - return Optional.absent(); + if (LOG.isDebugEnabled()) { + LOG.debug("Skipped checking {}. Time since last check {}ms " + + "is less than the min gap {}ms.", + target, msSinceLastCheck, minMsBetweenChecks); + } + return Optional.empty(); } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueBlockIterator.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueBlockIterator.java index 535af29c190c9..ad68c4dc96cd8 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueBlockIterator.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueBlockIterator.java @@ -28,14 +28,15 @@ import org.apache.hadoop.ozone.container.common.interfaces.BlockIterator; import org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils; import org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerLocationUtil; -import org.apache.hadoop.utils.MetaStoreIterator; -import org.apache.hadoop.utils.MetadataKeyFilters; -import org.apache.hadoop.utils.MetadataKeyFilters.KeyPrefixFilter; -import org.apache.hadoop.utils.MetadataStore; -import org.apache.hadoop.utils.MetadataStore.KeyValue; +import org.apache.hadoop.hdds.utils.MetaStoreIterator; +import org.apache.hadoop.hdds.utils.MetadataKeyFilters; +import org.apache.hadoop.hdds.utils.MetadataKeyFilters.KeyPrefixFilter; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; +import org.apache.hadoop.hdds.utils.MetadataStore.KeyValue; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.io.Closeable; import java.io.File; import java.io.IOException; import java.util.NoSuchElementException; @@ -48,12 +49,14 @@ * {@link MetadataKeyFilters#getNormalKeyFilter()} */ @InterfaceAudience.Public -public class KeyValueBlockIterator implements BlockIterator { +public class KeyValueBlockIterator implements BlockIterator, + Closeable { private static final Logger LOG = LoggerFactory.getLogger( KeyValueBlockIterator.class); private MetaStoreIterator blockIterator; + private final ReferenceCountedDB db; private static KeyPrefixFilter defaultBlockFilter = MetadataKeyFilters .getNormalKeyFilter(); private KeyPrefixFilter blockFilter; @@ -91,9 +94,9 @@ public KeyValueBlockIterator(long id, File path, KeyPrefixFilter filter) containerData; keyValueContainerData.setDbFile(KeyValueContainerLocationUtil .getContainerDBFile(metdataPath, containerId)); - MetadataStore metadataStore = BlockUtils.getDB(keyValueContainerData, new + db = BlockUtils.getDB(keyValueContainerData, new OzoneConfiguration()); - blockIterator = metadataStore.iterator(); + blockIterator = db.getStore().iterator(); blockFilter = filter; } @@ -125,8 +128,10 @@ public boolean hasNext() throws IOException { KeyValue block = blockIterator.next(); if (blockFilter.filterKey(null, block.getKey(), null)) { nextBlock = BlockUtils.getBlockData(block.getValue()); - LOG.trace("Block matching with filter found: blockID is : {} for " + - "containerID {}", nextBlock.getLocalID(), containerId); + if (LOG.isTraceEnabled()) { + LOG.trace("Block matching with filter found: blockID is : {} for " + + "containerID {}", nextBlock.getLocalID(), containerId); + } return true; } hasNext(); @@ -145,4 +150,8 @@ public void seekToLast() { nextBlock = null; blockIterator.seekToLast(); } + + public void close() { + db.close(); + } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java index 26b0ce1d788b2..a6e914b90b83d 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java @@ -39,6 +39,8 @@ .StorageContainerDatanodeProtocolProtos.ContainerReplicaProto; import org.apache.hadoop.hdds.scm.container.common.helpers .StorageContainerException; +import org.apache.hadoop.hdfs.util.Canceler; +import org.apache.hadoop.hdfs.util.DataTransferThrottler; import org.apache.hadoop.io.nativeio.NativeIO; import org.apache.hadoop.ozone.OzoneConfigKeys; import org.apache.hadoop.ozone.OzoneConsts; @@ -54,7 +56,6 @@ .KeyValueContainerLocationUtil; import org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil; import org.apache.hadoop.util.DiskChecker.DiskOutOfSpaceException; -import org.apache.hadoop.utils.MetadataStore; import com.google.common.base.Preconditions; import org.apache.commons.io.FileUtils; @@ -69,16 +70,21 @@ .Result.DISK_OUT_OF_SPACE; import static org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos .Result.ERROR_IN_COMPACT_DB; +import static org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos + .Result.ERROR_IN_DB_SYNC; import static org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos .Result.INVALID_CONTAINER_STATE; import static org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos .Result.UNSUPPORTED_REQUEST; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import org.slf4j.Logger; import org.slf4j.LoggerFactory; /** - * Class to perform KeyValue Container operations. + * Class to perform KeyValue Container operations. Any modifications to + * KeyValueContainer object should ideally be done via api exposed in + * KeyValueHandler class. */ public class KeyValueContainer implements Container { @@ -298,8 +304,14 @@ public void markContainerUnhealthy() throws StorageContainerException { @Override public void quasiClose() throws StorageContainerException { + // The DB must be synced during close operation + flushAndSyncDB(); + writeLock(); try { + // Second sync should be a very light operation as sync has already + // been done outside the lock. + flushAndSyncDB(); updateContainerData(containerData::quasiCloseContainer); } finally { writeUnlock(); @@ -308,16 +320,21 @@ public void quasiClose() throws StorageContainerException { @Override public void close() throws StorageContainerException { + // The DB must be synced during close operation + flushAndSyncDB(); + writeLock(); try { + // Second sync should be a very light operation as sync has already + // been done outside the lock. + flushAndSyncDB(); updateContainerData(containerData::closeContainer); } finally { writeUnlock(); } - - // It is ok if this operation takes a bit of time. - // Close container is not expected to be instantaneous. - compactDB(); + LOG.info("Container {} is closed with bcsId {}.", + containerData.getContainerID(), + containerData.getBlockCommitSequenceId()); } /** @@ -339,21 +356,21 @@ private void updateContainerData(Runnable update) updateContainerFile(containerFile); } catch (StorageContainerException ex) { - if (oldState != null) { - // Failed to update .container file. Reset the state to CLOSING + if (oldState != null + && containerData.getState() != ContainerDataProto.State.UNHEALTHY) { + // Failed to update .container file. Reset the state to old state only + // if the current state is not unhealthy. containerData.setState(oldState); } throw ex; } } - void compactDB() throws StorageContainerException { + private void compactDB() throws StorageContainerException { try { - MetadataStore db = BlockUtils.getDB(containerData, config); - db.compactDB(); - LOG.info("Container {} is closed with bcsId {}.", - containerData.getContainerID(), - containerData.getBlockCommitSequenceId()); + try(ReferenceCountedDB db = BlockUtils.getDB(containerData, config)) { + db.getStore().compactDB(); + } } catch (StorageContainerException ex) { throw ex; } catch (IOException ex) { @@ -362,6 +379,22 @@ void compactDB() throws StorageContainerException { } } + private void flushAndSyncDB() throws StorageContainerException { + try { + try (ReferenceCountedDB db = BlockUtils.getDB(containerData, config)) { + db.getStore().flushDB(true); + LOG.info("Container {} is synced with bcsId {}.", + containerData.getContainerID(), + containerData.getBlockCommitSequenceId()); + } + } catch (StorageContainerException ex) { + throw ex; + } catch (IOException ex) { + LOG.error("Error in DB sync while closing container", ex); + throw new StorageContainerException(ex, ERROR_IN_DB_SYNC); + } + } + @Override public KeyValueContainerData getContainerData() { return containerData; @@ -493,6 +526,7 @@ public void exportContainerData(OutputStream destination, "Only closed containers could be exported: ContainerId=" + getContainerData().getContainerID()); } + compactDB(); packer.pack(this, destination); } @@ -522,6 +556,8 @@ public boolean hasReadLock() { * Acquire write lock. */ public void writeLock() { + // TODO: The lock for KeyValueContainer object should not be exposed + // publicly. this.lock.writeLock().lock(); } @@ -579,6 +615,11 @@ public void updateBlockCommitSequenceId(long blockCommitSequenceId) { containerData.updateBlockCommitSequenceId(blockCommitSequenceId); } + @Override + public long getBlockCommitSequenceId() { + return containerData.getBlockCommitSequenceId(); + } + /** * Returns KeyValueContainerReport for the KeyValueContainer. @@ -642,60 +683,33 @@ public File getContainerDBFile() { .getContainerID() + OzoneConsts.DN_CONTAINER_DB); } - /** - * run integrity checks on the Container metadata. - */ - public void check() throws StorageContainerException { - ContainerCheckLevel level = ContainerCheckLevel.NO_CHECK; + public boolean scanMetaData() { long containerId = containerData.getContainerID(); + KeyValueContainerCheck checker = + new KeyValueContainerCheck(containerData.getMetadataPath(), config, + containerId); + return checker.fastCheck(); + } - switch (containerData.getState()) { - case OPEN: - level = ContainerCheckLevel.FAST_CHECK; - LOG.info("Doing Fast integrity checks for Container ID : {}," - + " because it is OPEN", containerId); - break; - case CLOSING: - level = ContainerCheckLevel.FAST_CHECK; - LOG.info("Doing Fast integrity checks for Container ID : {}," - + " because it is CLOSING", containerId); - break; - case CLOSED: - case QUASI_CLOSED: - level = ContainerCheckLevel.FULL_CHECK; - LOG.debug("Doing Full integrity checks for Container ID : {}," - + " because it is in {} state", containerId, - containerData.getState()); - break; - default: - throw new StorageContainerException( - "Invalid Container state found for Container : " + containerData - .getContainerID(), INVALID_CONTAINER_STATE); - } + @Override + public boolean shouldScanData() { + return containerData.getState() == ContainerDataProto.State.CLOSED + || containerData.getState() == ContainerDataProto.State.QUASI_CLOSED; + } - if (level == ContainerCheckLevel.NO_CHECK) { - LOG.debug("Skipping integrity checks for Container Id : {}", containerId); - return; + public boolean scanData(DataTransferThrottler throttler, Canceler canceler) { + if (!shouldScanData()) { + throw new IllegalStateException("The checksum verification can not be" + + " done for container in state " + + containerData.getState()); } + long containerId = containerData.getContainerID(); KeyValueContainerCheck checker = new KeyValueContainerCheck(containerData.getMetadataPath(), config, containerId); - switch (level) { - case FAST_CHECK: - checker.fastCheck(); - break; - case FULL_CHECK: - checker.fullCheck(); - break; - case NO_CHECK: - LOG.debug("Skipping integrity checks for Container Id : {}", containerId); - break; - default: - // we should not be here at all, scuttle the ship! - Preconditions.checkNotNull(0, "Invalid Containercheck level"); - } + return checker.fullCheck(throttler, canceler); } private enum ContainerCheckLevel { diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java index bdfdf21b4c250..a4bd37623113f 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java @@ -22,6 +22,11 @@ import com.google.common.primitives.Longs; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; +import org.apache.hadoop.hdfs.util.Canceler; +import org.apache.hadoop.hdfs.util.DataTransferThrottler; +import org.apache.hadoop.ozone.common.Checksum; +import org.apache.hadoop.ozone.common.ChecksumData; +import org.apache.hadoop.ozone.common.OzoneChecksumException; import org.apache.hadoop.ozone.container.common.helpers.BlockData; import org.apache.hadoop.ozone.container.common.helpers.ChunkInfo; import org.apache.hadoop.ozone.container.common.helpers.ContainerUtils; @@ -30,12 +35,15 @@ import org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils; import org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils; import org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerLocationUtil; -import org.apache.hadoop.utils.MetadataStore; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import java.io.File; +import java.io.FileInputStream; import java.io.IOException; -import java.util.List; +import java.io.InputStream; +import java.util.Arrays; +import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -72,37 +80,22 @@ public KeyValueContainerCheck(String metadataPath, Configuration conf, * These checks do not look inside the metadata files. * Applicable for OPEN containers. * - * @return true : corruption detected, false : no corruption. + * @return true : integrity checks pass, false : otherwise. */ public boolean fastCheck() { - boolean corruption = false; + LOG.info("Running basic checks for container {};", containerID); + boolean valid = false; try { - basicChecks(); + loadContainerData(); + checkLayout(); + checkContainerFile(); + valid = true; } catch (IOException e) { handleCorruption(e); - corruption = true; } - return corruption; - } - - /** - * Checks : - * 1. check directory layout - * 2. check container file - * - * @return void - */ - - private void basicChecks() throws IOException { - - LOG.trace("Running basic checks for container {};", containerID); - - loadContainerData(); - - checkLayout(); - checkContainerFile(); + return valid; } /** @@ -114,21 +107,22 @@ private void basicChecks() throws IOException { *

    * fullCheck is a superset of fastCheck * - * @return true : corruption detected, false : no corruption. + * @return true : integrity checks pass, false : otherwise. */ - public boolean fullCheck() { - boolean corruption = false; + public boolean fullCheck(DataTransferThrottler throttler, Canceler canceler) { + boolean valid; try { - basicChecks(); - checkBlockDB(); - + valid = fastCheck(); + if (valid) { + scanData(throttler, canceler); + } } catch (IOException e) { handleCorruption(e); - corruption = true; + valid = false; } - return corruption; + return valid; } /** @@ -147,7 +141,7 @@ private void checkLayout() throws IOException { private void checkDirPath(String path) throws IOException { File dirPath = new File(path); - String errStr = null; + String errStr; try { if (!dirPath.isDirectory()) { @@ -168,7 +162,7 @@ private void checkDirPath(String path) throws IOException { } private void checkContainerFile() throws IOException { - /** + /* * compare the values in the container file loaded from disk, * with the values we are expecting */ @@ -199,25 +193,23 @@ private void checkContainerFile() throws IOException { } KeyValueContainerData kvData = onDiskContainerData; - if (!metadataPath.toString().equals(kvData.getMetadataPath())) { + if (!metadataPath.equals(kvData.getMetadataPath())) { String errStr = "Bad metadata path in Containerdata for " + containerID + "Expected [" - + metadataPath.toString() + "] Got [" + kvData.getMetadataPath() + + metadataPath + "] Got [" + kvData.getMetadataPath() + "]"; throw new IOException(errStr); } } - private void checkBlockDB() throws IOException { - /** + private void scanData(DataTransferThrottler throttler, Canceler canceler) + throws IOException { + /* * Check the integrity of the DB inside each container. - * In Scope: * 1. iterate over each key (Block) and locate the chunks for the block - * 2. garbage detection : chunks which exist in the filesystem, - * but not in the DB. This function is implemented as HDDS-1202 - * Not in scope: - * 1. chunk checksum verification. this is left to a separate - * slow chunk scanner + * 2. garbage detection (TBD): chunks which exist in the filesystem, + * but not in the DB. This function will be implemented in HDDS-1202 + * 3. chunk checksum verification. */ Preconditions.checkState(onDiskContainerData != null, "invoke loadContainerData prior to calling this function"); @@ -234,43 +226,67 @@ private void checkBlockDB() throws IOException { throw new IOException(dbFileErrorMsg); } - onDiskContainerData.setDbFile(dbFile); - MetadataStore db = BlockUtils - .getDB(onDiskContainerData, checkConfig); - - iterateBlockDB(db); - } - - private void iterateBlockDB(MetadataStore db) - throws IOException { - Preconditions.checkState(db != null); - - // get "normal" keys from the Block DB - KeyValueBlockIterator kvIter = new KeyValueBlockIterator(containerID, - new File(onDiskContainerData.getContainerPath())); - - // ensure there is a chunk file for each key in the DB - while (kvIter.hasNext()) { - BlockData block = kvIter.nextBlock(); - - List chunkInfoList = block.getChunks(); - for (ContainerProtos.ChunkInfo chunk : chunkInfoList) { - File chunkFile; - chunkFile = ChunkUtils.getChunkFile(onDiskContainerData, - ChunkInfo.getFromProtoBuf(chunk)); - - if (!chunkFile.exists()) { - // concurrent mutation in Block DB? lookup the block again. - byte[] bdata = db.get( - Longs.toByteArray(block.getBlockID().getLocalID())); - if (bdata == null) { - LOG.trace("concurrency with delete, ignoring deleted block"); - break; // skip to next block from kvIter - } else { - String errorStr = "Missing chunk file " - + chunkFile.getAbsolutePath(); - throw new IOException(errorStr); + try(ReferenceCountedDB db = + BlockUtils.getDB(onDiskContainerData, checkConfig); + KeyValueBlockIterator kvIter = new KeyValueBlockIterator(containerID, + new File(onDiskContainerData.getContainerPath()))) { + + while(kvIter.hasNext()) { + BlockData block = kvIter.nextBlock(); + for(ContainerProtos.ChunkInfo chunk : block.getChunks()) { + File chunkFile = ChunkUtils.getChunkFile(onDiskContainerData, + ChunkInfo.getFromProtoBuf(chunk)); + if (!chunkFile.exists()) { + // concurrent mutation in Block DB? lookup the block again. + byte[] bdata = db.getStore().get( + Longs.toByteArray(block.getBlockID().getLocalID())); + if (bdata != null) { + throw new IOException("Missing chunk file " + + chunkFile.getAbsolutePath()); + } + } else if (chunk.getChecksumData().getType() + != ContainerProtos.ChecksumType.NONE){ + int length = chunk.getChecksumData().getChecksumsList().size(); + ChecksumData cData = new ChecksumData( + chunk.getChecksumData().getType(), + chunk.getChecksumData().getBytesPerChecksum(), + chunk.getChecksumData().getChecksumsList()); + Checksum cal = new Checksum(cData.getChecksumType(), + cData.getBytesPerChecksum()); + long bytesRead = 0; + byte[] buffer = new byte[cData.getBytesPerChecksum()]; + try (InputStream fs = new FileInputStream(chunkFile)) { + for (int i = 0; i < length; i++) { + int v = fs.read(buffer); + if (v == -1) { + break; + } + bytesRead += v; + throttler.throttle(v, canceler); + ByteString expected = cData.getChecksums().get(i); + ByteString actual = cal.computeChecksum(buffer, 0, v) + .getChecksums().get(0); + if (!Arrays.equals(expected.toByteArray(), + actual.toByteArray())) { + throw new OzoneChecksumException(String + .format("Inconsistent read for chunk=%s len=%d expected" + + " checksum %s actual checksum %s for block %s", + chunk.getChunkName(), chunk.getLen(), + Arrays.toString(expected.toByteArray()), + Arrays.toString(actual.toByteArray()), + block.getBlockID())); + } + + } + if (bytesRead != chunk.getLen()) { + throw new OzoneChecksumException(String + .format("Inconsistent read for chunk=%s expected length=%d" + + " actual length=%d for block %s", + chunk.getChunkName(), + chunk.getLen(), bytesRead, block.getBlockID())); + } + } } } } @@ -278,9 +294,8 @@ private void iterateBlockDB(MetadataStore db) } private void loadContainerData() throws IOException { - File containerFile = KeyValueContainer - .getContainerFile(metadataPath.toString(), containerID); + .getContainerFile(metadataPath, containerID); onDiskContainerData = (KeyValueContainerData) ContainerDataYaml .readContainerFile(containerFile); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java index 531fb02830658..bc418839f28bb 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java @@ -18,15 +18,16 @@ package org.apache.hadoop.ozone.container.keyvalue; -import java.io.FileInputStream; import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; import java.nio.ByteBuffer; import java.util.HashMap; import java.util.LinkedList; import java.util.List; import java.util.Map; -import java.util.concurrent.TimeUnit; import java.util.concurrent.locks.ReentrantLock; +import java.util.function.Function; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.StorageUnit; @@ -46,7 +47,7 @@ import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos .PutSmallFileRequestProto; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.Type; -import org.apache.hadoop.hdds.scm.ByteStringHelper; +import org.apache.hadoop.hdds.scm.ByteStringConversion; import org.apache.hadoop.hdds.scm.ScmConfigKeys; import org.apache.hadoop.hdds.scm.container.common.helpers .StorageContainerException; @@ -72,12 +73,10 @@ import org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils; import org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil; import org.apache.hadoop.ozone.container.keyvalue.helpers.SmallFileUtils; -import org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerImpl; +import org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerFactory; import org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl; import org.apache.hadoop.ozone.container.keyvalue.interfaces.ChunkManager; import org.apache.hadoop.ozone.container.keyvalue.interfaces.BlockManager; -import org.apache.hadoop.ozone.container.keyvalue.statemachine.background - .BlockDeletingService; import org.apache.hadoop.util.AutoCloseableLock; import org.apache.hadoop.util.ReflectionUtils; @@ -86,15 +85,8 @@ import org.apache.ratis.thirdparty.com.google.protobuf.ByteString; import static org.apache.hadoop.hdds.HddsConfigKeys .HDDS_DATANODE_VOLUME_CHOOSING_POLICY; -import static org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.Result.*; -import static org.apache.hadoop.ozone.OzoneConfigKeys - .OZONE_BLOCK_DELETING_SERVICE_INTERVAL; -import static org.apache.hadoop.ozone.OzoneConfigKeys - .OZONE_BLOCK_DELETING_SERVICE_INTERVAL_DEFAULT; -import static org.apache.hadoop.ozone.OzoneConfigKeys - .OZONE_BLOCK_DELETING_SERVICE_TIMEOUT; -import static org.apache.hadoop.ozone.OzoneConfigKeys - .OZONE_BLOCK_DELETING_SERVICE_TIMEOUT_DEFAULT; +import static org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos. + Result.*; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -109,9 +101,9 @@ public class KeyValueHandler extends Handler { private final ContainerType containerType; private final BlockManager blockManager; private final ChunkManager chunkManager; - private final BlockDeletingService blockDeletingService; private final VolumeChoosingPolicy volumeChoosingPolicy; private final long maxContainerSize; + private final Function byteBufferToByteString; // A lock that is held during container creation. private final AutoCloseableLock containerCreationLock; @@ -125,19 +117,7 @@ public KeyValueHandler(Configuration config, StateContext context, doSyncWrite = conf.getBoolean(OzoneConfigKeys.DFS_CONTAINER_CHUNK_WRITE_SYNC_KEY, OzoneConfigKeys.DFS_CONTAINER_CHUNK_WRITE_SYNC_DEFAULT); - chunkManager = new ChunkManagerImpl(doSyncWrite); - long svcInterval = config - .getTimeDuration(OZONE_BLOCK_DELETING_SERVICE_INTERVAL, - OZONE_BLOCK_DELETING_SERVICE_INTERVAL_DEFAULT, - TimeUnit.MILLISECONDS); - long serviceTimeout = config - .getTimeDuration(OZONE_BLOCK_DELETING_SERVICE_TIMEOUT, - OZONE_BLOCK_DELETING_SERVICE_TIMEOUT_DEFAULT, - TimeUnit.MILLISECONDS); - this.blockDeletingService = - new BlockDeletingService(containerSet, svcInterval, serviceTimeout, - TimeUnit.MILLISECONDS, config); - blockDeletingService.start(); + chunkManager = ChunkManagerFactory.getChunkManager(config, doSyncWrite); volumeChoosingPolicy = ReflectionUtils.newInstance(conf.getClass( HDDS_DATANODE_VOLUME_CHOOSING_POLICY, RoundRobinVolumeChoosingPolicy .class, VolumeChoosingPolicy.class), conf); @@ -147,10 +127,8 @@ public KeyValueHandler(Configuration config, StateContext context, // this handler lock is used for synchronizing createContainer Requests, // so using a fair lock here. containerCreationLock = new AutoCloseableLock(new ReentrantLock(true)); - boolean isUnsafeByteOperationsEnabled = conf.getBoolean( - OzoneConfigKeys.OZONE_UNSAFEBYTEOPERATIONS_ENABLED, - OzoneConfigKeys.OZONE_UNSAFEBYTEOPERATIONS_ENABLED_DEFAULT); - ByteStringHelper.init(isUnsafeByteOperationsEnabled); + byteBufferToByteString = + ByteStringConversion.createByteBufferConversion(conf); } @VisibleForTesting @@ -158,6 +136,10 @@ public VolumeChoosingPolicy getVolumeChoosingPolicyForTesting() { return volumeChoosingPolicy; } + @Override + public void stop() { + } + @Override public ContainerCommandResponseProto handle( ContainerCommandRequestProto request, Container container, @@ -224,8 +206,10 @@ public BlockManager getBlockManager() { ContainerCommandResponseProto handleCreateContainer( ContainerCommandRequestProto request, KeyValueContainer kvContainer) { if (!request.hasCreateContainer()) { - LOG.debug("Malformed Create Container request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Create Container request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } // Create Container request should be passed a null container as the @@ -287,8 +271,10 @@ public void populateContainerPathFields(KeyValueContainer container, ContainerCommandResponseProto handleReadContainer( ContainerCommandRequestProto request, KeyValueContainer kvContainer) { if (!request.hasReadContainer()) { - LOG.debug("Malformed Read Container request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Read Container request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } @@ -314,8 +300,10 @@ ContainerCommandResponseProto handleUpdateContainer( ContainerCommandRequestProto request, KeyValueContainer kvContainer) { if (!request.hasUpdateContainer()) { - LOG.debug("Malformed Update Container request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Update Container request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } @@ -348,8 +336,10 @@ ContainerCommandResponseProto handleDeleteContainer( ContainerCommandRequestProto request, KeyValueContainer kvContainer) { if (!request.hasDeleteContainer()) { - LOG.debug("Malformed Delete container request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Delete container request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } @@ -370,8 +360,10 @@ ContainerCommandResponseProto handleCloseContainer( ContainerCommandRequestProto request, KeyValueContainer kvContainer) { if (!request.hasCloseContainer()) { - LOG.debug("Malformed Update Container request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Update Container request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } try { @@ -397,8 +389,10 @@ ContainerCommandResponseProto handlePutBlock( long blockLength; if (!request.hasPutBlock()) { - LOG.debug("Malformed Put Key request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Put Key request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } @@ -433,8 +427,10 @@ ContainerCommandResponseProto handleGetBlock( ContainerCommandRequestProto request, KeyValueContainer kvContainer) { if (!request.hasGetBlock()) { - LOG.debug("Malformed Get Key request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Get Key request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } @@ -472,8 +468,10 @@ ContainerCommandResponseProto handleGetBlock( ContainerCommandResponseProto handleGetCommittedBlockLength( ContainerCommandRequestProto request, KeyValueContainer kvContainer) { if (!request.hasGetCommittedBlockLength()) { - LOG.debug("Malformed Get Key request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Get Key request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } @@ -508,8 +506,10 @@ ContainerCommandResponseProto handleDeleteBlock( ContainerCommandRequestProto request, KeyValueContainer kvContainer) { if (!request.hasDeleteBlock()) { - LOG.debug("Malformed Delete Key request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Delete Key request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } @@ -539,13 +539,15 @@ ContainerCommandResponseProto handleReadChunk( DispatcherContext dispatcherContext) { if (!request.hasReadChunk()) { - LOG.debug("Malformed Read Chunk request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Read Chunk request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } // The container can become unhealthy after the lock is released. - // The operation will likely fail/timeout in that happens. + // The operation will likely fail/timeout if that happens. try { checkContainerIsHealthy(kvContainer); } catch (StorageContainerException sce) { @@ -553,7 +555,7 @@ ContainerCommandResponseProto handleReadChunk( } ChunkInfo chunkInfo; - byte[] data; + ByteBuffer data; try { BlockID blockID = BlockID.getFromProtobuf( request.getReadChunk().getBlockID()); @@ -567,7 +569,7 @@ ContainerCommandResponseProto handleReadChunk( data = chunkManager .readChunk(kvContainer, blockID, chunkInfo, dispatcherContext); - metrics.incContainerBytesStats(Type.ReadChunk, data.length); + metrics.incContainerBytesStats(Type.ReadChunk, chunkInfo.getLen()); } catch (StorageContainerException ex) { return ContainerUtils.logAndReturnError(LOG, ex, request); } catch (IOException ex) { @@ -576,7 +578,18 @@ ContainerCommandResponseProto handleReadChunk( request); } - return ChunkUtils.getReadChunkResponse(request, data, chunkInfo); + Preconditions.checkNotNull(data, "Chunk data is null"); + + ContainerProtos.ReadChunkResponseProto.Builder response = + ContainerProtos.ReadChunkResponseProto.newBuilder(); + response.setChunkData(chunkInfo.getProtoBufMessage()); + response.setData(byteBufferToByteString.apply(data)); + response.setBlockID(request.getReadChunk().getBlockID()); + + ContainerCommandResponseProto.Builder builder = + ContainerUtils.getSuccessResponseBuilder(request); + builder.setReadChunk(response); + return builder.build(); } /** @@ -607,8 +620,10 @@ ContainerCommandResponseProto handleDeleteChunk( ContainerCommandRequestProto request, KeyValueContainer kvContainer) { if (!request.hasDeleteChunk()) { - LOG.debug("Malformed Delete Chunk request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Delete Chunk request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } @@ -650,8 +665,10 @@ ContainerCommandResponseProto handleWriteChunk( DispatcherContext dispatcherContext) { if (!request.hasWriteChunk()) { - LOG.debug("Malformed Write Chunk request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Write Chunk request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } @@ -705,8 +722,10 @@ ContainerCommandResponseProto handlePutSmallFile( DispatcherContext dispatcherContext) { if (!request.hasPutSmallFile()) { - LOG.debug("Malformed Put Small File request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Put Small File request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } PutSmallFileRequestProto putSmallFileReq = @@ -763,8 +782,10 @@ ContainerCommandResponseProto handleGetSmallFile( ContainerCommandRequestProto request, KeyValueContainer kvContainer) { if (!request.hasGetSmallFile()) { - LOG.debug("Malformed Get Small File request. trace ID: {}", - request.getTraceID()); + if (LOG.isDebugEnabled()) { + LOG.debug("Malformed Get Small File request. trace ID: {}", + request.getTraceID()); + } return ContainerUtils.malformedRequest(request); } @@ -790,9 +811,9 @@ ContainerCommandResponseProto handleGetSmallFile( for (ContainerProtos.ChunkInfo chunk : responseData.getChunks()) { // if the block is committed, all chunks must have been committed. // Tmp chunk files won't exist here. - byte[] data = chunkManager.readChunk(kvContainer, blockID, + ByteBuffer data = chunkManager.readChunk(kvContainer, blockID, ChunkInfo.getFromProtoBuf(chunk), dispatcherContext); - ByteString current = ByteString.copyFrom(data); + ByteString current = byteBufferToByteString.apply(data); dataBuf = dataBuf.concat(current); chunkInfo = chunk; } @@ -860,13 +881,14 @@ private void checkContainerOpen(KeyValueContainer kvContainer) throw new StorageContainerException(msg, result); } - public Container importContainer(long containerID, long maxSize, - String originPipelineId, - String originNodeId, - FileInputStream rawContainerStream, - TarContainerPacker packer) + @Override + public Container importContainer(final long containerID, + final long maxSize, final String originPipelineId, + final String originNodeId, final InputStream rawContainerStream, + final TarContainerPacker packer) throws IOException { + // TODO: Add layout version! KeyValueContainerData containerData = new KeyValueContainerData(containerID, maxSize, originPipelineId, originNodeId); @@ -881,55 +903,114 @@ public Container importContainer(long containerID, long maxSize, } + @Override + public void exportContainer(final Container container, + final OutputStream outputStream, + final TarContainerPacker packer) + throws IOException{ + container.readLock(); + try { + final KeyValueContainer kvc = (KeyValueContainer) container; + kvc.exportContainerData(outputStream, packer); + } finally { + container.readUnlock(); + } + } + @Override public void markContainerForClose(Container container) throws IOException { - State currentState = container.getContainerState(); - // Move the container to CLOSING state only if it's OPEN - if (currentState == State.OPEN) { - container.markContainerForClose(); - sendICR(container); + container.writeLock(); + try { + // Move the container to CLOSING state only if it's OPEN + if (container.getContainerState() == State.OPEN) { + container.markContainerForClose(); + sendICR(container); + } + } finally { + container.writeUnlock(); } } @Override - public void quasiCloseContainer(Container container) + public void markContainerUnhealthy(Container container) throws IOException { - final State state = container.getContainerState(); - // Quasi close call is idempotent. - if (state == State.QUASI_CLOSED) { - return; + container.writeLock(); + try { + if (container.getContainerState() != State.UNHEALTHY) { + try { + container.markContainerUnhealthy(); + } catch (IOException ex) { + // explicitly catch IOException here since the this operation + // will fail if the Rocksdb metadata is corrupted. + long id = container.getContainerData().getContainerID(); + LOG.warn("Unexpected error while marking container " + id + + " as unhealthy", ex); + } finally { + sendICR(container); + } + } + } finally { + container.writeUnlock(); } - // The container has to be in CLOSING state. - if (state != State.CLOSING) { - ContainerProtos.Result error = state == State.INVALID ? - INVALID_CONTAINER_STATE : CONTAINER_INTERNAL_ERROR; - throw new StorageContainerException("Cannot quasi close container #" + - container.getContainerData().getContainerID() + " while in " + - state + " state.", error); + } + + @Override + public void quasiCloseContainer(Container container) + throws IOException { + container.writeLock(); + try { + final State state = container.getContainerState(); + // Quasi close call is idempotent. + if (state == State.QUASI_CLOSED) { + return; + } + // The container has to be in CLOSING state. + if (state != State.CLOSING) { + ContainerProtos.Result error = + state == State.INVALID ? INVALID_CONTAINER_STATE : + CONTAINER_INTERNAL_ERROR; + throw new StorageContainerException( + "Cannot quasi close container #" + container.getContainerData() + .getContainerID() + " while in " + state + " state.", error); + } + container.quasiClose(); + sendICR(container); + } finally { + container.writeUnlock(); } - container.quasiClose(); - sendICR(container); } @Override public void closeContainer(Container container) throws IOException { - final State state = container.getContainerState(); - // Close call is idempotent. - if (state == State.CLOSED) { - return; - } - // The container has to be either in CLOSING or in QUASI_CLOSED state. - if (state != State.CLOSING && state != State.QUASI_CLOSED) { - ContainerProtos.Result error = state == State.INVALID ? - INVALID_CONTAINER_STATE : CONTAINER_INTERNAL_ERROR; - throw new StorageContainerException("Cannot close container #" + - container.getContainerData().getContainerID() + " while in " + - state + " state.", error); + container.writeLock(); + try { + final State state = container.getContainerState(); + // Close call is idempotent. + if (state == State.CLOSED) { + return; + } + if (state == State.UNHEALTHY) { + throw new StorageContainerException( + "Cannot close container #" + container.getContainerData() + .getContainerID() + " while in " + state + " state.", + ContainerProtos.Result.CONTAINER_UNHEALTHY); + } + // The container has to be either in CLOSING or in QUASI_CLOSED state. + if (state != State.CLOSING && state != State.QUASI_CLOSED) { + ContainerProtos.Result error = + state == State.INVALID ? INVALID_CONTAINER_STATE : + CONTAINER_INTERNAL_ERROR; + throw new StorageContainerException( + "Cannot close container #" + container.getContainerData() + .getContainerID() + " while in " + state + " state.", error); + } + container.close(); + sendICR(container); + } finally { + container.writeUnlock(); } - container.close(); - sendICR(container); } @Override diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/BlockUtils.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/BlockUtils.java index 996b5922fe57c..da7c8579d887b 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/BlockUtils.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/BlockUtils.java @@ -37,7 +37,7 @@ import org.apache.hadoop.ozone.container.common.helpers.ContainerUtils; import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerData; import org.apache.hadoop.ozone.container.common.utils.ContainerCache; -import org.apache.hadoop.utils.MetadataStore; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import java.io.IOException; @@ -66,7 +66,7 @@ private BlockUtils() { * @return MetadataStore handle. * @throws StorageContainerException */ - public static MetadataStore getDB(KeyValueContainerData containerData, + public static ReferenceCountedDB getDB(KeyValueContainerData containerData, Configuration conf) throws StorageContainerException { Preconditions.checkNotNull(containerData); diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/ChunkUtils.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/ChunkUtils.java index 2781bfacca10e..8ca59b5914644 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/ChunkUtils.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/ChunkUtils.java @@ -18,14 +18,12 @@ package org.apache.hadoop.ozone.container.keyvalue.helpers; +import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos .ContainerCommandRequestProto; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos .ContainerCommandResponseProto; -import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos - .ReadChunkResponseProto; -import org.apache.hadoop.hdds.scm.ByteStringHelper; import org.apache.hadoop.hdds.scm.container.common.helpers .StorageContainerException; import org.apache.hadoop.io.IOUtils; @@ -36,16 +34,20 @@ import org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerImpl; import org.apache.hadoop.ozone.container.common.volume.VolumeIOStats; import org.apache.hadoop.util.Time; +import org.apache.ratis.util.function.CheckedSupplier; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.io.File; import java.io.IOException; import java.nio.ByteBuffer; -import java.nio.channels.AsynchronousFileChannel; +import java.nio.channels.FileChannel; import java.nio.channels.FileLock; +import java.nio.file.Path; import java.nio.file.StandardOpenOption; import java.security.NoSuchAlgorithmException; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ExecutionException; import static org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.Result.*; @@ -55,6 +57,8 @@ */ public final class ChunkUtils { + private static final Set LOCKS = ConcurrentHashMap.newKeySet(); + /** Never constructed. **/ private ChunkUtils() { @@ -66,15 +70,14 @@ private ChunkUtils() { * @param chunkFile - File to write data to. * @param chunkInfo - Data stream to write. * @param data - The data buffer. - * @param volumeIOStats + * @param volumeIOStats statistics collector * @param sync whether to do fsync or not - * @throws StorageContainerException */ public static void writeData(File chunkFile, ChunkInfo chunkInfo, ByteBuffer data, VolumeIOStats volumeIOStats, boolean sync) throws StorageContainerException, ExecutionException, InterruptedException, NoSuchAlgorithmException { - int bufferSize = data.capacity(); + final int bufferSize = data.remaining(); Logger log = LoggerFactory.getLogger(ChunkManagerImpl.class); if (bufferSize != chunkInfo.getLen()) { String err = String.format("data array does not match the length " + @@ -84,59 +87,47 @@ public static void writeData(File chunkFile, ChunkInfo chunkInfo, throw new StorageContainerException(err, INVALID_WRITE_SIZE); } - AsynchronousFileChannel file = null; - FileLock lock = null; + Path path = chunkFile.toPath(); + long startTime = Time.monotonicNow(); + processFileExclusively(path, () -> { + FileChannel file = null; + try { + // skip SYNC and DSYNC to reduce contention on file.lock + file = FileChannel.open(path, + StandardOpenOption.CREATE, + StandardOpenOption.WRITE, + StandardOpenOption.SPARSE); + + int size; + try (FileLock ignored = file.lock()) { + size = file.write(data, chunkInfo.getOffset()); + } - try { - long writeTimeStart = Time.monotonicNow(); - file = sync ? - AsynchronousFileChannel.open(chunkFile.toPath(), - StandardOpenOption.CREATE, - StandardOpenOption.WRITE, - StandardOpenOption.SPARSE, - StandardOpenOption.SYNC) : - AsynchronousFileChannel.open(chunkFile.toPath(), - StandardOpenOption.CREATE, - StandardOpenOption.WRITE, - StandardOpenOption.SPARSE); - lock = file.lock().get(); - int size = file.write(data, chunkInfo.getOffset()).get(); - // Increment volumeIO stats here. - volumeIOStats.incWriteTime(Time.monotonicNow() - writeTimeStart); - volumeIOStats.incWriteOpCount(); - volumeIOStats.incWriteBytes(size); - if (size != bufferSize) { - log.error("Invalid write size found. Size:{} Expected: {} ", size, - bufferSize); - throw new StorageContainerException("Invalid write size found. " + - "Size: " + size + " Expected: " + bufferSize, INVALID_WRITE_SIZE); + // Increment volumeIO stats here. + volumeIOStats.incWriteTime(Time.monotonicNow() - startTime); + volumeIOStats.incWriteOpCount(); + volumeIOStats.incWriteBytes(size); + if (size != bufferSize) { + log.error("Invalid write size found. Size:{} Expected: {} ", size, + bufferSize); + throw new StorageContainerException("Invalid write size found. " + + "Size: " + size + " Expected: " + bufferSize, INVALID_WRITE_SIZE); + } + } catch (StorageContainerException ex) { + throw ex; + } catch (IOException e) { + throw new StorageContainerException(e, IO_EXCEPTION); + } finally { + closeFile(file, sync); } - } catch (StorageContainerException ex) { - throw ex; - } catch(IOException e) { - throw new StorageContainerException(e, IO_EXCEPTION); - } finally { - if (lock != null) { - try { - lock.release(); - } catch (IOException e) { - log.error("Unable to release lock ??, Fatal Error."); - throw new StorageContainerException(e, CONTAINER_INTERNAL_ERROR); + return null; + }); - } - } - if (file != null) { - try { - file.close(); - } catch (IOException e) { - throw new StorageContainerException("Error closing chunk file", - e, CONTAINER_INTERNAL_ERROR); - } - } + if (log.isDebugEnabled()) { + log.debug("Write Chunk completed for chunkFile: {}, size {}", chunkFile, + bufferSize); } - log.debug("Write Chunk completed for chunkFile: {}, size {}", chunkFile, - bufferSize); } /** @@ -144,15 +135,11 @@ public static void writeData(File chunkFile, ChunkInfo chunkInfo, * * @param chunkFile - file where data lives. * @param data - chunk definition. - * @param volumeIOStats + * @param volumeIOStats statistics collector * @return ByteBuffer - * @throws StorageContainerException - * @throws ExecutionException - * @throws InterruptedException */ public static ByteBuffer readData(File chunkFile, ChunkInfo data, - VolumeIOStats volumeIOStats) throws StorageContainerException, - ExecutionException, InterruptedException { + VolumeIOStats volumeIOStats) throws StorageContainerException { Logger log = LoggerFactory.getLogger(ChunkManagerImpl.class); if (!chunkFile.exists()) { @@ -163,38 +150,37 @@ public static ByteBuffer readData(File chunkFile, ChunkInfo data, data.toString(), UNABLE_TO_FIND_CHUNK); } - AsynchronousFileChannel file = null; - FileLock lock = null; - try { - long readStartTime = Time.monotonicNow(); - file = - AsynchronousFileChannel.open(chunkFile.toPath(), - StandardOpenOption.READ); - lock = file.lock(data.getOffset(), data.getLen(), true).get(); - - ByteBuffer buf = ByteBuffer.allocate((int) data.getLen()); - file.read(buf, data.getOffset()).get(); - - // Increment volumeIO stats here. - volumeIOStats.incReadTime(Time.monotonicNow() - readStartTime); - volumeIOStats.incReadOpCount(); - volumeIOStats.incReadBytes(data.getLen()); - - return buf; - } catch (IOException e) { - throw new StorageContainerException(e, IO_EXCEPTION); - } finally { - if (lock != null) { - try { - lock.release(); - } catch (IOException e) { - log.error("I/O error is lock release."); + long offset = data.getOffset(); + long len = data.getLen(); + ByteBuffer buf = ByteBuffer.allocate((int) len); + + Path path = chunkFile.toPath(); + long startTime = Time.monotonicNow(); + return processFileExclusively(path, () -> { + FileChannel file = null; + + try { + file = FileChannel.open(path, StandardOpenOption.READ); + + try (FileLock ignored = file.lock(offset, len, true)) { + file.read(buf, offset); + buf.flip(); + } + + // Increment volumeIO stats here. + volumeIOStats.incReadTime(Time.monotonicNow() - startTime); + volumeIOStats.incReadOpCount(); + volumeIOStats.incReadBytes(len); + + return buf; + } catch (IOException e) { + throw new StorageContainerException(e, IO_EXCEPTION); + } finally { + if (file != null) { + IOUtils.closeStream(file); } } - if (file != null) { - IOUtils.closeStream(file); - } - } + }); } /** @@ -298,30 +284,36 @@ public static ContainerCommandResponseProto getChunkResponseSuccess( return ContainerUtils.getSuccessResponse(msg); } - /** - * Gets a response to the read chunk calls. - * - * @param msg - Msg - * @param data - Data - * @param info - Info - * @return Response. - */ - public static ContainerCommandResponseProto getReadChunkResponse( - ContainerCommandRequestProto msg, byte[] data, ChunkInfo info) { - Preconditions.checkNotNull(msg); - Preconditions.checkNotNull(data, "Chunk data is null"); - Preconditions.checkNotNull(info, "Chunk Info is null"); - - ReadChunkResponseProto.Builder response = - ReadChunkResponseProto.newBuilder(); - response.setChunkData(info.getProtoBufMessage()); - response.setData( - ByteStringHelper.getByteString(data)); - response.setBlockID(msg.getReadChunk().getBlockID()); - - ContainerCommandResponseProto.Builder builder = - ContainerUtils.getSuccessResponseBuilder(msg); - builder.setReadChunk(response); - return builder.build(); + @VisibleForTesting + static T processFileExclusively( + Path path, CheckedSupplier op + ) throws E { + for (;;) { + if (LOCKS.add(path)) { + break; + } + } + + try { + return op.get(); + } finally { + LOCKS.remove(path); + } + } + + private static void closeFile(FileChannel file, boolean sync) + throws StorageContainerException { + if (file != null) { + try { + if (sync) { + // ensure data and metadata is persisted + file.force(true); + } + file.close(); + } catch (IOException e) { + throw new StorageContainerException("Error closing chunk file", + e, CONTAINER_INTERNAL_ERROR); + } + } } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java index 7a309555a37c9..3733b06b73549 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java @@ -24,21 +24,25 @@ import java.util.List; import java.util.Map; +import com.google.common.primitives.Longs; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos .ContainerCommandRequestProto; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos .ContainerCommandResponseProto; +import org.apache.hadoop.hdfs.DFSUtil; +import org.apache.hadoop.ozone.OzoneConsts; import org.apache.hadoop.ozone.container.common.helpers.ContainerUtils; import org.apache.hadoop.ozone.container.common.helpers.BlockData; import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerData; -import org.apache.hadoop.utils.MetadataKeyFilters; -import org.apache.hadoop.utils.MetadataStore; -import org.apache.hadoop.utils.MetadataStoreBuilder; +import org.apache.hadoop.hdds.utils.MetadataKeyFilters; +import org.apache.hadoop.hdds.utils.MetadataStore; +import org.apache.hadoop.hdds.utils.MetadataStoreBuilder; import com.google.common.base.Preconditions; import org.apache.commons.io.FileUtils; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -174,22 +178,30 @@ public static void parseKVContainerData(KeyValueContainerData kvContainerData, } kvContainerData.setDbFile(dbFile); - MetadataStore metadata = BlockUtils.getDB(kvContainerData, config); - long bytesUsed = 0; - List> liveKeys = metadata - .getRangeKVs(null, Integer.MAX_VALUE, - MetadataKeyFilters.getNormalKeyFilter()); - bytesUsed = liveKeys.parallelStream().mapToLong(e-> { - BlockData blockData; - try { - blockData = BlockUtils.getBlockData(e.getValue()); - return blockData.getSize(); - } catch (IOException ex) { - return 0L; + try(ReferenceCountedDB metadata = + BlockUtils.getDB(kvContainerData, config)) { + long bytesUsed = 0; + List> liveKeys = metadata.getStore() + .getRangeKVs(null, Integer.MAX_VALUE, + MetadataKeyFilters.getNormalKeyFilter()); + + bytesUsed = liveKeys.parallelStream().mapToLong(e-> { + BlockData blockData; + try { + blockData = BlockUtils.getBlockData(e.getValue()); + return blockData.getSize(); + } catch (IOException ex) { + return 0L; + } + }).sum(); + kvContainerData.setBytesUsed(bytesUsed); + kvContainerData.setKeyCount(liveKeys.size()); + byte[] bcsId = metadata.getStore().get(DFSUtil.string2Bytes( + OzoneConsts.BLOCK_COMMIT_SEQUENCE_ID_PREFIX)); + if (bcsId != null) { + kvContainerData.updateBlockCommitSequenceId(Longs.fromByteArray(bcsId)); } - }).sum(); - kvContainerData.setBytesUsed(bytesUsed); - kvContainerData.setKeyCount(liveKeys.size()); + } } /** diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java index 3033dd9017d2d..4272861c57e2d 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java @@ -33,9 +33,9 @@ import org.apache.hadoop.ozone.container.common.interfaces.Container; import org.apache.hadoop.ozone.container.keyvalue.interfaces.BlockManager; import org.apache.hadoop.ozone.container.common.utils.ContainerCache; -import org.apache.hadoop.utils.BatchOperation; -import org.apache.hadoop.utils.MetadataKeyFilters; -import org.apache.hadoop.utils.MetadataStore; +import org.apache.hadoop.hdds.utils.BatchOperation; +import org.apache.hadoop.hdds.utils.MetadataKeyFilters; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -84,47 +84,49 @@ public long putBlock(Container container, BlockData data) throws IOException { "cannot be negative"); // We are not locking the key manager since LevelDb serializes all actions // against a single DB. We rely on DB level locking to avoid conflicts. - MetadataStore db = BlockUtils.getDB((KeyValueContainerData) container - .getContainerData(), config); + try(ReferenceCountedDB db = BlockUtils. + getDB((KeyValueContainerData) container.getContainerData(), config)) { + // This is a post condition that acts as a hint to the user. + // Should never fail. + Preconditions.checkNotNull(db, "DB cannot be null here"); - // This is a post condition that acts as a hint to the user. - // Should never fail. - Preconditions.checkNotNull(db, "DB cannot be null here"); + long bcsId = data.getBlockCommitSequenceId(); + long containerBCSId = ((KeyValueContainerData) container. + getContainerData()).getBlockCommitSequenceId(); - long bcsId = data.getBlockCommitSequenceId(); - long containerBCSId = ((KeyValueContainerData) container.getContainerData()) - .getBlockCommitSequenceId(); - - // default blockCommitSequenceId for any block is 0. It the putBlock - // request is not coming via Ratis(for test scenarios), it will be 0. - // In such cases, we should overwrite the block as well - if (bcsId != 0) { - if (bcsId <= containerBCSId) { - // Since the blockCommitSequenceId stored in the db is greater than - // equal to blockCommitSequenceId to be updated, it means the putBlock - // transaction is reapplied in the ContainerStateMachine on restart. - // It also implies that the given block must already exist in the db. - // just log and return - LOG.warn("blockCommitSequenceId " + containerBCSId - + " in the Container Db is greater than" + " the supplied value " - + bcsId + " .Ignoring it"); - return data.getSize(); + // default blockCommitSequenceId for any block is 0. It the putBlock + // request is not coming via Ratis(for test scenarios), it will be 0. + // In such cases, we should overwrite the block as well + if (bcsId != 0) { + if (bcsId <= containerBCSId) { + // Since the blockCommitSequenceId stored in the db is greater than + // equal to blockCommitSequenceId to be updated, it means the putBlock + // transaction is reapplied in the ContainerStateMachine on restart. + // It also implies that the given block must already exist in the db. + // just log and return + LOG.warn("blockCommitSequenceId " + containerBCSId + + " in the Container Db is greater than" + " the supplied value " + + bcsId + " .Ignoring it"); + return data.getSize(); + } + } + // update the blockData as well as BlockCommitSequenceId here + BatchOperation batch = new BatchOperation(); + batch.put(Longs.toByteArray(data.getLocalID()), + data.getProtoBufMessage().toByteArray()); + batch.put(blockCommitSequenceIdKey, + Longs.toByteArray(bcsId)); + db.getStore().writeBatch(batch); + container.updateBlockCommitSequenceId(bcsId); + // Increment keycount here + container.getContainerData().incrKeyCount(); + if (LOG.isDebugEnabled()) { + LOG.debug( + "Block " + data.getBlockID() + " successfully committed with bcsId " + + bcsId + " chunk size " + data.getChunks().size()); } + return data.getSize(); } - // update the blockData as well as BlockCommitSequenceId here - BatchOperation batch = new BatchOperation(); - batch.put(Longs.toByteArray(data.getLocalID()), - data.getProtoBufMessage().toByteArray()); - batch.put(blockCommitSequenceIdKey, - Longs.toByteArray(bcsId)); - db.writeBatch(batch); - container.updateBlockCommitSequenceId(bcsId); - // Increment keycount here - container.getContainerData().incrKeyCount(); - LOG.debug( - "Block " + data.getBlockID() + " successfully committed with bcsId " - + bcsId + " chunk size " + data.getChunks().size()); - return data.getSize(); } /** @@ -146,32 +148,33 @@ public BlockData getBlock(Container container, BlockID blockID) KeyValueContainerData containerData = (KeyValueContainerData) container .getContainerData(); - MetadataStore db = BlockUtils.getDB(containerData, config); - // This is a post condition that acts as a hint to the user. - // Should never fail. - Preconditions.checkNotNull(db, "DB cannot be null here"); + try(ReferenceCountedDB db = BlockUtils.getDB(containerData, config)) { + // This is a post condition that acts as a hint to the user. + // Should never fail. + Preconditions.checkNotNull(db, "DB cannot be null here"); - long containerBCSId = containerData.getBlockCommitSequenceId(); - if (containerBCSId < bcsId) { - throw new StorageContainerException( - "Unable to find the block with bcsID " + bcsId + " .Container " - + container.getContainerData().getContainerID() + " bcsId is " - + containerBCSId + ".", UNKNOWN_BCSID); - } - byte[] kData = db.get(Longs.toByteArray(blockID.getLocalID())); - if (kData == null) { - throw new StorageContainerException("Unable to find the block." + blockID, - NO_SUCH_BLOCK); - } - ContainerProtos.BlockData blockData = - ContainerProtos.BlockData.parseFrom(kData); - long id = blockData.getBlockID().getBlockCommitSequenceId(); - if (id < bcsId) { - throw new StorageContainerException( - "bcsId " + bcsId + " mismatches with existing block Id " - + id + " for block " + blockID + ".", BCSID_MISMATCH); + long containerBCSId = containerData.getBlockCommitSequenceId(); + if (containerBCSId < bcsId) { + throw new StorageContainerException( + "Unable to find the block with bcsID " + bcsId + " .Container " + + container.getContainerData().getContainerID() + " bcsId is " + + containerBCSId + ".", UNKNOWN_BCSID); + } + byte[] kData = db.getStore().get(Longs.toByteArray(blockID.getLocalID())); + if (kData == null) { + throw new StorageContainerException("Unable to find the block." + + blockID, NO_SUCH_BLOCK); + } + ContainerProtos.BlockData blockData = + ContainerProtos.BlockData.parseFrom(kData); + long id = blockData.getBlockID().getBlockCommitSequenceId(); + if (id < bcsId) { + throw new StorageContainerException( + "bcsId " + bcsId + " mismatches with existing block Id " + + id + " for block " + blockID + ".", BCSID_MISMATCH); + } + return BlockData.getFromProtoBuf(blockData); } - return BlockData.getFromProtoBuf(blockData); } /** @@ -187,18 +190,19 @@ public long getCommittedBlockLength(Container container, BlockID blockID) throws IOException { KeyValueContainerData containerData = (KeyValueContainerData) container .getContainerData(); - MetadataStore db = BlockUtils.getDB(containerData, config); - // This is a post condition that acts as a hint to the user. - // Should never fail. - Preconditions.checkNotNull(db, "DB cannot be null here"); - byte[] kData = db.get(Longs.toByteArray(blockID.getLocalID())); - if (kData == null) { - throw new StorageContainerException("Unable to find the block.", - NO_SUCH_BLOCK); + try(ReferenceCountedDB db = BlockUtils.getDB(containerData, config)) { + // This is a post condition that acts as a hint to the user. + // Should never fail. + Preconditions.checkNotNull(db, "DB cannot be null here"); + byte[] kData = db.getStore().get(Longs.toByteArray(blockID.getLocalID())); + if (kData == null) { + throw new StorageContainerException("Unable to find the block.", + NO_SUCH_BLOCK); + } + ContainerProtos.BlockData blockData = + ContainerProtos.BlockData.parseFrom(kData); + return blockData.getSize(); } - ContainerProtos.BlockData blockData = - ContainerProtos.BlockData.parseFrom(kData); - return blockData.getSize(); } /** @@ -218,24 +222,25 @@ public void deleteBlock(Container container, BlockID blockID) throws KeyValueContainerData cData = (KeyValueContainerData) container .getContainerData(); - MetadataStore db = BlockUtils.getDB(cData, config); - // This is a post condition that acts as a hint to the user. - // Should never fail. - Preconditions.checkNotNull(db, "DB cannot be null here"); - // Note : There is a race condition here, since get and delete - // are not atomic. Leaving it here since the impact is refusing - // to delete a Block which might have just gotten inserted after - // the get check. - byte[] kKey = Longs.toByteArray(blockID.getLocalID()); - byte[] kData = db.get(kKey); - if (kData == null) { - throw new StorageContainerException("Unable to find the block.", - NO_SUCH_BLOCK); - } - db.delete(kKey); + try(ReferenceCountedDB db = BlockUtils.getDB(cData, config)) { + // This is a post condition that acts as a hint to the user. + // Should never fail. + Preconditions.checkNotNull(db, "DB cannot be null here"); + // Note : There is a race condition here, since get and delete + // are not atomic. Leaving it here since the impact is refusing + // to delete a Block which might have just gotten inserted after + // the get check. + byte[] kKey = Longs.toByteArray(blockID.getLocalID()); - // Decrement blockcount here - container.getContainerData().decrKeyCount(); + byte[] kData = db.getStore().get(kKey); + if (kData == null) { + throw new StorageContainerException("Unable to find the block.", + NO_SUCH_BLOCK); + } + db.getStore().delete(kKey); + // Decrement blockcount here + container.getContainerData().decrKeyCount(); + } } /** @@ -255,21 +260,26 @@ public List listBlock(Container container, long startLocalID, int Preconditions.checkArgument(count > 0, "Count must be a positive number."); container.readLock(); - List result = null; - KeyValueContainerData cData = (KeyValueContainerData) container - .getContainerData(); - MetadataStore db = BlockUtils.getDB(cData, config); - result = new ArrayList<>(); - byte[] startKeyInBytes = Longs.toByteArray(startLocalID); - List> range = - db.getSequentialRangeKVs(startKeyInBytes, count, - MetadataKeyFilters.getNormalKeyFilter()); - for (Map.Entry entry : range) { - BlockData value = BlockUtils.getBlockData(entry.getValue()); - BlockData data = new BlockData(value.getBlockID()); - result.add(data); + try { + List result = null; + KeyValueContainerData cData = + (KeyValueContainerData) container.getContainerData(); + try (ReferenceCountedDB db = BlockUtils.getDB(cData, config)) { + result = new ArrayList<>(); + byte[] startKeyInBytes = Longs.toByteArray(startLocalID); + List> range = db.getStore() + .getSequentialRangeKVs(startKeyInBytes, count, + MetadataKeyFilters.getNormalKeyFilter()); + for (Map.Entry entry : range) { + BlockData value = BlockUtils.getBlockData(entry.getValue()); + BlockData data = new BlockData(value.getBlockID()); + result.add(data); + } + return result; + } + } finally { + container.readUnlock(); } - return result; } /** @@ -278,5 +288,4 @@ public List listBlock(Container container, long startLocalID, int public void shutdown() { BlockUtils.shutdownCache(ContainerCache.getInstance(config)); } - -} +} \ No newline at end of file diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/ChunkManagerDummyImpl.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/ChunkManagerDummyImpl.java new file mode 100644 index 0000000000000..fa9e205786e00 --- /dev/null +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/ChunkManagerDummyImpl.java @@ -0,0 +1,162 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.container.keyvalue.impl; + +import com.google.common.base.Preconditions; +import org.apache.hadoop.hdds.client.BlockID; +import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException; +import org.apache.hadoop.ozone.container.common.helpers.ChunkInfo; +import org.apache.hadoop.ozone.container.common.interfaces.Container; +import org.apache.hadoop.ozone.container.common.transport.server.ratis.DispatcherContext; +import org.apache.hadoop.ozone.container.common.volume.HddsVolume; +import org.apache.hadoop.ozone.container.common.volume.VolumeIOStats; +import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerData; +import org.apache.hadoop.util.Time; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.nio.ByteBuffer; + +import static org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.Result.*; + +/** + * Implementation of ChunkManager built for running performance tests. + * Chunks are not written to disk, Reads are returned with zero-filled buffers + */ +public class ChunkManagerDummyImpl extends ChunkManagerImpl { + static final Logger LOG = LoggerFactory.getLogger( + ChunkManagerDummyImpl.class); + + public ChunkManagerDummyImpl(boolean sync) { + super(sync); + } + + /** + * writes a given chunk. + * + * @param container - Container for the chunk + * @param blockID - ID of the block + * @param info - ChunkInfo + * @param data - data of the chunk + * @param dispatcherContext - dispatcherContextInfo + * @throws StorageContainerException + */ + @Override + public void writeChunk(Container container, BlockID blockID, ChunkInfo info, + ByteBuffer data, DispatcherContext dispatcherContext) + throws StorageContainerException { + long writeTimeStart = Time.monotonicNow(); + + Preconditions.checkNotNull(dispatcherContext); + DispatcherContext.WriteChunkStage stage = dispatcherContext.getStage(); + + Logger log = LoggerFactory.getLogger(ChunkManagerImpl.class); + + try { + KeyValueContainerData containerData = + (KeyValueContainerData) container.getContainerData(); + HddsVolume volume = containerData.getVolume(); + VolumeIOStats volumeIOStats = volume.getVolumeIOStats(); + int bufferSize; + + switch (stage) { + case WRITE_DATA: + bufferSize = data.capacity(); + if (bufferSize != info.getLen()) { + String err = String.format("data array does not match the length " + + "specified. DataLen: %d Byte Array: %d", + info.getLen(), bufferSize); + log.error(err); + throw new StorageContainerException(err, INVALID_WRITE_SIZE); + } + + // Increment volumeIO stats here. + volumeIOStats.incWriteTime(Time.monotonicNow() - writeTimeStart); + volumeIOStats.incWriteOpCount(); + volumeIOStats.incWriteBytes(info.getLen()); + break; + case COMMIT_DATA: + updateContainerWriteStats(container, info, false); + break; + case COMBINED: + updateContainerWriteStats(container, info, false); + break; + default: + throw new IOException("Can not identify write operation."); + } + } catch (IOException ex) { + LOG.error("write data failed. error: {}", ex); + throw new StorageContainerException("Internal error: ", ex, + CONTAINER_INTERNAL_ERROR); + } + } + + /** + * return a zero-filled buffer. + * + * @param container - Container for the chunk + * @param blockID - ID of the block. + * @param info - ChunkInfo. + * @param dispatcherContext dispatcher context info. + * @return byte array + * TODO: Right now we do not support partial reads and writes of chunks. + * TODO: Explore if we need to do that for ozone. + */ + @Override + public ByteBuffer readChunk(Container container, BlockID blockID, + ChunkInfo info, DispatcherContext dispatcherContext) { + + long readStartTime = Time.monotonicNow(); + + KeyValueContainerData containerData = (KeyValueContainerData) container + .getContainerData(); + ByteBuffer data; + HddsVolume volume = containerData.getVolume(); + VolumeIOStats volumeIOStats = volume.getVolumeIOStats(); + + data = ByteBuffer.allocate((int) info.getLen()); + + // Increment volumeIO stats here. + volumeIOStats.incReadTime(Time.monotonicNow() - readStartTime); + volumeIOStats.incReadOpCount(); + volumeIOStats.incReadBytes(info.getLen()); + + return data; + } + + /** + * Delete a given chunk - Do nothing except stats. + * + * @param container - Container for the chunk + * @param blockID - ID of the block + * @param info - Chunk Info + */ + @Override + public void deleteChunk(Container container, BlockID blockID, + ChunkInfo info) { + Preconditions.checkNotNull(blockID, "Block ID cannot be null."); + KeyValueContainerData containerData = + (KeyValueContainerData) container.getContainerData(); + + if (info.getOffset() == 0) { + containerData.decrBytesUsed(info.getLen()); + } + } +} \ No newline at end of file diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/ChunkManagerFactory.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/ChunkManagerFactory.java new file mode 100644 index 0000000000000..85495783cc833 --- /dev/null +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/ChunkManagerFactory.java @@ -0,0 +1,91 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.container.keyvalue.impl; + +import com.google.common.base.Preconditions; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.ozone.container.keyvalue.interfaces.ChunkManager; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_CONTAINER_PERSISTDATA; +import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_CONTAINER_PERSISTDATA_DEFAULT; +import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_CONTAINER_SCRUB_ENABLED; +import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_CONTAINER_SCRUB_ENABLED_DEFAULT; + +/** + * Select an appropriate ChunkManager implementation as per config setting. + * Ozone ChunkManager is a Singleton + */ +public final class ChunkManagerFactory { + static final Logger LOG = LoggerFactory.getLogger(ChunkManagerFactory.class); + + private static volatile ChunkManager instance = null; + private static boolean syncChunks = false; + + private ChunkManagerFactory() { + } + + public static ChunkManager getChunkManager(Configuration config, + boolean sync) { + if (instance == null) { + synchronized (ChunkManagerFactory.class) { + if (instance == null) { + instance = createChunkManager(config, sync); + syncChunks = sync; + } + } + } + + Preconditions.checkArgument((syncChunks == sync), + "value of sync conflicts with previous invocation"); + return instance; + } + + private static ChunkManager createChunkManager(Configuration config, + boolean sync) { + ChunkManager manager = null; + boolean persist = config.getBoolean(HDDS_CONTAINER_PERSISTDATA, + HDDS_CONTAINER_PERSISTDATA_DEFAULT); + + if (!persist) { + boolean scrubber = config.getBoolean( + HDDS_CONTAINER_SCRUB_ENABLED, + HDDS_CONTAINER_SCRUB_ENABLED_DEFAULT); + if (scrubber) { + // Data Scrubber needs to be disabled for non-persistent chunks. + LOG.warn("Failed to set " + HDDS_CONTAINER_PERSISTDATA + " to false." + + " Please set " + HDDS_CONTAINER_SCRUB_ENABLED + + " also to false to enable non-persistent containers."); + persist = true; + } + } + + if (persist) { + manager = new ChunkManagerImpl(sync); + } else { + LOG.warn(HDDS_CONTAINER_PERSISTDATA + + " is set to false. This should be used only for testing." + + " All user data will be discarded."); + manager = new ChunkManagerDummyImpl(sync); + } + + return manager; + } +} \ No newline at end of file diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/ChunkManagerImpl.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/ChunkManagerImpl.java index e4814cb26f693..e22841eec8a49 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/ChunkManagerImpl.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/ChunkManagerImpl.java @@ -87,10 +87,11 @@ public void writeChunk(Container container, BlockID blockID, ChunkInfo info, boolean isOverwrite = ChunkUtils.validateChunkForOverwrite( chunkFile, info); File tmpChunkFile = getTmpChunkFile(chunkFile, dispatcherContext); - - LOG.debug( - "writing chunk:{} chunk stage:{} chunk file:{} tmp chunk file:{}", - info.getChunkName(), stage, chunkFile, tmpChunkFile); + if (LOG.isDebugEnabled()) { + LOG.debug( + "writing chunk:{} chunk stage:{} chunk file:{} tmp chunk file:{}", + info.getChunkName(), stage, chunkFile, tmpChunkFile); + } switch (stage) { case WRITE_DATA: @@ -142,18 +143,12 @@ public void writeChunk(Container container, BlockID blockID, ChunkInfo info, // the same term and log index appended as the current transaction commitChunk(tmpChunkFile, chunkFile); // Increment container stats here, as we commit the data. - containerData.incrBytesUsed(info.getLen()); - containerData.incrWriteCount(); - containerData.incrWriteBytes(info.getLen()); + updateContainerWriteStats(container, info, isOverwrite); break; case COMBINED: // directly write to the chunk file ChunkUtils.writeData(chunkFile, info, data, volumeIOStats, doSyncWrite); - if (!isOverwrite) { - containerData.incrBytesUsed(info.getLen()); - } - containerData.incrWriteCount(); - containerData.incrWriteBytes(info.getLen()); + updateContainerWriteStats(container, info, isOverwrite); break; default: throw new IOException("Can not identify write operation."); @@ -176,6 +171,18 @@ public void writeChunk(Container container, BlockID blockID, ChunkInfo info, } } + protected void updateContainerWriteStats(Container container, ChunkInfo info, + boolean isOverwrite) { + KeyValueContainerData containerData = (KeyValueContainerData) container + .getContainerData(); + + if (!isOverwrite) { + containerData.incrBytesUsed(info.getLen()); + } + containerData.incrWriteCount(); + containerData.incrWriteBytes(info.getLen()); + } + /** * reads the data defined by a chunk. * @@ -188,43 +195,34 @@ public void writeChunk(Container container, BlockID blockID, ChunkInfo info, * TODO: Right now we do not support partial reads and writes of chunks. * TODO: Explore if we need to do that for ozone. */ - public byte[] readChunk(Container container, BlockID blockID, ChunkInfo info, - DispatcherContext dispatcherContext) throws StorageContainerException { - try { - KeyValueContainerData containerData = (KeyValueContainerData) container - .getContainerData(); - ByteBuffer data; - HddsVolume volume = containerData.getVolume(); - VolumeIOStats volumeIOStats = volume.getVolumeIOStats(); + public ByteBuffer readChunk(Container container, BlockID blockID, + ChunkInfo info, DispatcherContext dispatcherContext) + throws StorageContainerException { + KeyValueContainerData containerData = (KeyValueContainerData) container + .getContainerData(); + ByteBuffer data; + HddsVolume volume = containerData.getVolume(); + VolumeIOStats volumeIOStats = volume.getVolumeIOStats(); - // Checking here, which layout version the container is, and reading - // the chunk file in that format. - // In version1, we verify checksum if it is available and return data - // of the chunk file. - if (containerData.getLayOutVersion() == ChunkLayOutVersion - .getLatestVersion().getVersion()) { - File chunkFile = ChunkUtils.getChunkFile(containerData, info); + // Checking here, which layout version the container is, and reading + // the chunk file in that format. + // In version1, we verify checksum if it is available and return data + // of the chunk file. + if (containerData.getLayOutVersion() == ChunkLayOutVersion + .getLatestVersion().getVersion()) { + File chunkFile = ChunkUtils.getChunkFile(containerData, info); - // In case the chunk file does not exist but tmp chunk file exist, - // read from tmp chunk file if readFromTmpFile is set to true - if (!chunkFile.exists() && dispatcherContext.isReadFromTmpFile()) { - chunkFile = getTmpChunkFile(chunkFile, dispatcherContext); - } - data = ChunkUtils.readData(chunkFile, info, volumeIOStats); - containerData.incrReadCount(); - long length = chunkFile.length(); - containerData.incrReadBytes(length); - return data.array(); + // In case the chunk file does not exist but tmp chunk file exist, + // read from tmp chunk file if readFromTmpFile is set to true + if (!chunkFile.exists() && dispatcherContext != null + && dispatcherContext.isReadFromTmpFile()) { + chunkFile = getTmpChunkFile(chunkFile, dispatcherContext); } - } catch (ExecutionException ex) { - LOG.error("read data failed. error: {}", ex); - throw new StorageContainerException("Internal error: ", - ex, CONTAINER_INTERNAL_ERROR); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - LOG.error("read data failed. error: {}", e); - throw new StorageContainerException("Internal error: ", - e, CONTAINER_INTERNAL_ERROR); + data = ChunkUtils.readData(chunkFile, info, volumeIOStats); + containerData.incrReadCount(); + long length = chunkFile.length(); + containerData.incrReadBytes(length); + return data; } return null; } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/interfaces/ChunkManager.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/interfaces/ChunkManager.java index 5a6898f558a88..5adb6415ec1c2 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/interfaces/ChunkManager.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/interfaces/ChunkManager.java @@ -59,7 +59,7 @@ void writeChunk(Container container, BlockID blockID, ChunkInfo info, * TODO: Right now we do not support partial reads and writes of chunks. * TODO: Explore if we need to do that for ozone. */ - byte[] readChunk(Container container, BlockID blockID, ChunkInfo info, + ByteBuffer readChunk(Container container, BlockID blockID, ChunkInfo info, DispatcherContext dispatcherContext) throws StorageContainerException; /** diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java index 61a303fcdd20d..bc3f51a54ef5f 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java @@ -20,12 +20,15 @@ import com.google.common.collect.Lists; import org.apache.hadoop.hdds.scm.ScmConfigKeys; +import org.apache.hadoop.hdds.scm.pipeline.PipelineID; import org.apache.hadoop.ozone.container.common.impl.ContainerData; -import org.apache.hadoop.ozone.container.common.impl.ContainerSet; import org.apache.hadoop.ozone.container.common.impl.TopNOrderedContainerDeletionChoosingPolicy; +import org.apache.hadoop.ozone.container.common.interfaces.Container; import org.apache.hadoop.ozone.container.common.interfaces.ContainerDeletionChoosingPolicy; +import org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis; import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerData; import org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils; +import org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer; import org.apache.hadoop.util.ReflectionUtils; import org.apache.ratis.thirdparty.com.google.protobuf .InvalidProtocolBufferException; @@ -37,21 +40,24 @@ import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; import org.apache.hadoop.ozone.OzoneConsts; import org.apache.hadoop.util.Time; -import org.apache.hadoop.utils.BackgroundService; -import org.apache.hadoop.utils.BackgroundTask; -import org.apache.hadoop.utils.BackgroundTaskQueue; -import org.apache.hadoop.utils.BackgroundTaskResult; -import org.apache.hadoop.utils.BatchOperation; -import org.apache.hadoop.utils.MetadataKeyFilters.KeyPrefixFilter; -import org.apache.hadoop.utils.MetadataStore; +import org.apache.hadoop.hdds.utils.BackgroundService; +import org.apache.hadoop.hdds.utils.BackgroundTask; +import org.apache.hadoop.hdds.utils.BackgroundTaskQueue; +import org.apache.hadoop.hdds.utils.BackgroundTaskResult; +import org.apache.hadoop.hdds.utils.BatchOperation; +import org.apache.hadoop.hdds.utils.MetadataKeyFilters.KeyPrefixFilter; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.io.File; +import java.io.IOException; import java.util.LinkedList; import java.util.List; import java.util.Map; +import java.util.UUID; import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; import static org.apache.hadoop.ozone.OzoneConfigKeys .OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL; @@ -67,12 +73,12 @@ * of deleting staled ozone blocks. */ // TODO: Fix BlockDeletingService to work with new StorageLayer -public class BlockDeletingService extends BackgroundService{ +public class BlockDeletingService extends BackgroundService { private static final Logger LOG = LoggerFactory.getLogger(BlockDeletingService.class); - private ContainerSet containerSet; + private OzoneContainer ozoneContainer; private ContainerDeletionChoosingPolicy containerDeletionPolicy; private final Configuration conf; @@ -88,22 +94,23 @@ public class BlockDeletingService extends BackgroundService{ // Core pool size for container tasks private final static int BLOCK_DELETING_SERVICE_CORE_POOL_SIZE = 10; - public BlockDeletingService(ContainerSet containerSet, long serviceInterval, - long serviceTimeout, TimeUnit timeUnit, Configuration conf) { + public BlockDeletingService(OzoneContainer ozoneContainer, + long serviceInterval, long serviceTimeout, TimeUnit timeUnit, + Configuration conf) { super("BlockDeletingService", serviceInterval, timeUnit, BLOCK_DELETING_SERVICE_CORE_POOL_SIZE, serviceTimeout); - this.containerSet = containerSet; + this.ozoneContainer = ozoneContainer; containerDeletionPolicy = ReflectionUtils.newInstance(conf.getClass( ScmConfigKeys.OZONE_SCM_KEY_VALUE_CONTAINER_DELETION_CHOOSING_POLICY, TopNOrderedContainerDeletionChoosingPolicy.class, ContainerDeletionChoosingPolicy.class), conf); this.conf = conf; - this.blockLimitPerTask = conf.getInt( - OZONE_BLOCK_DELETING_LIMIT_PER_CONTAINER, - OZONE_BLOCK_DELETING_LIMIT_PER_CONTAINER_DEFAULT); - this.containerLimitPerInterval = conf.getInt( - OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL, - OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL_DEFAULT); + this.blockLimitPerTask = + conf.getInt(OZONE_BLOCK_DELETING_LIMIT_PER_CONTAINER, + OZONE_BLOCK_DELETING_LIMIT_PER_CONTAINER_DEFAULT); + this.containerLimitPerInterval = + conf.getInt(OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL, + OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL_DEFAULT); } @@ -117,8 +124,8 @@ public BackgroundTaskQueue getTasks() { // We must ensure there is no empty container in this result. // The chosen result depends on what container deletion policy is // configured. - containers = containerSet.chooseContainerForBlockDeletion( - containerLimitPerInterval, containerDeletionPolicy); + containers = chooseContainerForBlockDeletion(containerLimitPerInterval, + containerDeletionPolicy); if (containers.size() > 0) { LOG.info("Plan to choose {} containers for block deletion, " + "actually returns {} valid containers.", @@ -143,6 +150,64 @@ public BackgroundTaskQueue getTasks() { return queue; } + public List chooseContainerForBlockDeletion(int count, + ContainerDeletionChoosingPolicy deletionPolicy) + throws StorageContainerException { + Map containerDataMap = + ozoneContainer.getContainerSet().getContainerMap().entrySet().stream() + .filter(e -> isDeletionAllowed(e.getValue().getContainerData(), + deletionPolicy)).collect(Collectors + .toMap(Map.Entry::getKey, e -> e.getValue().getContainerData())); + return deletionPolicy + .chooseContainerForBlockDeletion(count, containerDataMap); + } + + private boolean isDeletionAllowed(ContainerData containerData, + ContainerDeletionChoosingPolicy deletionPolicy) { + if (!deletionPolicy + .isValidContainerType(containerData.getContainerType())) { + return false; + } else if (!containerData.isClosed()) { + return false; + } else { + if (ozoneContainer.getWriteChannel() instanceof XceiverServerRatis) { + XceiverServerRatis ratisServer = + (XceiverServerRatis) ozoneContainer.getWriteChannel(); + PipelineID pipelineID = PipelineID + .valueOf(UUID.fromString(containerData.getOriginPipelineId())); + // in case te ratis group does not exist, just mark it for deletion. + if (!ratisServer.isExist(pipelineID.getProtobuf())) { + return true; + } + try { + long minReplicatedIndex = + ratisServer.getMinReplicatedIndex(pipelineID); + long containerBCSID = containerData.getBlockCommitSequenceId(); + if (minReplicatedIndex >= 0 && minReplicatedIndex < containerBCSID) { + LOG.warn("Close Container log Index {} is not replicated across all" + + "the servers in the pipeline {} as the min replicated " + + "index is {}. Deletion is not allowed in this container " + + "yet.", containerBCSID, + containerData.getOriginPipelineId(), minReplicatedIndex); + return false; + } else { + return true; + } + } catch (IOException ioe) { + // in case of any exception check again whether the pipeline exist + // and in case the pipeline got destroyed, just mark it for deletion + if (!ratisServer.isExist(pipelineID.getProtobuf())) { + return true; + } else { + LOG.info(ioe.getMessage()); + return false; + } + } + } + return true; + } + } + private static class ContainerBackgroundTaskResult implements BackgroundTaskResult { private List deletedBlockIds; @@ -183,71 +248,80 @@ private class BlockDeletingTask @Override public BackgroundTaskResult call() throws Exception { ContainerBackgroundTaskResult crr = new ContainerBackgroundTaskResult(); + final Container container = ozoneContainer.getContainerSet() + .getContainer(containerData.getContainerID()); + container.writeLock(); long startTime = Time.monotonicNow(); // Scan container's db and get list of under deletion blocks - MetadataStore meta = BlockUtils.getDB( - (KeyValueContainerData) containerData, conf); - // # of blocks to delete is throttled - KeyPrefixFilter filter = - new KeyPrefixFilter().addFilter(OzoneConsts.DELETING_KEY_PREFIX); - List> toDeleteBlocks = - meta.getSequentialRangeKVs(null, blockLimitPerTask, filter); - if (toDeleteBlocks.isEmpty()) { - LOG.debug("No under deletion block found in container : {}", - containerData.getContainerID()); - } + try (ReferenceCountedDB meta = BlockUtils.getDB(containerData, conf)) { + // # of blocks to delete is throttled + KeyPrefixFilter filter = + new KeyPrefixFilter().addFilter(OzoneConsts.DELETING_KEY_PREFIX); + List> toDeleteBlocks = + meta.getStore().getSequentialRangeKVs(null, blockLimitPerTask, + filter); + if (toDeleteBlocks.isEmpty()) { + LOG.debug("No under deletion block found in container : {}", + containerData.getContainerID()); + } - List succeedBlocks = new LinkedList<>(); - LOG.debug("Container : {}, To-Delete blocks : {}", - containerData.getContainerID(), toDeleteBlocks.size()); - File dataDir = new File(containerData.getChunksPath()); - if (!dataDir.exists() || !dataDir.isDirectory()) { - LOG.error("Invalid container data dir {} : " - + "does not exist or not a directory", dataDir.getAbsolutePath()); - return crr; - } + List succeedBlocks = new LinkedList<>(); + LOG.debug("Container : {}, To-Delete blocks : {}", + containerData.getContainerID(), toDeleteBlocks.size()); + File dataDir = new File(containerData.getChunksPath()); + if (!dataDir.exists() || !dataDir.isDirectory()) { + LOG.error("Invalid container data dir {} : " + + "does not exist or not a directory", dataDir.getAbsolutePath()); + return crr; + } - toDeleteBlocks.forEach(entry -> { - String blockName = DFSUtil.bytes2String(entry.getKey()); - LOG.debug("Deleting block {}", blockName); - try { - ContainerProtos.BlockData data = - ContainerProtos.BlockData.parseFrom(entry.getValue()); - for (ContainerProtos.ChunkInfo chunkInfo : data.getChunksList()) { - File chunkFile = dataDir.toPath() - .resolve(chunkInfo.getChunkName()).toFile(); - if (FileUtils.deleteQuietly(chunkFile)) { - LOG.debug("block {} chunk {} deleted", blockName, - chunkFile.getAbsolutePath()); + toDeleteBlocks.forEach(entry -> { + String blockName = DFSUtil.bytes2String(entry.getKey()); + LOG.debug("Deleting block {}", blockName); + try { + ContainerProtos.BlockData data = + ContainerProtos.BlockData.parseFrom(entry.getValue()); + for (ContainerProtos.ChunkInfo chunkInfo : data.getChunksList()) { + File chunkFile = dataDir.toPath() + .resolve(chunkInfo.getChunkName()).toFile(); + if (FileUtils.deleteQuietly(chunkFile)) { + if (LOG.isDebugEnabled()) { + LOG.debug("block {} chunk {} deleted", blockName, + chunkFile.getAbsolutePath()); + } + } } + succeedBlocks.add(blockName); + } catch (InvalidProtocolBufferException e) { + LOG.error("Failed to parse block info for block {}", blockName, e); } - succeedBlocks.add(blockName); - } catch (InvalidProtocolBufferException e) { - LOG.error("Failed to parse block info for block {}", blockName, e); + }); + + // Once files are deleted... replace deleting entries with deleted + // entries + BatchOperation batch = new BatchOperation(); + succeedBlocks.forEach(entry -> { + String blockId = + entry.substring(OzoneConsts.DELETING_KEY_PREFIX.length()); + String deletedEntry = OzoneConsts.DELETED_KEY_PREFIX + blockId; + batch.put(DFSUtil.string2Bytes(deletedEntry), + DFSUtil.string2Bytes(blockId)); + batch.delete(DFSUtil.string2Bytes(entry)); + }); + meta.getStore().writeBatch(batch); + // update count of pending deletion blocks in in-memory container status + containerData.decrPendingDeletionBlocks(succeedBlocks.size()); + + if (!succeedBlocks.isEmpty()) { + LOG.info("Container: {}, deleted blocks: {}, task elapsed time: {}ms", + containerData.getContainerID(), succeedBlocks.size(), + Time.monotonicNow() - startTime); } - }); - - // Once files are deleted... replace deleting entries with deleted entries - BatchOperation batch = new BatchOperation(); - succeedBlocks.forEach(entry -> { - String blockId = - entry.substring(OzoneConsts.DELETING_KEY_PREFIX.length()); - String deletedEntry = OzoneConsts.DELETED_KEY_PREFIX + blockId; - batch.put(DFSUtil.string2Bytes(deletedEntry), - DFSUtil.string2Bytes(blockId)); - batch.delete(DFSUtil.string2Bytes(entry)); - }); - meta.writeBatch(batch); - // update count of pending deletion blocks in in-memory container status - containerData.decrPendingDeletionBlocks(succeedBlocks.size()); - - if (!succeedBlocks.isEmpty()) { - LOG.info("Container: {}, deleted blocks: {}, task elapsed time: {}ms", - containerData.getContainerID(), succeedBlocks.size(), - Time.monotonicNow() - startTime); + crr.addAll(succeedBlocks); + return crr; + } finally { + container.writeUnlock(); } - crr.addAll(succeedBlocks); - return crr; } @Override diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerController.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerController.java index 3c0f25164945b..8bbdec96695e2 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerController.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerController.java @@ -26,10 +26,13 @@ import org.apache.hadoop.ozone.container.common.impl.ContainerSet; import org.apache.hadoop.ozone.container.common.interfaces.Container; import org.apache.hadoop.ozone.container.common.interfaces.Handler; +import org.apache.hadoop.ozone.container.common.volume.HddsVolume; import org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker; -import java.io.FileInputStream; import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.util.Iterator; import java.util.Map; /** @@ -71,6 +74,18 @@ public void markContainerForClose(final long containerId) } } + /** + * Marks the container as UNHEALTHY. + * + * @param containerId Id of the container to update + * @throws IOException in case of exception + */ + public void markContainerUnhealthy(final long containerId) + throws IOException { + Container container = containerSet.getContainer(containerId); + getHandler(container).markContainerUnhealthy(container); + } + /** * Returns the container report. * @@ -106,24 +121,32 @@ public void closeContainer(final long containerId) throws IOException { public Container importContainer(final ContainerType type, final long containerId, final long maxSize, final String originPipelineId, - final String originNodeId, final FileInputStream rawContainerStream, + final String originNodeId, final InputStream rawContainerStream, final TarContainerPacker packer) throws IOException { return handlers.get(type).importContainer(containerId, maxSize, originPipelineId, originNodeId, rawContainerStream, packer); } + public void exportContainer(final ContainerType type, + final long containerId, final OutputStream outputStream, + final TarContainerPacker packer) throws IOException { + handlers.get(type).exportContainer( + containerSet.getContainer(containerId), outputStream, packer); + } + /** * Deletes a container given its Id. * @param containerId Id of the container to be deleted * @param force if this is set to true, we delete container without checking * state of the container. - * @throws IOException */ public void deleteContainer(final long containerId, boolean force) throws IOException { final Container container = containerSet.getContainer(containerId); - getHandler(container).deleteContainer(container, force); + if (container != null) { + getHandler(container).deleteContainer(container, force); + } } /** @@ -135,4 +158,20 @@ public void deleteContainer(final long containerId, boolean force) private Handler getHandler(final Container container) { return handlers.get(container.getContainerType()); } + + public Iterator> getContainers() { + return containerSet.getContainerIterator(); + } + + /** + * Return an iterator of containers which are associated with the specified + * volume. + * + * @param volume the HDDS volume which should be used to filter containers + * @return {@literal Iterator} + */ + public Iterator> getContainers(HddsVolume volume) { + return containerSet.getContainerIterator(volume); + } + } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerDataScanner.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerDataScanner.java new file mode 100644 index 0000000000000..1141951dcc00f --- /dev/null +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerDataScanner.java @@ -0,0 +1,178 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ozone.container.ozoneimpl; + +import java.io.IOException; +import java.util.Iterator; +import java.util.concurrent.TimeUnit; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.hdfs.util.Canceler; +import org.apache.hadoop.hdfs.util.DataTransferThrottler; +import org.apache.hadoop.ozone.container.common.interfaces.Container; +import org.apache.hadoop.ozone.container.common.volume.HddsVolume; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * VolumeScanner scans a single volume. Each VolumeScanner has its own thread. + *

    They are all managed by the DataNode's BlockScanner. + */ +public class ContainerDataScanner extends Thread { + public static final Logger LOG = + LoggerFactory.getLogger(ContainerDataScanner.class); + + /** + * The volume that we're scanning. + */ + private final HddsVolume volume; + private final ContainerController controller; + private final DataTransferThrottler throttler; + private final Canceler canceler; + private final ContainerDataScrubberMetrics metrics; + private final long dataScanInterval; + + /** + * True if the thread is stopping.

    + * Protected by this object's lock. + */ + private volatile boolean stopping = false; + + + public ContainerDataScanner(ContainerScrubberConfiguration conf, + ContainerController controller, + HddsVolume volume) { + this.controller = controller; + this.volume = volume; + dataScanInterval = conf.getDataScanInterval(); + throttler = new HddsDataTransferThrottler(conf.getBandwidthPerVolume()); + canceler = new Canceler(); + metrics = ContainerDataScrubberMetrics.create(volume.toString()); + setName("ContainerDataScanner(" + volume + ")"); + setDaemon(true); + } + + @Override + public void run() { + if (LOG.isTraceEnabled()) { + LOG.trace("{}: thread starting.", this); + } + try { + while (!stopping) { + runIteration(); + metrics.resetNumContainersScanned(); + metrics.resetNumUnhealthyContainers(); + } + LOG.info("{} exiting.", this); + } catch (Throwable e) { + LOG.error("{} exiting because of exception ", this, e); + } finally { + if (metrics != null) { + metrics.unregister(); + } + } + } + + @VisibleForTesting + public void runIteration() { + long startTime = System.nanoTime(); + Iterator> itr = controller.getContainers(volume); + while (!stopping && itr.hasNext()) { + Container c = itr.next(); + if (c.shouldScanData()) { + try { + if (!c.scanData(throttler, canceler)) { + metrics.incNumUnHealthyContainers(); + controller.markContainerUnhealthy( + c.getContainerData().getContainerID()); + } + } catch (IOException ex) { + long containerId = c.getContainerData().getContainerID(); + LOG.warn("Unexpected exception while scanning container " + + containerId, ex); + } finally { + metrics.incNumContainersScanned(); + } + } + } + long totalDuration = System.nanoTime() - startTime; + if (!stopping) { + if (metrics.getNumContainersScanned() > 0) { + metrics.incNumScanIterations(); + LOG.info("Completed an iteration of container data scrubber in" + + " {} minutes." + + " Number of iterations (since the data-node restart) : {}" + + ", Number of containers scanned in this iteration : {}" + + ", Number of unhealthy containers found in this iteration : {}", + TimeUnit.NANOSECONDS.toMinutes(totalDuration), + metrics.getNumScanIterations(), + metrics.getNumContainersScanned(), + metrics.getNumUnHealthyContainers()); + } + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(totalDuration); + long remainingSleep = dataScanInterval - elapsedMillis; + if (remainingSleep > 0) { + try { + Thread.sleep(remainingSleep); + } catch (InterruptedException ignored) { + } + } + } + } + + public synchronized void shutdown() { + this.stopping = true; + this.canceler.cancel("ContainerDataScanner("+volume+") is shutting down"); + this.interrupt(); + try { + this.join(); + } catch (InterruptedException ex) { + LOG.warn("Unexpected exception while stopping data scanner for volume " + + volume, ex); + } + } + + @VisibleForTesting + public ContainerDataScrubberMetrics getMetrics() { + return metrics; + } + + @Override + public String toString() { + return "ContainerDataScanner(" + volume + + ", " + volume.getStorageID() + ")"; + } + + private class HddsDataTransferThrottler extends DataTransferThrottler { + HddsDataTransferThrottler(long bandwidthPerSec) { + super(bandwidthPerSec); + } + + @Override + public synchronized void throttle(long numOfBytes) { + ContainerDataScanner.this.metrics.incNumBytesScanned(numOfBytes); + super.throttle(numOfBytes); + } + + @Override + public synchronized void throttle(long numOfBytes, Canceler c) { + ContainerDataScanner.this.metrics.incNumBytesScanned(numOfBytes); + super.throttle(numOfBytes, c); + } + } +} diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerDataScrubberMetrics.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerDataScrubberMetrics.java new file mode 100644 index 0000000000000..3cf4f588322a6 --- /dev/null +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerDataScrubberMetrics.java @@ -0,0 +1,113 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ozone.container.ozoneimpl; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.metrics2.MetricsSystem; +import org.apache.hadoop.metrics2.annotation.Metric; +import org.apache.hadoop.metrics2.annotation.Metrics; +import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; +import org.apache.hadoop.metrics2.lib.MutableCounterInt; +import org.apache.hadoop.metrics2.lib.MutableGaugeInt; +import org.apache.hadoop.metrics2.lib.MutableRate; + +import java.util.concurrent.ThreadLocalRandom; + +/** + * This class captures the container data scrubber metrics on the data-node. + **/ +@InterfaceAudience.Private +@Metrics(about="DataNode container data scrubber metrics", context="dfs") +public final class ContainerDataScrubberMetrics { + private final String name; + private final MetricsSystem ms; + @Metric("number of containers scanned in the current iteration") + private MutableGaugeInt numContainersScanned; + @Metric("number of unhealthy containers found in the current iteration") + private MutableGaugeInt numUnHealthyContainers; + @Metric("number of iterations of scanner completed since the restart") + private MutableCounterInt numScanIterations; + @Metric("disk bandwidth used by the container data scrubber per volume") + private MutableRate numBytesScanned; + + public int getNumContainersScanned() { + return numContainersScanned.value(); + } + + public void incNumContainersScanned() { + numContainersScanned.incr(); + } + + public void resetNumContainersScanned() { + numContainersScanned.decr(getNumContainersScanned()); + } + + public int getNumUnHealthyContainers() { + return numUnHealthyContainers.value(); + } + + public void incNumUnHealthyContainers() { + numUnHealthyContainers.incr(); + } + + public void resetNumUnhealthyContainers() { + numUnHealthyContainers.decr(getNumUnHealthyContainers()); + } + + public int getNumScanIterations() { + return numScanIterations.value(); + } + + public void incNumScanIterations() { + numScanIterations.incr(); + } + + public double getNumBytesScannedMean() { + return numBytesScanned.lastStat().mean(); + } + + public long getNumBytesScannedSampleCount() { + return numBytesScanned.lastStat().numSamples(); + } + + public double getNumBytesScannedStdDev() { + return numBytesScanned.lastStat().stddev(); + } + + public void incNumBytesScanned(long bytes) { + numBytesScanned.add(bytes); + } + + public void unregister() { + ms.unregisterSource(name); + } + + private ContainerDataScrubberMetrics(String name, MetricsSystem ms) { + this.name = name; + this.ms = ms; + } + + public static ContainerDataScrubberMetrics create(final String volumeName) { + MetricsSystem ms = DefaultMetricsSystem.instance(); + String name = "ContainerDataScrubberMetrics-"+ (volumeName.isEmpty() + ? "UndefinedDataNodeVolume"+ ThreadLocalRandom.current().nextInt() + : volumeName.replace(':', '-')); + + return ms.register(name, null, new ContainerDataScrubberMetrics(name, ms)); + } +} diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerMetadataScanner.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerMetadataScanner.java new file mode 100644 index 0000000000000..46aaf73a12dd0 --- /dev/null +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerMetadataScanner.java @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ozone.container.ozoneimpl; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.ozone.container.common.interfaces.Container; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.Iterator; +import java.util.concurrent.TimeUnit; + +/** + * This class is responsible to perform metadata verification of the + * containers. + */ +public class ContainerMetadataScanner extends Thread { + public static final Logger LOG = + LoggerFactory.getLogger(ContainerMetadataScanner.class); + + private final ContainerController controller; + private final long metadataScanInterval; + private final ContainerMetadataScrubberMetrics metrics; + /** + * True if the thread is stopping.

    + * Protected by this object's lock. + */ + private boolean stopping = false; + + public ContainerMetadataScanner(ContainerScrubberConfiguration conf, + ContainerController controller) { + this.controller = controller; + this.metadataScanInterval = conf.getMetadataScanInterval(); + this.metrics = ContainerMetadataScrubberMetrics.create(); + setName("ContainerMetadataScanner"); + setDaemon(true); + } + + @Override + public void run() { + /* + * the outer daemon loop exits on shutdown() + */ + LOG.info("Background ContainerMetadataScanner starting up"); + while (!stopping) { + runIteration(); + if(!stopping) { + metrics.resetNumUnhealthyContainers(); + metrics.resetNumContainersScanned(); + } + } + } + + @VisibleForTesting + void runIteration() { + long start = System.nanoTime(); + Iterator> containerIt = controller.getContainers(); + while (!stopping && containerIt.hasNext()) { + Container container = containerIt.next(); + try { + scrub(container); + } catch (IOException e) { + LOG.info("Unexpected error while scrubbing container {}", + container.getContainerData().getContainerID()); + } finally { + metrics.incNumContainersScanned(); + } + } + long interval = System.nanoTime()-start; + if (!stopping) { + metrics.incNumScanIterations(); + LOG.info("Completed an iteration of container metadata scrubber in" + + " {} minutes." + + " Number of iterations (since the data-node restart) : {}" + + ", Number of containers scanned in this iteration : {}" + + ", Number of unhealthy containers found in this iteration : {}", + TimeUnit.NANOSECONDS.toMinutes(interval), + metrics.getNumScanIterations(), + metrics.getNumContainersScanned(), + metrics.getNumUnHealthyContainers()); + // ensure to delay next metadata scan with respect to user config. + if (interval < metadataScanInterval) { + try { + Thread.sleep(metadataScanInterval - interval); + } catch (InterruptedException e) { + LOG.info("Background ContainerMetadataScanner interrupted." + + " Going to exit"); + } + } + } + } + + @VisibleForTesting + public void scrub(Container container) throws IOException { + if (!container.scanMetaData()) { + metrics.incNumUnHealthyContainers(); + controller.markContainerUnhealthy( + container.getContainerData().getContainerID()); + } + } + + @VisibleForTesting + public ContainerMetadataScrubberMetrics getMetrics() { + return metrics; + } + + public synchronized void shutdown() { + this.stopping = true; + this.interrupt(); + try { + this.join(); + } catch (InterruptedException ex) { + LOG.warn("Unexpected exception while stopping metadata scanner.", ex); + } + } +} diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerMetadataScrubberMetrics.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerMetadataScrubberMetrics.java new file mode 100644 index 0000000000000..3effc351b0051 --- /dev/null +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerMetadataScrubberMetrics.java @@ -0,0 +1,92 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ozone.container.ozoneimpl; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.metrics2.MetricsSystem; +import org.apache.hadoop.metrics2.annotation.Metric; +import org.apache.hadoop.metrics2.annotation.Metrics; +import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; +import org.apache.hadoop.metrics2.lib.MutableCounterInt; +import org.apache.hadoop.metrics2.lib.MutableGaugeInt; + +/** + * This class captures the container meta-data scrubber metrics on the + * data-node. + **/ +@InterfaceAudience.Private +@Metrics(about="DataNode container data scrubber metrics", context="dfs") +public final class ContainerMetadataScrubberMetrics { + private final String name; + private final MetricsSystem ms; + @Metric("number of containers scanned in the current iteration") + private MutableGaugeInt numContainersScanned; + @Metric("number of unhealthy containers found in the current iteration") + private MutableGaugeInt numUnHealthyContainers; + @Metric("number of iterations of scanner completed since the restart") + private MutableCounterInt numScanIterations; + + public int getNumContainersScanned() { + return numContainersScanned.value(); + } + + public void incNumContainersScanned() { + numContainersScanned.incr(); + } + + public void resetNumContainersScanned() { + numContainersScanned.decr(getNumContainersScanned()); + } + + public int getNumUnHealthyContainers() { + return numUnHealthyContainers.value(); + } + + public void incNumUnHealthyContainers() { + numUnHealthyContainers.incr(); + } + + public void resetNumUnhealthyContainers() { + numUnHealthyContainers.decr(getNumUnHealthyContainers()); + } + + public int getNumScanIterations() { + return numScanIterations.value(); + } + + public void incNumScanIterations() { + numScanIterations.incr(); + } + + public void unregister() { + ms.unregisterSource(name); + } + + private ContainerMetadataScrubberMetrics(String name, MetricsSystem ms) { + this.name = name; + this.ms = ms; + } + + public static ContainerMetadataScrubberMetrics create() { + MetricsSystem ms = DefaultMetricsSystem.instance(); + String name = "ContainerMetadataScrubberMetrics"; + return ms.register(name, null, + new ContainerMetadataScrubberMetrics(name, ms)); + } + +} diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java index 0192fd5dd1b57..621da70735df2 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java @@ -27,24 +27,28 @@ import org.apache.hadoop.hdfs.DFSUtil; import org.apache.hadoop.ozone.OzoneConsts; import org.apache.hadoop.ozone.common.Storage; +import org.apache.hadoop.ozone.container.common.helpers.BlockData; +import org.apache.hadoop.ozone.container.common.helpers.ChunkInfo; import org.apache.hadoop.ozone.container.common.helpers.ContainerUtils; import org.apache.hadoop.ozone.container.common.impl.ContainerData; import org.apache.hadoop.ozone.container.common.impl.ContainerSet; import org.apache.hadoop.ozone.container.common.volume.HddsVolume; import org.apache.hadoop.ozone.container.common.volume.VolumeSet; +import org.apache.hadoop.ozone.container.keyvalue.KeyValueBlockIterator; import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer; import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerData; import org.apache.hadoop.ozone.container.common.impl.ContainerDataYaml; import org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils; import org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil; -import org.apache.hadoop.utils.MetadataKeyFilters; -import org.apache.hadoop.utils.MetadataStore; +import org.apache.hadoop.hdds.utils.MetadataKeyFilters; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.io.File; import java.io.FileFilter; import java.io.IOException; +import java.util.List; /** * Class used to read .container files from Volume and build container map. @@ -161,14 +165,21 @@ private void verifyContainerFile(long containerID, File containerFile) { "Skipping loading of this container.", containerFile); return; } - verifyContainerData(containerData); + verifyAndFixupContainerData(containerData); } catch (IOException ex) { LOG.error("Failed to parse ContainerFile for ContainerID: {}", containerID, ex); } } - public void verifyContainerData(ContainerData containerData) + /** + * verify ContainerData loaded from disk and fix-up stale members. + * Specifically blockCommitSequenceId, delete related metadata + * and bytesUsed + * @param containerData + * @throws IOException + */ + public void verifyAndFixupContainerData(ContainerData containerData) throws IOException { switch (containerData.getContainerType()) { case KeyValueContainer: @@ -180,28 +191,36 @@ public void verifyContainerData(ContainerData containerData) KeyValueContainerUtil.parseKVContainerData(kvContainerData, config); KeyValueContainer kvContainer = new KeyValueContainer( kvContainerData, config); - MetadataStore containerDB = BlockUtils.getDB(kvContainerData, config); - MetadataKeyFilters.KeyPrefixFilter filter = - new MetadataKeyFilters.KeyPrefixFilter() - .addFilter(OzoneConsts.DELETING_KEY_PREFIX); - int numPendingDeletionBlocks = - containerDB.getSequentialRangeKVs(null, Integer.MAX_VALUE, filter) - .size(); - kvContainerData.incrPendingDeletionBlocks(numPendingDeletionBlocks); - byte[] delTxnId = containerDB.get( - DFSUtil.string2Bytes(OzoneConsts.DELETE_TRANSACTION_KEY_PREFIX)); - if (delTxnId != null) { - kvContainerData - .updateDeleteTransactionId(Longs.fromByteArray(delTxnId)); - } - // sets the BlockCommitSequenceId. - byte[] bcsId = containerDB.get( - DFSUtil.string2Bytes(OzoneConsts.BLOCK_COMMIT_SEQUENCE_ID_PREFIX)); - if (bcsId != null) { - kvContainerData - .updateBlockCommitSequenceId(Longs.fromByteArray(bcsId)); + try(ReferenceCountedDB containerDB = BlockUtils.getDB(kvContainerData, + config)) { + MetadataKeyFilters.KeyPrefixFilter filter = + new MetadataKeyFilters.KeyPrefixFilter() + .addFilter(OzoneConsts.DELETING_KEY_PREFIX); + int numPendingDeletionBlocks = + containerDB.getStore().getSequentialRangeKVs(null, + Integer.MAX_VALUE, filter) + .size(); + kvContainerData.incrPendingDeletionBlocks(numPendingDeletionBlocks); + byte[] delTxnId = containerDB.getStore().get( + DFSUtil.string2Bytes(OzoneConsts.DELETE_TRANSACTION_KEY_PREFIX)); + if (delTxnId != null) { + kvContainerData + .updateDeleteTransactionId(Longs.fromByteArray(delTxnId)); + } + // sets the BlockCommitSequenceId. + byte[] bcsId = containerDB.getStore().get(DFSUtil.string2Bytes( + OzoneConsts.BLOCK_COMMIT_SEQUENCE_ID_PREFIX)); + if (bcsId != null) { + kvContainerData + .updateBlockCommitSequenceId(Longs.fromByteArray(bcsId)); + } + if (kvContainer.getContainerState() + == ContainerProtos.ContainerDataProto.State.OPEN) { + // commitSpace for Open Containers relies on usedBytes + initializeUsedBytes(kvContainer); + } + containerSet.addContainer(kvContainer); } - containerSet.addContainer(kvContainer); } else { throw new StorageContainerException("Container File is corrupted. " + "ContainerType is KeyValueContainer but cast to " + @@ -215,4 +234,28 @@ public void verifyContainerData(ContainerData containerData) ContainerProtos.Result.UNKNOWN_CONTAINER_TYPE); } } + + private void initializeUsedBytes(KeyValueContainer container) + throws IOException { + try (KeyValueBlockIterator blockIter = new KeyValueBlockIterator( + container.getContainerData().getContainerID(), + new File(container.getContainerData().getContainerPath()))) { + long usedBytes = 0; + + while (blockIter.hasNext()) { + BlockData block = blockIter.nextBlock(); + long blockLen = 0; + + List chunkInfoList = block.getChunks(); + for (ContainerProtos.ChunkInfo chunk : chunkInfoList) { + ChunkInfo info = ChunkInfo.getFromProtoBuf(chunk); + blockLen += info.getLen(); + } + + usedBytes += blockLen; + } + + container.getContainerData().setBytesUsed(usedBytes); + } + } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScrubber.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScrubber.java deleted file mode 100644 index dea7323d757fd..0000000000000 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScrubber.java +++ /dev/null @@ -1,158 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.ozone.container.ozoneimpl; - -import com.google.common.base.Preconditions; -import org.apache.commons.net.ntp.TimeStamp; -import org.apache.hadoop.hdds.conf.OzoneConfiguration; -import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException; -import org.apache.hadoop.ozone.container.common.impl.ContainerSet; -import org.apache.hadoop.ozone.container.common.interfaces.Container; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.util.Iterator; - -/** - * Background Metadata scrubbing for Ozone Containers. - * Future scope : data(chunks) checksum verification. - */ -public class ContainerScrubber implements Runnable { - private static final Logger LOG = - LoggerFactory.getLogger(ContainerScrubber.class); - private final ContainerSet containerSet; - private final OzoneConfiguration config; - private final long timePerContainer = 10000; // 10 sec in millis - private boolean halt; - private Thread scrubThread; - - public ContainerScrubber(ContainerSet cSet, OzoneConfiguration conf) { - Preconditions.checkNotNull(cSet, - "ContainerScrubber received a null ContainerSet"); - Preconditions.checkNotNull(conf); - this.containerSet = cSet; - this.config = conf; - this.halt = false; - this.scrubThread = null; - } - - @Override public void run() { - /** - * the outer daemon loop exits on down() - */ - LOG.info("Background ContainerScrubber starting up"); - while (true) { - - scrub(); - - if (this.halt) { - break; // stop and exit if requested - } - - try { - Thread.sleep(300000); /* 5 min between scans */ - } catch (InterruptedException e) { - LOG.info("Background ContainerScrubber interrupted. Going to exit"); - } - } - } - - /** - * Start the scrub scanner thread. - */ - public void up() { - - this.halt = false; - if (this.scrubThread == null) { - this.scrubThread = new Thread(this); - scrubThread.start(); - } else { - LOG.info("Scrubber up called multiple times. Scrub thread already up."); - } - } - - /** - * Stop the scrub scanner thread. Wait for thread to exit - */ - public void down() { - - this.halt = true; - if (scrubThread == null) { - LOG.info("Scrubber down invoked, but scrub thread is not running"); - return; - } - - this.scrubThread.interrupt(); - try { - this.scrubThread.join(); - } catch (Exception e) { - LOG.warn("Exception when waiting for Container Scrubber thread ", e); - } finally { - this.scrubThread = null; - } - } - - /** - * Current implementation : fixed rate scrub, no feedback loop. - * Dynamic throttling based on system load monitoring to be - * implemented later as jira [XXX] - * - * @param startTime - */ - private void throttleScrubber(TimeStamp startTime) { - TimeStamp endTime = new TimeStamp(System.currentTimeMillis()); - long timeTaken = endTime.getTime() - startTime.getTime(); - - if (timeTaken < timePerContainer) { - try { - Thread.sleep(timePerContainer - timeTaken); - } catch (InterruptedException e) { - LOG.debug("Ignoring interrupted sleep inside throttle"); - } - } - } - - private void scrub() { - - Iterator containerIt = containerSet.getContainerIterator(); - long count = 0; - - while (containerIt.hasNext()) { - TimeStamp startTime = new TimeStamp(System.currentTimeMillis()); - Container container = containerIt.next(); - - if (this.halt) { - break; // stop if requested - } - - try { - container.check(); - } catch (StorageContainerException e) { - LOG.error("Error unexpected exception {} for Container {}", e, - container.getContainerData().getContainerID()); - // XXX Action required here - } - count++; - - throttleScrubber(startTime); - } - - LOG.debug("iterator ran integrity checks on {} containers", count); - } -} \ No newline at end of file diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScrubberConfiguration.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScrubberConfiguration.java new file mode 100644 index 0000000000000..454ce84310aaf --- /dev/null +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScrubberConfiguration.java @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ozone.container.ozoneimpl; + +import org.apache.hadoop.hdds.conf.Config; +import org.apache.hadoop.hdds.conf.ConfigGroup; +import org.apache.hadoop.hdds.conf.ConfigTag; +import org.apache.hadoop.hdds.conf.ConfigType; + +/** + * This class defines configuration parameters for container scrubber. + **/ +@ConfigGroup(prefix = "hdds.containerscrub") +public class ContainerScrubberConfiguration { + private boolean enabled; + private long metadataScanInterval; + private long dataScanInterval; + private long bandwidthPerVolume; + + @Config(key = "enabled", + type = ConfigType.BOOLEAN, + defaultValue = "false", + tags = {ConfigTag.STORAGE}, + description = "Config parameter to enable container scrubber.") + public void setEnabled(boolean enabled) { + this.enabled = enabled; + } + + public boolean isEnabled() { + return enabled; + } + + @Config(key = "metadata.scan.interval", + type = ConfigType.TIME, + defaultValue = "3h", + tags = {ConfigTag.STORAGE}, + description = "Config parameter define time interval in milliseconds" + + " between two metadata scans by container scrubber.") + public void setMetadataScanInterval(long metadataScanInterval) { + this.metadataScanInterval = metadataScanInterval; + } + + public long getMetadataScanInterval() { + return metadataScanInterval; + } + + @Config(key = "data.scan.interval", + type = ConfigType.TIME, + defaultValue = "1m", + tags = { ConfigTag.STORAGE }, + description = "Minimum time interval between two iterations of container" + + " data scanning. If an iteration takes less time than this, the" + + " scanner will wait before starting the next iteration." + ) + public void setDataScanInterval(long dataScanInterval) { + this.dataScanInterval = dataScanInterval; + } + + public long getDataScanInterval() { + return dataScanInterval; + } + + @Config(key = "volume.bytes.per.second", + type = ConfigType.LONG, + defaultValue = "1048576", + tags = {ConfigTag.STORAGE}, + description = "Config parameter to throttle I/O bandwidth used" + + " by scrubber per volume.") + public void setBandwidthPerVolume(long bandwidthPerVolume) { + this.bandwidthPerVolume = bandwidthPerVolume; + } + + public long getBandwidthPerVolume() { + return bandwidthPerVolume; + } +} diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java index f34334d1034e4..a026f0e8757b4 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java @@ -20,7 +20,6 @@ import com.google.common.annotations.VisibleForTesting; import com.google.common.collect.Maps; -import org.apache.hadoop.hdds.HddsConfigKeys; import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdds.protocol.DatanodeDetails; import org.apache.hadoop.hdds.protocol.datanode.proto @@ -42,6 +41,7 @@ import org.apache.hadoop.ozone.container.common.volume.HddsVolume; import org.apache.hadoop.ozone.container.common.volume.VolumeSet; +import org.apache.hadoop.ozone.container.keyvalue.statemachine.background.BlockDeletingService; import org.apache.hadoop.ozone.container.replication.GrpcReplicationService; import org.apache.hadoop.ozone.container.replication .OnDemandContainerReplicationSource; @@ -52,7 +52,11 @@ import java.io.*; import java.util.ArrayList; import java.util.Iterator; +import java.util.List; import java.util.Map; +import java.util.concurrent.TimeUnit; + +import static org.apache.hadoop.ozone.OzoneConfigKeys.*; /** * Ozone main class sets up the network servers and initializes the container @@ -71,7 +75,9 @@ public class OzoneContainer { private final XceiverServerSpi writeChannel; private final XceiverServerSpi readChannel; private final ContainerController controller; - private ContainerScrubber scrubber; + private ContainerMetadataScanner metadataScanner; + private List dataScanners; + private final BlockDeletingService blockDeletingService; /** * Construct OzoneContainer object. @@ -87,7 +93,7 @@ public OzoneContainer(DatanodeDetails datanodeDetails, OzoneConfiguration this.config = conf; this.volumeSet = new VolumeSet(datanodeDetails.getUuidString(), conf); this.containerSet = new ContainerSet(); - this.scrubber = null; + this.metadataScanner = null; buildContainerSet(); final ContainerMetrics metrics = ContainerMetrics.create(conf); @@ -107,11 +113,22 @@ public OzoneContainer(DatanodeDetails datanodeDetails, OzoneConfiguration */ this.controller = new ContainerController(containerSet, handlers); this.writeChannel = XceiverServerRatis.newXceiverServerRatis( - datanodeDetails, config, hddsDispatcher, context, certClient); + datanodeDetails, config, hddsDispatcher, controller, certClient, + context); this.readChannel = new XceiverServerGrpc( datanodeDetails, config, hddsDispatcher, certClient, createReplicationService()); - + long svcInterval = config + .getTimeDuration(OZONE_BLOCK_DELETING_SERVICE_INTERVAL, + OZONE_BLOCK_DELETING_SERVICE_INTERVAL_DEFAULT, + TimeUnit.MILLISECONDS); + long serviceTimeout = config + .getTimeDuration(OZONE_BLOCK_DELETING_SERVICE_TIMEOUT, + OZONE_BLOCK_DELETING_SERVICE_TIMEOUT_DEFAULT, + TimeUnit.MILLISECONDS); + this.blockDeletingService = + new BlockDeletingService(this, svcInterval, serviceTimeout, + TimeUnit.MILLISECONDS, config); } private GrpcReplicationService createReplicationService() { @@ -152,18 +169,24 @@ private void buildContainerSet() { * Start background daemon thread for performing container integrity checks. */ private void startContainerScrub() { - boolean enabled = config.getBoolean( - HddsConfigKeys.HDDS_CONTAINERSCRUB_ENABLED, - HddsConfigKeys.HDDS_CONTAINERSCRUB_ENABLED_DEFAULT); + ContainerScrubberConfiguration c = config.getObject( + ContainerScrubberConfiguration.class); + boolean enabled = c.isEnabled(); if (!enabled) { - LOG.info("Background container scrubber has been disabled by {}", - HddsConfigKeys.HDDS_CONTAINERSCRUB_ENABLED); + LOG.info("Background container scanner has been disabled."); } else { - if (this.scrubber == null) { - this.scrubber = new ContainerScrubber(containerSet, config); + if (this.metadataScanner == null) { + this.metadataScanner = new ContainerMetadataScanner(c, controller); + } + this.metadataScanner.start(); + + dataScanners = new ArrayList<>(); + for (HddsVolume v : volumeSet.getVolumesList()) { + ContainerDataScanner s = new ContainerDataScanner(c, controller, v); + s.start(); + dataScanners.add(s); } - scrubber.up(); } } @@ -171,10 +194,14 @@ private void startContainerScrub() { * Stop the scanner thread and wait for thread to die. */ private void stopContainerScrub() { - if (scrubber == null) { + if (metadataScanner == null) { return; } - scrubber.down(); + metadataScanner.shutdown(); + metadataScanner = null; + for (ContainerDataScanner s : dataScanners) { + s.shutdown(); + } } /** @@ -189,6 +216,7 @@ public void start(String scmId) throws IOException { readChannel.start(); hddsDispatcher.init(); hddsDispatcher.setScmId(scmId); + blockDeletingService.start(); } /** @@ -200,8 +228,11 @@ public void stop() { stopContainerScrub(); writeChannel.stop(); readChannel.stop(); + this.handlers.values().forEach(Handler::stop); hddsDispatcher.shutdown(); volumeSet.shutdown(); + blockDeletingService.shutdown(); + ContainerMetrics.remove(); } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/GrpcReplicationClient.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/GrpcReplicationClient.java index 768d2667c1be1..8494a15274424 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/GrpcReplicationClient.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/GrpcReplicationClient.java @@ -24,6 +24,7 @@ import java.nio.file.Files; import java.nio.file.Path; import java.util.concurrent.CompletableFuture; +import java.util.concurrent.TimeUnit; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos .CopyContainerRequestProto; @@ -92,6 +93,11 @@ private Path getWorkingDirectory() { public void shutdown() { channel.shutdown(); + try { + channel.awaitTermination(5, TimeUnit.SECONDS); + } catch (Exception e) { + LOG.error("failed to shutdown replication channel", e); + } } /** diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/OnDemandContainerReplicationSource.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/OnDemandContainerReplicationSource.java index 28b8713aa3b68..d318ffa257f7d 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/OnDemandContainerReplicationSource.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/OnDemandContainerReplicationSource.java @@ -21,7 +21,6 @@ import java.io.OutputStream; import org.apache.hadoop.ozone.container.common.interfaces.Container; -import org.apache.hadoop.ozone.container.common.interfaces.ContainerPacker; import org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker; import com.google.common.base.Preconditions; @@ -41,7 +40,7 @@ public class OnDemandContainerReplicationSource private ContainerController controller; - private ContainerPacker packer = new TarContainerPacker(); + private TarContainerPacker packer = new TarContainerPacker(); public OnDemandContainerReplicationSource( ContainerController controller) { @@ -59,18 +58,11 @@ public void copyData(long containerId, OutputStream destination) Container container = controller.getContainer(containerId); - Preconditions - .checkNotNull(container, "Container is not found " + containerId); + Preconditions.checkNotNull( + container, "Container is not found " + containerId); - switch (container.getContainerType()) { - case KeyValueContainer: - packer.pack(container, - destination); - break; - default: - LOG.warn("Container type " + container.getContainerType() - + " is not replicable as no compression algorithm for that."); - } + controller.exportContainer( + container.getContainerType(), containerId, destination, packer); } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationSupervisor.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationSupervisor.java index c59d643c47b98..7a07c4df71eb5 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationSupervisor.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationSupervisor.java @@ -22,6 +22,7 @@ import java.util.concurrent.LinkedBlockingQueue; import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicLong; import com.google.common.util.concurrent.ThreadFactoryBuilder; import org.apache.hadoop.ozone.container.common.impl.ContainerSet; @@ -42,6 +43,7 @@ public class ReplicationSupervisor { private final ContainerSet containerSet; private final ContainerReplicator replicator; private final ThreadPoolExecutor executor; + private final AtomicLong replicationCounter; /** * A set of container IDs that are currently being downloaded @@ -56,6 +58,7 @@ public ReplicationSupervisor( this.containerSet = containerSet; this.replicator = replicator; this.containersInFlight = ConcurrentHashMap.newKeySet(); + replicationCounter = new AtomicLong(); this.executor = new ThreadPoolExecutor( 0, poolSize, 60, TimeUnit.SECONDS, new LinkedBlockingQueue<>(), @@ -123,7 +126,12 @@ public void run() { } } finally { containersInFlight.remove(task.getContainerId()); + replicationCounter.incrementAndGet(); } } } + + public long getReplicationCounter() { + return replicationCounter.get(); + } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/SimpleContainerDownloader.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/SimpleContainerDownloader.java index 032dc7db9bbc7..37a44acf74c9f 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/SimpleContainerDownloader.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/SimpleContainerDownloader.java @@ -18,15 +18,10 @@ package org.apache.hadoop.ozone.container.replication; -import java.io.IOException; import java.nio.file.Path; import java.nio.file.Paths; import java.util.List; import java.util.concurrent.CompletableFuture; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.concurrent.ThreadFactory; -import java.util.concurrent.TimeUnit; import java.util.function.Function; import org.apache.hadoop.conf.Configuration; @@ -34,7 +29,6 @@ import org.apache.hadoop.hdds.protocol.DatanodeDetails.Port.Name; import org.apache.hadoop.ozone.OzoneConfigKeys; -import com.google.common.util.concurrent.ThreadFactoryBuilder; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -52,8 +46,6 @@ public class SimpleContainerDownloader implements ContainerDownloader { private final Path workingDirectory; - private ExecutorService executor; - public SimpleContainerDownloader(Configuration conf) { String workDirString = @@ -65,12 +57,6 @@ public SimpleContainerDownloader(Configuration conf) { } else { workingDirectory = Paths.get(workDirString); } - - ThreadFactory build = new ThreadFactoryBuilder().setDaemon(true) - .setNameFormat("Container downloader thread - %d").build(); - executor = Executors.newSingleThreadExecutor(build); - LOG.info("Starting container downloader service to copy " - + "containers to replicate."); } @Override @@ -110,11 +96,7 @@ public CompletableFuture getContainerDataFromReplicas(long containerId, } @Override - public void close() throws IOException { - try { - executor.awaitTermination(10, TimeUnit.SECONDS); - } catch (InterruptedException e) { - LOG.error("Can't stop container downloader gracefully", e); - } + public void close() { + // noop } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/RegisteredCommand.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/RegisteredCommand.java index 3a5da72f482a5..42778cb6e4959 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/RegisteredCommand.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/RegisteredCommand.java @@ -17,7 +17,8 @@ */ package org.apache.hadoop.ozone.protocol.commands; -import com.google.common.base.Preconditions; +import com.google.common.base.Strings; +import org.apache.hadoop.hdds.protocol.DatanodeDetails; import org.apache.hadoop.hdds.protocol.proto .StorageContainerDatanodeProtocolProtos.SCMRegisteredResponseProto; import org.apache.hadoop.hdds.protocol.proto @@ -28,23 +29,15 @@ * Response to Datanode Register call. */ public class RegisteredCommand { - private String datanodeUUID; private String clusterID; private ErrorCode error; - private String hostname; - private String ipAddress; + private DatanodeDetails datanode; - public RegisteredCommand(final ErrorCode error, final String datanodeUUID, + public RegisteredCommand(final ErrorCode error, final DatanodeDetails node, final String clusterID) { - this(error, datanodeUUID, clusterID, null, null); - } - public RegisteredCommand(final ErrorCode error, final String datanodeUUID, - final String clusterID, final String hostname, final String ipAddress) { - this.datanodeUUID = datanodeUUID; + this.datanode = node; this.clusterID = clusterID; this.error = error; - this.hostname = hostname; - this.ipAddress = ipAddress; } /** @@ -57,12 +50,12 @@ public static Builder newBuilder() { } /** - * Returns datanode UUID. + * Returns datanode. * - * @return - Datanode ID. + * @return - Datanode. */ - public String getDatanodeUUID() { - return datanodeUUID; + public DatanodeDetails getDatanode() { + return datanode; } /** @@ -83,79 +76,54 @@ public ErrorCode getError() { return error; } - /** - * Returns the hostname. - * - * @return - hostname - */ - public String getHostName() { - return hostname; - } - - /** - * Returns the ipAddress of the dataNode. - */ - public String getIpAddress() { - return ipAddress; - } - /** * Gets the protobuf message of this object. * * @return A protobuf message. */ - public byte[] getProtoBufMessage() { + public SCMRegisteredResponseProto getProtoBufMessage() { SCMRegisteredResponseProto.Builder builder = SCMRegisteredResponseProto.newBuilder() + // TODO : Fix this later when we have multiple SCM support. + // .setAddressList(addressList) .setClusterID(this.clusterID) - .setDatanodeUUID(this.datanodeUUID) + .setDatanodeUUID(this.datanode.getUuidString()) .setErrorCode(this.error); - if (hostname != null && ipAddress != null) { - builder.setHostname(hostname).setIpAddress(ipAddress); + if (!Strings.isNullOrEmpty(datanode.getHostName())) { + builder.setHostname(datanode.getHostName()); + } + if (!Strings.isNullOrEmpty(datanode.getIpAddress())) { + builder.setIpAddress(datanode.getIpAddress()); + } + if (!Strings.isNullOrEmpty(datanode.getNetworkName())) { + builder.setNetworkName(datanode.getNetworkName()); } - return builder.build().toByteArray(); + if (!Strings.isNullOrEmpty(datanode.getNetworkLocation())) { + builder.setNetworkLocation(datanode.getNetworkLocation()); + } + + return builder.build(); } /** * A builder class to verify all values are sane. */ public static class Builder { - private String datanodeUUID; + private DatanodeDetails datanode; private String clusterID; private ErrorCode error; - private String ipAddress; - private String hostname; /** - * sets UUID. + * sets datanode details. * - * @param dnUUID - datanode UUID + * @param node - datanode details * @return Builder */ - public Builder setDatanodeUUID(String dnUUID) { - this.datanodeUUID = dnUUID; + public Builder setDatanode(DatanodeDetails node) { + this.datanode = node; return this; } - /** - * Create this object from a Protobuf message. - * - * @param response - RegisteredCmdResponseProto - * @return RegisteredCommand - */ - public RegisteredCommand getFromProtobuf(SCMRegisteredResponseProto - response) { - Preconditions.checkNotNull(response); - if (response.hasHostname() && response.hasIpAddress()) { - return new RegisteredCommand(response.getErrorCode(), - response.getDatanodeUUID(), response.getClusterID(), - response.getHostname(), response.getIpAddress()); - } else { - return new RegisteredCommand(response.getErrorCode(), - response.getDatanodeUUID(), response.getClusterID()); - } - } - /** * Sets cluster ID. * @@ -178,38 +146,19 @@ public Builder setErrorCode(ErrorCode errorCode) { return this; } - /** - * sets the hostname. - */ - public Builder setHostname(String host) { - this.hostname = host; - return this; - } - - public Builder setIpAddress(String ipAddr) { - this.ipAddress = ipAddr; - return this; - } - /** * Build the command object. * * @return RegisteredCommand */ public RegisteredCommand build() { - if ((this.error == ErrorCode.success) && (this.datanodeUUID == null - || this.datanodeUUID.isEmpty()) || (this.clusterID == null - || this.clusterID.isEmpty())) { + if ((this.error == ErrorCode.success) && (this.datanode == null + || Strings.isNullOrEmpty(this.datanode.getUuidString()) + || Strings.isNullOrEmpty(this.clusterID))) { throw new IllegalArgumentException("On success, RegisteredCommand " + "needs datanodeUUID and ClusterID."); } - if (hostname != null && ipAddress != null) { - return new RegisteredCommand(this.error, this.datanodeUUID, - this.clusterID, this.hostname, this.ipAddress); - } else { - return new RegisteredCommand(this.error, this.datanodeUUID, - this.clusterID); - } + return new RegisteredCommand(this.error, this.datanode, this.clusterID); } } } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerDatanodeProtocolClientSideTranslatorPB.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerDatanodeProtocolClientSideTranslatorPB.java index 4e1e27e180172..9b446666e5d11 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerDatanodeProtocolClientSideTranslatorPB.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerDatanodeProtocolClientSideTranslatorPB.java @@ -24,6 +24,9 @@ import org.apache.hadoop.hdds.protocol.proto .StorageContainerDatanodeProtocolProtos.ContainerReportsProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.SCMDatanodeRequest; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.SCMDatanodeRequest.Builder; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.SCMDatanodeResponse; import org.apache.hadoop.hdds.protocol.proto .StorageContainerDatanodeProtocolProtos.SCMHeartbeatRequestProto; import org.apache.hadoop.hdds.protocol.proto @@ -38,6 +41,7 @@ .StorageContainerDatanodeProtocolProtos.SCMVersionRequestProto; import org.apache.hadoop.hdds.protocol.proto .StorageContainerDatanodeProtocolProtos.SCMVersionResponseProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.Type; import org.apache.hadoop.ipc.ProtobufHelper; import org.apache.hadoop.ipc.ProtocolTranslator; import org.apache.hadoop.ipc.RPC; @@ -45,6 +49,7 @@ import java.io.Closeable; import java.io.IOException; +import java.util.function.Consumer; /** * This class is the client-side translator to translate the requests made on @@ -96,6 +101,25 @@ public Object getUnderlyingProxyObject() { return rpcProxy; } + /** + * Helper method to wrap the request and send the message. + */ + private SCMDatanodeResponse submitRequest(Type type, + Consumer builderConsumer) throws IOException { + final SCMDatanodeResponse response; + try { + Builder builder = SCMDatanodeRequest.newBuilder() + .setCmdType(type); + builderConsumer.accept(builder); + SCMDatanodeRequest wrapper = builder.build(); + + response = rpcProxy.submitRequest(NULL_RPC_CONTROLLER, wrapper); + } catch (ServiceException ex) { + throw ProtobufHelper.getRemoteException(ex); + } + return response; + } + /** * Returns SCM version. * @@ -104,16 +128,11 @@ public Object getUnderlyingProxyObject() { */ @Override public SCMVersionResponseProto getVersion(SCMVersionRequestProto - unused) throws IOException { - SCMVersionRequestProto request = - SCMVersionRequestProto.newBuilder().build(); - final SCMVersionResponseProto response; - try { - response = rpcProxy.getVersion(NULL_RPC_CONTROLLER, request); - } catch (ServiceException ex) { - throw ProtobufHelper.getRemoteException(ex); - } - return response; + request) throws IOException { + return submitRequest(Type.GetVersion, + (builder) -> builder + .setGetVersionRequest(SCMVersionRequestProto.newBuilder().build())) + .getGetVersionResponse(); } /** @@ -126,13 +145,9 @@ public SCMVersionResponseProto getVersion(SCMVersionRequestProto @Override public SCMHeartbeatResponseProto sendHeartbeat( SCMHeartbeatRequestProto heartbeat) throws IOException { - final SCMHeartbeatResponseProto resp; - try { - resp = rpcProxy.sendHeartbeat(NULL_RPC_CONTROLLER, heartbeat); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } - return resp; + return submitRequest(Type.SendHeartbeat, + (builder) -> builder.setSendHeartbeatRequest(heartbeat)) + .getSendHeartbeatResponse(); } /** @@ -155,13 +170,8 @@ public SCMRegisteredResponseProto register( req.setContainerReport(containerReportsRequestProto); req.setPipelineReports(pipelineReportsProto); req.setNodeReport(nodeReport); - final SCMRegisteredResponseProto response; - try { - response = rpcProxy.register(NULL_RPC_CONTROLLER, req.build()); - } catch (ServiceException e) { - throw ProtobufHelper.getRemoteException(e); - } - return response; + return submitRequest(Type.Register, + (builder) -> builder.setRegisterRequest(req)) + .getRegisterResponse(); } - } diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerDatanodeProtocolServerSideTranslatorPB.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerDatanodeProtocolServerSideTranslatorPB.java index 862233276aa2e..ed704ebf4310c 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerDatanodeProtocolServerSideTranslatorPB.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerDatanodeProtocolServerSideTranslatorPB.java @@ -16,29 +16,24 @@ */ package org.apache.hadoop.ozone.protocolPB; -import com.google.protobuf.RpcController; -import com.google.protobuf.ServiceException; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.PipelineReportsProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.NodeReportProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.SCMRegisterRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.SCMRegisteredResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.SCMVersionRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.SCMVersionResponseProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.ContainerReportsProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.SCMHeartbeatRequestProto; -import org.apache.hadoop.hdds.protocol.proto - .StorageContainerDatanodeProtocolProtos.SCMHeartbeatResponseProto; +import java.io.IOException; + +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.ContainerReportsProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.NodeReportProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.PipelineReportsProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.SCMDatanodeRequest; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.SCMDatanodeResponse; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.SCMRegisterRequestProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.SCMRegisteredResponseProto; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.Status; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.Type; +import org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher; import org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol; -import java.io.IOException; +import com.google.protobuf.RpcController; +import com.google.protobuf.ServiceException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** * This class is the server-side translator that forwards requests received on @@ -48,47 +43,71 @@ public class StorageContainerDatanodeProtocolServerSideTranslatorPB implements StorageContainerDatanodeProtocolPB { + private static final Logger LOG = LoggerFactory + .getLogger(StorageContainerDatanodeProtocolServerSideTranslatorPB.class); + private final StorageContainerDatanodeProtocol impl; + private final OzoneProtocolMessageDispatcher dispatcher; public StorageContainerDatanodeProtocolServerSideTranslatorPB( - StorageContainerDatanodeProtocol impl) { + StorageContainerDatanodeProtocol impl, + ProtocolMessageMetrics protocolMessageMetrics) { this.impl = impl; + dispatcher = + new OzoneProtocolMessageDispatcher<>("SCMDatanodeProtocol", + protocolMessageMetrics, + LOG); } - @Override - public SCMVersionResponseProto getVersion(RpcController controller, - SCMVersionRequestProto request) - throws ServiceException { - try { - return impl.getVersion(request); - } catch (IOException e) { - throw new ServiceException(e); - } + public SCMRegisteredResponseProto register( + SCMRegisterRequestProto request) throws IOException { + ContainerReportsProto containerRequestProto = request + .getContainerReport(); + NodeReportProto dnNodeReport = request.getNodeReport(); + PipelineReportsProto pipelineReport = request.getPipelineReports(); + return impl.register(request.getDatanodeDetails(), dnNodeReport, + containerRequestProto, pipelineReport); + } @Override - public SCMRegisteredResponseProto register(RpcController controller, - SCMRegisterRequestProto request) throws ServiceException { - try { - ContainerReportsProto containerRequestProto = request - .getContainerReport(); - NodeReportProto dnNodeReport = request.getNodeReport(); - PipelineReportsProto pipelineReport = request.getPipelineReports(); - return impl.register(request.getDatanodeDetails(), dnNodeReport, - containerRequestProto, pipelineReport); - } catch (IOException e) { - throw new ServiceException(e); - } + public SCMDatanodeResponse submitRequest(RpcController controller, + SCMDatanodeRequest request) throws ServiceException { + return dispatcher.processRequest(request, this::processMessage, + request.getCmdType(), request.getTraceID()); } - @Override - public SCMHeartbeatResponseProto sendHeartbeat(RpcController controller, - SCMHeartbeatRequestProto request) throws ServiceException { + public SCMDatanodeResponse processMessage(SCMDatanodeRequest request) + throws ServiceException { try { - return impl.sendHeartbeat(request); + Type cmdType = request.getCmdType(); + switch (cmdType) { + case GetVersion: + return SCMDatanodeResponse.newBuilder() + .setCmdType(cmdType) + .setStatus(Status.OK) + .setGetVersionResponse( + impl.getVersion(request.getGetVersionRequest())) + .build(); + case SendHeartbeat: + return SCMDatanodeResponse.newBuilder() + .setCmdType(cmdType) + .setStatus(Status.OK) + .setSendHeartbeatResponse( + impl.sendHeartbeat(request.getSendHeartbeatRequest())) + .build(); + case Register: + return SCMDatanodeResponse.newBuilder() + .setCmdType(cmdType) + .setStatus(Status.OK) + .setRegisterResponse(register(request.getRegisterRequest())) + .build(); + default: + throw new ServiceException("Unknown command type: " + cmdType); + } } catch (IOException e) { throw new ServiceException(e); } } - } \ No newline at end of file diff --git a/hadoop-hdds/container-service/src/main/proto/StorageContainerDatanodeProtocol.proto b/hadoop-hdds/container-service/src/main/proto/StorageContainerDatanodeProtocol.proto index 718a999ee101d..a975cd5605fc7 100644 --- a/hadoop-hdds/container-service/src/main/proto/StorageContainerDatanodeProtocol.proto +++ b/hadoop-hdds/container-service/src/main/proto/StorageContainerDatanodeProtocol.proto @@ -34,6 +34,45 @@ package hadoop.hdds; import "hdds.proto"; + +message SCMDatanodeRequest { + required Type cmdType = 1; // Type of the command + + optional string traceID = 2; + + optional SCMVersionRequestProto getVersionRequest = 3; + optional SCMRegisterRequestProto registerRequest = 4; + optional SCMHeartbeatRequestProto sendHeartbeatRequest = 5; +} + +message SCMDatanodeResponse { + required Type cmdType = 1; // Type of the command + + optional string traceID = 2; + + optional bool success = 3 [default = true]; + + optional string message = 4; + + required Status status = 5; + + optional SCMVersionResponseProto getVersionResponse = 6; + optional SCMRegisteredResponseProto registerResponse = 7; + optional SCMHeartbeatResponseProto sendHeartbeatResponse = 8; + +} + +enum Type { + GetVersion = 1; + Register = 2; + SendHeartbeat = 3; +} + +enum Status { + OK = 1; + ERROR = 2; +} + /** * Request for version info of the software stack on the server. */ @@ -70,6 +109,8 @@ message SCMRegisteredResponseProto { optional SCMNodeAddressList addressList = 4; optional string hostname = 5; optional string ipAddress = 6; + optional string networkName = 7; + optional string networkLocation = 8; } /** @@ -211,6 +252,8 @@ message PipelineActionsProto { message ClosePipelineInfo { enum Reason { PIPELINE_FAILED = 1; + PIPELINE_LOG_FAILED = 2; + STATEMACHINE_TRANSACTION_FAILED = 3; } required PipelineID pipelineID = 1; optional Reason reason = 3; @@ -381,21 +424,6 @@ message ReplicateContainerCommandProto { */ service StorageContainerDatanodeProtocolService { - /** - * Gets the version information from the SCM. - */ - rpc getVersion (SCMVersionRequestProto) returns (SCMVersionResponseProto); - - /** - * Registers a data node with SCM. - */ - rpc register (SCMRegisterRequestProto) returns (SCMRegisteredResponseProto); - - /** - * Send heartbeat from datanode to SCM. HB's under SCM looks more - * like life line protocol than HB's under HDFS. In other words, it is - * extremely light weight and contains no data payload. - */ - rpc sendHeartbeat (SCMHeartbeatRequestProto) returns (SCMHeartbeatResponseProto); - + //Message sent from Datanode to SCM as a heartbeat. + rpc submitRequest (SCMDatanodeRequest) returns (SCMDatanodeResponse); } diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/TestHddsDatanodeService.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/TestHddsDatanodeService.java index f54fa75acf0b9..af56d0643d5b4 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/TestHddsDatanodeService.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/TestHddsDatanodeService.java @@ -65,13 +65,13 @@ public void tearDown() { public void testStartup() throws IOException { service = HddsDatanodeService.createHddsDatanodeService(args); service.start(conf); - service.join(); assertNotNull(service.getDatanodeDetails()); assertNotNull(service.getDatanodeDetails().getHostName()); assertFalse(service.getDatanodeStateMachine().isDaemonStopped()); service.stop(); + service.join(); service.close(); } diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/TestHddsSecureDatanodeInit.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/TestHddsSecureDatanodeInit.java index 20d5eef67fe4f..04fd3a499aa0a 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/TestHddsSecureDatanodeInit.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/TestHddsSecureDatanodeInit.java @@ -66,6 +66,7 @@ public class TestHddsSecureDatanodeInit { private static KeyCodec keyCodec; private static CertificateCodec certCodec; private static X509CertificateHolder certHolder; + private final static String DN_COMPONENT = DNCertificateClient.COMPONENT_NAME; @BeforeClass public static void setUp() throws Exception { @@ -93,8 +94,8 @@ public static void setUp() throws Exception { service.initializeCertificateClient(conf); return null; }); - certCodec = new CertificateCodec(securityConfig); - keyCodec = new KeyCodec(securityConfig); + certCodec = new CertificateCodec(securityConfig, DN_COMPONENT); + keyCodec = new KeyCodec(securityConfig, DN_COMPONENT); dnLogs.clearOutput(); privateKey = service.getCertificateClient().getPrivateKey(); publicKey = service.getCertificateClient().getPublicKey(); @@ -115,12 +116,14 @@ public static void tearDown() { @Before public void setUpDNCertClient(){ - FileUtils.deleteQuietly(Paths.get(securityConfig.getKeyLocation() - .toString(), securityConfig.getPrivateKeyFileName()).toFile()); - FileUtils.deleteQuietly(Paths.get(securityConfig.getKeyLocation() - .toString(), securityConfig.getPublicKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + securityConfig.getKeyLocation(DN_COMPONENT).toString(), + securityConfig.getPrivateKeyFileName()).toFile()); + FileUtils.deleteQuietly(Paths.get( + securityConfig.getKeyLocation(DN_COMPONENT).toString(), + securityConfig.getPublicKeyFileName()).toFile()); FileUtils.deleteQuietly(Paths.get(securityConfig - .getCertificateLocation().toString(), + .getCertificateLocation(DN_COMPONENT).toString(), securityConfig.getCertificateFileName()).toFile()); dnLogs.clearOutput(); client = new DNCertificateClient(securityConfig, diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java index 514c8224bca9d..5a7c30ca68f79 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java @@ -29,12 +29,14 @@ import org.apache.hadoop.ipc.ProtobufRpcEngine; import org.apache.hadoop.ipc.RPC; import org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol; +import org.apache.hadoop.ozone.protocolPB.ProtocolMessageMetrics; import org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolPB; import org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolServerSideTranslatorPB; import org.apache.hadoop.test.GenericTestUtils; import com.google.protobuf.BlockingService; import static org.apache.hadoop.hdds.scm.ScmConfigKeys.HDDS_DATANODE_DIR_KEY; +import org.mockito.Mockito; /** * Test Endpoint class. @@ -91,7 +93,7 @@ public static RPC.Server startScmRpcServer(Configuration configuration, StorageContainerDatanodeProtocolService. newReflectiveBlockingService( new StorageContainerDatanodeProtocolServerSideTranslatorPB( - server)); + server, Mockito.mock(ProtocolMessageMetrics.class))); RPC.Server scmServer = startRpcServer(configuration, rpcServerAddresss, StorageContainerDatanodeProtocolPB.class, scmDatanodeService, diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestContainerCache.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestContainerCache.java new file mode 100644 index 0000000000000..b6584d17017b7 --- /dev/null +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestContainerCache.java @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.container.common; + +import org.apache.hadoop.fs.FileSystemTestHelper; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; +import org.apache.hadoop.ozone.OzoneConfigKeys; +import org.apache.hadoop.ozone.container.common.utils.ContainerCache; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; +import org.apache.hadoop.hdds.utils.MetadataStore; +import org.apache.hadoop.hdds.utils.MetadataStoreBuilder; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; + +import java.io.File; + + +/** + * Test ContainerCache with evictions. + */ +public class TestContainerCache { + private static String testRoot = new FileSystemTestHelper().getTestRootDir(); + + @Rule + public ExpectedException thrown = ExpectedException.none(); + + private void createContainerDB(OzoneConfiguration conf, File dbFile) + throws Exception { + MetadataStore store = MetadataStoreBuilder.newBuilder().setConf(conf) + .setCreateIfMissing(true).setDbFile(dbFile).build(); + + // we close since the SCM pre-creates containers. + // we will open and put Db handle into a cache when keys are being created + // in a container. + + store.close(); + } + + @Test + public void testContainerCacheEviction() throws Exception { + File root = new File(testRoot); + root.mkdirs(); + + OzoneConfiguration conf = new OzoneConfiguration(); + conf.setInt(OzoneConfigKeys.OZONE_CONTAINER_CACHE_SIZE, 2); + + ContainerCache cache = ContainerCache.getInstance(conf); + File containerDir1 = new File(root, "cont1"); + File containerDir2 = new File(root, "cont2"); + File containerDir3 = new File(root, "cont3"); + File containerDir4 = new File(root, "cont4"); + + + createContainerDB(conf, containerDir1); + createContainerDB(conf, containerDir2); + createContainerDB(conf, containerDir3); + createContainerDB(conf, containerDir4); + + // Get 2 references out of the same db and verify the objects are same. + ReferenceCountedDB db1 = cache.getDB(1, "RocksDB", + containerDir1.getPath(), conf); + Assert.assertEquals(1, db1.getReferenceCount()); + ReferenceCountedDB db2 = cache.getDB(1, "RocksDB", + containerDir1.getPath(), conf); + Assert.assertEquals(2, db2.getReferenceCount()); + Assert.assertEquals(2, db1.getReferenceCount()); + Assert.assertEquals(db1, db2); + + // add one more references to ContainerCache. + ReferenceCountedDB db3 = cache.getDB(2, "RocksDB", + containerDir2.getPath(), conf); + Assert.assertEquals(1, db3.getReferenceCount()); + + // and close the reference + db3.close(); + Assert.assertEquals(0, db3.getReferenceCount()); + + Assert.assertTrue(cache.isFull()); + + // add one more reference to ContainerCache and verify that it will not + // evict the least recent entry as it has reference. + ReferenceCountedDB db4 = cache.getDB(3, "RocksDB", + containerDir3.getPath(), conf); + Assert.assertEquals(1, db4.getReferenceCount()); + + Assert.assertEquals(2, cache.size()); + Assert.assertNotNull(cache.get(containerDir1.getPath())); + Assert.assertNull(cache.get(containerDir2.getPath())); + + // Now close both the references for container1 + db1.close(); + db2.close(); + Assert.assertEquals(0, db1.getReferenceCount()); + Assert.assertEquals(0, db2.getReferenceCount()); + + + // The reference count for container1 is 0 but it is not evicted. + ReferenceCountedDB db5 = cache.getDB(1, "RocksDB", + containerDir1.getPath(), conf); + Assert.assertEquals(1, db5.getReferenceCount()); + Assert.assertEquals(db1, db5); + db5.close(); + db4.close(); + + + // Decrementing reference count below zero should fail. + thrown.expect(IllegalArgumentException.class); + db5.close(); + } +} diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerSet.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerSet.java index d65d972e90fe4..e1e7119727b30 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerSet.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerSet.java @@ -25,10 +25,12 @@ import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException; import org.apache.hadoop.ozone.container.common.interfaces.Container; +import org.apache.hadoop.ozone.container.common.volume.HddsVolume; import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer; import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerData; import org.apache.hadoop.test.GenericTestUtils; import org.junit.Test; +import org.mockito.Mockito; import java.io.IOException; import java.util.ArrayList; @@ -94,12 +96,11 @@ public void testIteratorsAndCount() throws StorageContainerException { assertEquals(10, containerSet.containerCount()); - // Using containerIterator. - Iterator containerIterator = containerSet.getContainerIterator(); + Iterator> iterator = containerSet.getContainerIterator(); int count = 0; - while(containerIterator.hasNext()) { - Container kv = containerIterator.next(); + while(iterator.hasNext()) { + Container kv = iterator.next(); ContainerData containerData = kv.getContainerData(); long containerId = containerData.getContainerID(); if (containerId%2 == 0) { @@ -114,7 +115,7 @@ public void testIteratorsAndCount() throws StorageContainerException { assertEquals(10, count); //Using containerMapIterator. - Iterator> containerMapIterator = containerSet + Iterator>> containerMapIterator = containerSet .getContainerMapIterator(); count = 0; @@ -135,6 +136,47 @@ public void testIteratorsAndCount() throws StorageContainerException { } + @Test + public void testIteratorPerVolume() throws StorageContainerException { + HddsVolume vol1 = Mockito.mock(HddsVolume.class); + Mockito.when(vol1.getStorageID()).thenReturn("uuid-1"); + HddsVolume vol2 = Mockito.mock(HddsVolume.class); + Mockito.when(vol2.getStorageID()).thenReturn("uuid-2"); + + ContainerSet containerSet = new ContainerSet(); + for (int i=0; i<10; i++) { + KeyValueContainerData kvData = new KeyValueContainerData(i, + (long) StorageUnit.GB.toBytes(5), UUID.randomUUID().toString(), + UUID.randomUUID().toString()); + if (i%2 == 0) { + kvData.setVolume(vol1); + } else { + kvData.setVolume(vol2); + } + kvData.setState(ContainerProtos.ContainerDataProto.State.CLOSED); + KeyValueContainer kv = new KeyValueContainer(kvData, new + OzoneConfiguration()); + containerSet.addContainer(kv); + } + + Iterator> iter1 = containerSet.getContainerIterator(vol1); + int count1 = 0; + while (iter1.hasNext()) { + Container c = iter1.next(); + assertEquals(0, (c.getContainerData().getContainerID() % 2)); + count1++; + } + assertEquals(5, count1); + + Iterator> iter2 = containerSet.getContainerIterator(vol2); + int count2 = 0; + while (iter2.hasNext()) { + Container c = iter2.next(); + assertEquals(1, (c.getContainerData().getContainerID() % 2)); + count2++; + } + assertEquals(5, count2); + } @Test public void testGetContainerReport() throws IOException { diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestHddsDispatcher.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestHddsDispatcher.java index d425820193698..fe27eeb02d6b1 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestHddsDispatcher.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestHddsDispatcher.java @@ -73,13 +73,15 @@ public class TestHddsDispatcher { public void testContainerCloseActionWhenFull() throws IOException { String testDir = GenericTestUtils.getTempPath( TestHddsDispatcher.class.getSimpleName()); + OzoneConfiguration conf = new OzoneConfiguration(); + conf.set(HDDS_DATANODE_DIR_KEY, testDir); + DatanodeDetails dd = randomDatanodeDetails(); + VolumeSet volumeSet = new VolumeSet(dd.getUuidString(), conf); + try { UUID scmId = UUID.randomUUID(); - OzoneConfiguration conf = new OzoneConfiguration(); - conf.set(HDDS_DATANODE_DIR_KEY, testDir); - DatanodeDetails dd = randomDatanodeDetails(); ContainerSet containerSet = new ContainerSet(); - VolumeSet volumeSet = new VolumeSet(dd.getUuidString(), conf); + DatanodeStateMachine stateMachine = Mockito.mock( DatanodeStateMachine.class); StateContext context = Mockito.mock(StateContext.class); @@ -118,6 +120,7 @@ public void testContainerCloseActionWhenFull() throws IOException { .addContainerActionIfAbsent(Mockito.any(ContainerAction.class)); } finally { + volumeSet.shutdown(); FileUtils.deleteDirectory(new File(testDir)); } diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerCommandHandler.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerCommandHandler.java index 1f6ed861a9313..a92f236138209 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerCommandHandler.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerCommandHandler.java @@ -16,306 +16,191 @@ */ package org.apache.hadoop.ozone.container.common.statemachine.commandhandler; -import org.apache.commons.io.FileUtils; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hdds.HddsConfigKeys; import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdds.protocol.DatanodeDetails; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; -import org.apache.hadoop.hdds.scm.ScmConfigKeys; -import org.apache.hadoop.hdds.scm.container.ContainerID; import org.apache.hadoop.hdds.scm.pipeline.PipelineID; -import org.apache.hadoop.ozone.OzoneConfigKeys; +import org.apache.hadoop.ozone.container.common.impl.ContainerSet; import org.apache.hadoop.ozone.container.common.interfaces.Container; +import org.apache.hadoop.ozone.container.common.interfaces.Handler; import org.apache.hadoop.ozone.container.common.statemachine .DatanodeStateMachine; import org.apache.hadoop.ozone.container.common.statemachine.StateContext; +import org.apache.hadoop.ozone.container.common.transport.server.XceiverServerSpi; +import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer; +import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerData; +import org.apache.hadoop.ozone.container.ozoneimpl.ContainerController; import org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer; import org.apache.hadoop.ozone.protocol.commands.CloseContainerCommand; -import org.apache.hadoop.test.GenericTestUtils; -import org.apache.ratis.RatisHelper; -import org.apache.ratis.client.RaftClient; -import org.apache.ratis.protocol.RaftGroup; -import org.apache.ratis.protocol.RaftGroupId; -import org.apache.ratis.protocol.RaftPeer; -import org.apache.ratis.retry.RetryPolicy; -import org.apache.ratis.rpc.SupportedRpcType; -import org.apache.ratis.util.TimeDuration; -import org.junit.AfterClass; -import org.junit.Assert; +import org.junit.Before; import org.junit.Test; -import org.mockito.Mockito; -import java.io.File; import java.io.IOException; -import java.util.Collections; -import java.util.Random; import java.util.UUID; -import java.util.concurrent.TimeUnit; + +import static java.util.Collections.singletonMap; +import static org.apache.hadoop.ozone.OzoneConsts.GB; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; /** * Test cases to verify CloseContainerCommandHandler in datanode. */ public class TestCloseContainerCommandHandler { - private final StateContext context = Mockito.mock(StateContext.class); - private final Random random = new Random(); - private static File testDir; + private static final long CONTAINER_ID = 123L; + + private OzoneContainer ozoneContainer; + private StateContext context; + private XceiverServerSpi writeChannel; + private Container container; + private Handler containerHandler; + private PipelineID pipelineID; + private PipelineID nonExistentPipelineID = PipelineID.randomId(); + + private CloseContainerCommandHandler subject = + new CloseContainerCommandHandler(); + + @Before + public void before() throws Exception { + context = mock(StateContext.class); + DatanodeStateMachine dnStateMachine = mock(DatanodeStateMachine.class); + when(dnStateMachine.getDatanodeDetails()) + .thenReturn(randomDatanodeDetails()); + when(context.getParent()).thenReturn(dnStateMachine); + + pipelineID = PipelineID.randomId(); + + KeyValueContainerData data = new KeyValueContainerData(CONTAINER_ID, GB, + pipelineID.getId().toString(), null); + + container = new KeyValueContainer(data, new OzoneConfiguration()); + ContainerSet containerSet = new ContainerSet(); + containerSet.addContainer(container); + + containerHandler = mock(Handler.class); + ContainerController controller = new ContainerController(containerSet, + singletonMap(ContainerProtos.ContainerType.KeyValueContainer, + containerHandler)); + + writeChannel = mock(XceiverServerSpi.class); + ozoneContainer = mock(OzoneContainer.class); + when(ozoneContainer.getController()).thenReturn(controller); + when(ozoneContainer.getContainerSet()).thenReturn(containerSet); + when(ozoneContainer.getWriteChannel()).thenReturn(writeChannel); + when(writeChannel.isExist(pipelineID.getProtobuf())).thenReturn(true); + when(writeChannel.isExist(nonExistentPipelineID.getProtobuf())) + .thenReturn(false); + } @Test - public void testCloseContainerViaRatis() - throws Exception { - final OzoneConfiguration conf = new OzoneConfiguration(); - final DatanodeDetails datanodeDetails = randomDatanodeDetails(); - final OzoneContainer ozoneContainer = - getOzoneContainer(conf, datanodeDetails); - ozoneContainer.start(UUID.randomUUID().toString()); - try { - final Container container = - createContainer(conf, datanodeDetails, ozoneContainer); - Mockito.verify(context.getParent(), - Mockito.times(1)).triggerHeartbeat(); - final long containerId = container.getContainerData().getContainerID(); - final PipelineID pipelineId = PipelineID.valueOf(UUID.fromString( - container.getContainerData().getOriginPipelineId())); - - // We have created a container via ratis. - // Now close the container on ratis. - final CloseContainerCommandHandler closeHandler = - new CloseContainerCommandHandler(); - final CloseContainerCommand command = new CloseContainerCommand( - containerId, pipelineId); - - closeHandler.handle(command, ozoneContainer, context, null); - - Assert.assertEquals(ContainerProtos.ContainerDataProto.State.CLOSED, - ozoneContainer.getContainerSet().getContainer(containerId) - .getContainerState()); - - Mockito.verify(context.getParent(), - Mockito.times(3)).triggerHeartbeat(); - } finally { - ozoneContainer.stop(); - } + public void closeContainerWithPipeline() throws Exception { + // close a container that's associated with an existing pipeline + subject.handle(closeWithKnownPipeline(), ozoneContainer, context, null); + + verify(containerHandler) + .markContainerForClose(container); + verify(writeChannel) + .submitRequest(any(), eq(pipelineID.getProtobuf())); + verify(containerHandler, never()) + .quasiCloseContainer(container); } @Test - public void testCloseContainerViaStandalone() - throws Exception { - final OzoneConfiguration conf = new OzoneConfiguration(); - final DatanodeDetails datanodeDetails = randomDatanodeDetails(); - final OzoneContainer ozoneContainer = - getOzoneContainer(conf, datanodeDetails); - ozoneContainer.start(UUID.randomUUID().toString()); - try { - final Container container = - createContainer(conf, datanodeDetails, ozoneContainer); - Mockito.verify(context.getParent(), - Mockito.times(1)).triggerHeartbeat(); - final long containerId = container.getContainerData().getContainerID(); - // To quasi close specify a pipeline which doesn't exist in the datanode. - final PipelineID pipelineId = PipelineID.randomId(); - - // We have created a container via ratis. Now quasi close it. - final CloseContainerCommandHandler closeHandler = - new CloseContainerCommandHandler(); - final CloseContainerCommand command = new CloseContainerCommand( - containerId, pipelineId); - - closeHandler.handle(command, ozoneContainer, context, null); - - Assert.assertEquals(ContainerProtos.ContainerDataProto.State.QUASI_CLOSED, - ozoneContainer.getContainerSet().getContainer(containerId) - .getContainerState()); - - Mockito.verify(context.getParent(), - Mockito.times(3)).triggerHeartbeat(); - } finally { - ozoneContainer.stop(); - } + public void closeContainerWithoutPipeline() throws IOException { + // close a container that's NOT associated with an open pipeline + subject.handle(closeWithUnknownPipeline(), ozoneContainer, context, null); + + verify(containerHandler) + .markContainerForClose(container); + verify(writeChannel, never()) + .submitRequest(any(), any()); + // Container in CLOSING state is moved to UNHEALTHY if pipeline does not + // exist. Container should not exist in CLOSING state without a pipeline. + verify(containerHandler) + .markContainerUnhealthy(container); + } + + @Test + public void forceCloseQuasiClosedContainer() throws Exception { + // force-close a container that's already quasi closed + container.getContainerData() + .setState(ContainerProtos.ContainerDataProto.State.QUASI_CLOSED); + + subject.handle(forceCloseWithoutPipeline(), ozoneContainer, context, null); + + verify(writeChannel, never()) + .submitRequest(any(), any()); + verify(containerHandler) + .closeContainer(container); } @Test - public void testQuasiCloseToClose() throws Exception { - final OzoneConfiguration conf = new OzoneConfiguration(); - final DatanodeDetails datanodeDetails = randomDatanodeDetails(); - final OzoneContainer ozoneContainer = - getOzoneContainer(conf, datanodeDetails); - ozoneContainer.start(UUID.randomUUID().toString()); - try { - final Container container = - createContainer(conf, datanodeDetails, ozoneContainer); - Mockito.verify(context.getParent(), - Mockito.times(1)).triggerHeartbeat(); - final long containerId = container.getContainerData().getContainerID(); - // A pipeline which doesn't exist in the datanode. - final PipelineID pipelineId = PipelineID.randomId(); - - // We have created a container via ratis. Now quasi close it. - final CloseContainerCommandHandler closeHandler = - new CloseContainerCommandHandler(); - final CloseContainerCommand command = new CloseContainerCommand( - containerId, pipelineId); - - closeHandler.handle(command, ozoneContainer, context, null); - - Assert.assertEquals(ContainerProtos.ContainerDataProto.State.QUASI_CLOSED, - ozoneContainer.getContainerSet().getContainer(containerId) - .getContainerState()); - - Mockito.verify(context.getParent(), - Mockito.times(3)).triggerHeartbeat(); - - // The container is quasi closed. Force close the container now. - final CloseContainerCommand closeCommand = new CloseContainerCommand( - containerId, pipelineId, true); - - closeHandler.handle(closeCommand, ozoneContainer, context, null); - - Assert.assertEquals(ContainerProtos.ContainerDataProto.State.CLOSED, - ozoneContainer.getContainerSet().getContainer(containerId) - .getContainerState()); - - Mockito.verify(context.getParent(), - Mockito.times(4)).triggerHeartbeat(); - } finally { - ozoneContainer.stop(); - } + public void forceCloseOpenContainer() throws Exception { + // force-close a container that's NOT associated with an open pipeline + subject.handle(forceCloseWithoutPipeline(), ozoneContainer, context, null); + + verify(writeChannel, never()) + .submitRequest(any(), any()); + // Container in CLOSING state is moved to UNHEALTHY if pipeline does not + // exist. Container should not exist in CLOSING state without a pipeline. + verify(containerHandler) + .markContainerUnhealthy(container); } @Test - public void testForceCloseOpenContainer() throws Exception { - final OzoneConfiguration conf = new OzoneConfiguration(); - final DatanodeDetails datanodeDetails = randomDatanodeDetails(); - final OzoneContainer ozoneContainer = - getOzoneContainer(conf, datanodeDetails); - ozoneContainer.start(UUID.randomUUID().toString()); - try { - final Container container = - createContainer(conf, datanodeDetails, ozoneContainer); - Mockito.verify(context.getParent(), - Mockito.times(1)).triggerHeartbeat(); - final long containerId = container.getContainerData().getContainerID(); - // A pipeline which doesn't exist in the datanode. - final PipelineID pipelineId = PipelineID.randomId(); - - final CloseContainerCommandHandler closeHandler = - new CloseContainerCommandHandler(); - - final CloseContainerCommand closeCommand = new CloseContainerCommand( - containerId, pipelineId, true); - - closeHandler.handle(closeCommand, ozoneContainer, context, null); - - Assert.assertEquals(ContainerProtos.ContainerDataProto.State.CLOSED, - ozoneContainer.getContainerSet().getContainer(containerId) - .getContainerState()); - - Mockito.verify(context.getParent(), - Mockito.times(3)).triggerHeartbeat(); - } finally { - ozoneContainer.stop(); - } + public void forceCloseOpenContainerWithPipeline() throws Exception { + // force-close a container that's associated with an existing pipeline + subject.handle(forceCloseWithPipeline(), ozoneContainer, context, null); + + verify(containerHandler) + .markContainerForClose(container); + verify(writeChannel) + .submitRequest(any(), any()); + verify(containerHandler, never()) + .quasiCloseContainer(container); + verify(containerHandler, never()) + .closeContainer(container); } @Test - public void testQuasiCloseClosedContainer() - throws Exception { - final OzoneConfiguration conf = new OzoneConfiguration(); - final DatanodeDetails datanodeDetails = randomDatanodeDetails(); - final OzoneContainer ozoneContainer = getOzoneContainer( - conf, datanodeDetails); - ozoneContainer.start(UUID.randomUUID().toString()); - try { - final Container container = createContainer( - conf, datanodeDetails, ozoneContainer); - Mockito.verify(context.getParent(), - Mockito.times(1)).triggerHeartbeat(); - final long containerId = container.getContainerData().getContainerID(); - final PipelineID pipelineId = PipelineID.valueOf(UUID.fromString( - container.getContainerData().getOriginPipelineId())); - - final CloseContainerCommandHandler closeHandler = - new CloseContainerCommandHandler(); - final CloseContainerCommand closeCommand = new CloseContainerCommand( - containerId, pipelineId); - - closeHandler.handle(closeCommand, ozoneContainer, context, null); - - Assert.assertEquals(ContainerProtos.ContainerDataProto.State.CLOSED, - ozoneContainer.getContainerSet().getContainer(containerId) - .getContainerState()); - - // The container is closed, now we send close command with - // pipeline id which doesn't exist. - // This should cause the datanode to trigger quasi close, since the - // container is already closed, this should do nothing. - // The command should not fail either. - final PipelineID randomPipeline = PipelineID.randomId(); - final CloseContainerCommand quasiCloseCommand = - new CloseContainerCommand(containerId, randomPipeline); - closeHandler.handle(quasiCloseCommand, ozoneContainer, context, null); - - Assert.assertEquals(ContainerProtos.ContainerDataProto.State.CLOSED, - ozoneContainer.getContainerSet().getContainer(containerId) - .getContainerState()); - } finally { - ozoneContainer.stop(); - } + public void closeAlreadyClosedContainer() throws Exception { + container.getContainerData() + .setState(ContainerProtos.ContainerDataProto.State.CLOSED); + + // Since the container is already closed, these commands should do nothing, + // neither should they fail + subject.handle(closeWithUnknownPipeline(), ozoneContainer, context, null); + subject.handle(closeWithKnownPipeline(), ozoneContainer, context, null); + + verify(containerHandler, never()) + .markContainerForClose(container); + verify(containerHandler, never()) + .quasiCloseContainer(container); + verify(containerHandler, never()) + .closeContainer(container); + verify(writeChannel, never()) + .submitRequest(any(), any()); } - private OzoneContainer getOzoneContainer(final OzoneConfiguration conf, - final DatanodeDetails datanodeDetails) throws IOException { - testDir = GenericTestUtils.getTestDir( - TestCloseContainerCommandHandler.class.getName() + UUID.randomUUID()); - conf.set(HddsConfigKeys.OZONE_METADATA_DIRS, testDir.getPath()); - conf.set(ScmConfigKeys.HDDS_DATANODE_DIR_KEY, testDir.getPath()); - conf.setBoolean(OzoneConfigKeys.DFS_CONTAINER_RATIS_IPC_RANDOM_PORT, true); - conf.setBoolean(OzoneConfigKeys.DFS_CONTAINER_IPC_RANDOM_PORT, true); - - final DatanodeStateMachine datanodeStateMachine = Mockito.mock( - DatanodeStateMachine.class); - Mockito.when(datanodeStateMachine.getDatanodeDetails()) - .thenReturn(datanodeDetails); - Mockito.when(context.getParent()).thenReturn(datanodeStateMachine); - final OzoneContainer ozoneContainer = new OzoneContainer( - datanodeDetails, conf, context, null); - return ozoneContainer; + private CloseContainerCommand closeWithKnownPipeline() { + return new CloseContainerCommand(CONTAINER_ID, pipelineID); } - private Container createContainer(final Configuration conf, - final DatanodeDetails datanodeDetails, - final OzoneContainer ozoneContainer) throws Exception { - final PipelineID pipelineID = PipelineID.randomId(); - final RaftGroupId raftGroupId = RaftGroupId.valueOf(pipelineID.getId()); - final RetryPolicy retryPolicy = RatisHelper.createRetryPolicy(conf); - final RaftPeer peer = RatisHelper.toRaftPeer(datanodeDetails); - final RaftGroup group = RatisHelper.newRaftGroup(raftGroupId, - Collections.singleton(datanodeDetails)); - final int maxOutstandingRequests = 100; - final RaftClient client = RatisHelper - .newRaftClient(SupportedRpcType.GRPC, peer, retryPolicy, - maxOutstandingRequests, - TimeDuration.valueOf(3, TimeUnit.SECONDS)); - Assert.assertTrue(client.groupAdd(group, peer.getId()).isSuccess()); - Thread.sleep(2000); - final ContainerID containerId = ContainerID.valueof( - random.nextLong() & Long.MAX_VALUE); - ContainerProtos.ContainerCommandRequestProto.Builder request = - ContainerProtos.ContainerCommandRequestProto.newBuilder(); - request.setCmdType(ContainerProtos.Type.CreateContainer); - request.setContainerID(containerId.getId()); - request.setCreateContainer( - ContainerProtos.CreateContainerRequestProto.getDefaultInstance()); - request.setDatanodeUuid(datanodeDetails.getUuidString()); - ozoneContainer.getWriteChannel().submitRequest( - request.build(), pipelineID.getProtobuf()); - - final Container container = ozoneContainer.getContainerSet().getContainer( - containerId.getId()); - Assert.assertEquals(ContainerProtos.ContainerDataProto.State.OPEN, - container.getContainerState()); - return container; + private CloseContainerCommand closeWithUnknownPipeline() { + return new CloseContainerCommand(CONTAINER_ID, nonExistentPipelineID); + } + + private CloseContainerCommand forceCloseWithPipeline() { + return new CloseContainerCommand(CONTAINER_ID, pipelineID, true); + } + + private CloseContainerCommand forceCloseWithoutPipeline() { + return new CloseContainerCommand(CONTAINER_ID, nonExistentPipelineID, true); } /** @@ -339,9 +224,4 @@ private static DatanodeDetails randomDatanodeDetails() { .addPort(restPort); return builder.build(); } - - @AfterClass - public static void teardown() throws IOException { - FileUtils.deleteDirectory(testDir); - } } \ No newline at end of file diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestHddsVolumeChecker.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestHddsVolumeChecker.java index f2a0c253fd381..2e267be01e8d0 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestHddsVolumeChecker.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestHddsVolumeChecker.java @@ -18,11 +18,9 @@ package org.apache.hadoop.ozone.container.common.volume; -import com.google.common.base.Optional; import com.google.common.util.concurrent.Futures; import com.google.common.util.concurrent.ListenableFuture; import org.apache.hadoop.hdfs.HdfsConfiguration; -import org.apache.hadoop.hdfs.server.datanode.checker.AsyncChecker; import org.apache.hadoop.hdfs.server.datanode.checker.Checkable; import org.apache.hadoop.hdfs.server.datanode.checker.VolumeCheckResult; import org.apache.hadoop.test.GenericTestUtils; @@ -42,6 +40,7 @@ import java.util.Collection; import java.util.List; import java.util.Set; +import java.util.Optional; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicLong; diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestRoundRobinVolumeChoosingPolicy.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestRoundRobinVolumeChoosingPolicy.java index 80594d3524536..d0fbf10269c4f 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestRoundRobinVolumeChoosingPolicy.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestRoundRobinVolumeChoosingPolicy.java @@ -24,6 +24,7 @@ import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.util.DiskChecker.DiskOutOfSpaceException; import org.apache.hadoop.util.ReflectionUtils; +import org.junit.After; import org.junit.Assert; import org.junit.Before; import org.junit.Test; @@ -40,10 +41,12 @@ public class TestRoundRobinVolumeChoosingPolicy { private RoundRobinVolumeChoosingPolicy policy; private List volumes; + private VolumeSet volumeSet; private final String baseDir = MiniDFSCluster.getBaseDirectory(); private final String volume1 = baseDir + "disk1"; private final String volume2 = baseDir + "disk2"; + private static final String DUMMY_IP_ADDR = "0.0.0.0"; @Before @@ -53,10 +56,18 @@ public void setup() throws Exception { conf.set(DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY, dataDirKey); policy = ReflectionUtils.newInstance( RoundRobinVolumeChoosingPolicy.class, null); - VolumeSet volumeSet = new VolumeSet(UUID.randomUUID().toString(), conf); + volumeSet = new VolumeSet(UUID.randomUUID().toString(), conf); volumes = volumeSet.getVolumesList(); } + @After + public void cleanUp() { + if (volumeSet != null) { + volumeSet.shutdown(); + volumeSet = null; + } + } + @Test public void testRRVolumeChoosingPolicy() throws Exception { HddsVolume hddsVolume1 = volumes.get(0); diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestVolumeSet.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestVolumeSet.java index f97ad4e873f7b..fa280ddb73084 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestVolumeSet.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestVolumeSet.java @@ -230,13 +230,14 @@ public void testFailVolumes() throws Exception{ ozoneConfig.set(HDDS_DATANODE_DIR_KEY, readOnlyVolumePath.getAbsolutePath() + "," + volumePath.getAbsolutePath()); volSet = new VolumeSet(UUID.randomUUID().toString(), ozoneConfig); - assertTrue(volSet.getFailedVolumesList().size() == 1); + assertEquals(1, volSet.getFailedVolumesList().size()); assertEquals(readOnlyVolumePath, volSet.getFailedVolumesList().get(0) .getHddsRootDir()); //Set back to writable try { readOnlyVolumePath.setWritable(true); + volSet.shutdown(); } finally { FileUtil.fullyDelete(volumePath); } diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestVolumeSetDiskChecks.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestVolumeSetDiskChecks.java index 472bb9891cb54..c5deff0fc7802 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestVolumeSetDiskChecks.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestVolumeSetDiskChecks.java @@ -100,6 +100,7 @@ public void testOzoneDirsAreCreated() throws IOException { for (String d : dirs) { assertTrue(new File(d).isDirectory()); } + volumeSet.shutdown(); } /** @@ -124,6 +125,7 @@ HddsVolumeChecker getVolumeChecker(Configuration configuration) assertThat(volumeSet.getFailedVolumesList().size(), is(numBadVolumes)); assertThat(volumeSet.getVolumesList().size(), is(numVolumes - numBadVolumes)); + volumeSet.shutdown(); } /** @@ -146,6 +148,7 @@ HddsVolumeChecker getVolumeChecker(Configuration configuration) assertEquals(volumeSet.getFailedVolumesList().size(), numVolumes); assertEquals(volumeSet.getVolumesList().size(), 0); + volumeSet.shutdown(); } /** diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestChunkManagerImpl.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestChunkManagerImpl.java index cf9ea891ebef4..84ab56da86459 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestChunkManagerImpl.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestChunkManagerImpl.java @@ -41,7 +41,6 @@ import java.io.File; import java.nio.ByteBuffer; -import java.util.Arrays; import java.util.UUID; import static java.nio.charset.StandardCharsets.UTF_8; @@ -65,7 +64,7 @@ public class TestChunkManagerImpl { private BlockID blockID; private ChunkManagerImpl chunkManager; private ChunkInfo chunkInfo; - private byte[] data; + private ByteBuffer data; @Rule public TemporaryFolder folder = new TemporaryFolder(); @@ -92,11 +91,11 @@ public void setUp() throws Exception { keyValueContainer.create(volumeSet, volumeChoosingPolicy, scmId); - data = "testing write chunks".getBytes(UTF_8); + data = ByteBuffer.wrap("testing write chunks".getBytes(UTF_8)); // Creating BlockData blockID = new BlockID(1L, 1L); chunkInfo = new ChunkInfo(String.format("%d.data.%d", blockID - .getLocalID(), 0), 0, data.length); + .getLocalID(), 0), 0, data.capacity()); // Create a ChunkManager object. chunkManager = new ChunkManagerImpl(true); @@ -118,8 +117,8 @@ public void testWriteChunkStageWriteAndCommit() throws Exception { // As no chunks are written to the volume writeBytes should be 0 checkWriteIOStats(0, 0); - chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, - ByteBuffer.wrap(data), new DispatcherContext.Builder() + chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, data, + new DispatcherContext.Builder() .setStage(DispatcherContext.WriteChunkStage.WRITE_DATA).build()); // Now a chunk file is being written with Stage WRITE_DATA, so it should // create a temporary chunk file. @@ -137,13 +136,13 @@ public void testWriteChunkStageWriteAndCommit() throws Exception { // As chunk write stage is WRITE_DATA, temp chunk file will be created. assertTrue(tempChunkFile.exists()); - checkWriteIOStats(data.length, 1); + checkWriteIOStats(data.capacity(), 1); - chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, - ByteBuffer.wrap(data), new DispatcherContext.Builder() + chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, data, + new DispatcherContext.Builder() .setStage(DispatcherContext.WriteChunkStage.COMMIT_DATA).build()); - checkWriteIOStats(data.length, 1); + checkWriteIOStats(data.capacity(), 1); // Old temp file should have been renamed to chunk file. assertTrue(chunksPath.listFiles().length == 1); @@ -160,8 +159,8 @@ public void testWriteChunkIncorrectLength() throws Exception { long randomLength = 200L; chunkInfo = new ChunkInfo(String.format("%d.data.%d", blockID .getLocalID(), 0), 0, randomLength); - chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, - ByteBuffer.wrap(data), getDispatcherContext()); + chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, data, + getDispatcherContext()); fail("testWriteChunkIncorrectLength failed"); } catch (StorageContainerException ex) { // As we got an exception, writeBytes should be 0. @@ -181,35 +180,36 @@ public void testWriteChunkStageCombinedData() throws Exception { // Initially chunks folder should be empty. assertTrue(chunksPath.listFiles().length == 0); checkWriteIOStats(0, 0); - chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, - ByteBuffer.wrap(data), getDispatcherContext()); + chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, data, + getDispatcherContext()); // Now a chunk file is being written with Stage COMBINED_DATA, so it should // create a chunk file. assertTrue(chunksPath.listFiles().length == 1); File chunkFile = ChunkUtils.getChunkFile(keyValueContainerData, chunkInfo); assertTrue(chunkFile.exists()); - checkWriteIOStats(data.length, 1); + checkWriteIOStats(data.capacity(), 1); } @Test public void testReadChunk() throws Exception { checkWriteIOStats(0, 0); - chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, - ByteBuffer.wrap(data), getDispatcherContext()); - checkWriteIOStats(data.length, 1); + chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, data, + getDispatcherContext()); + checkWriteIOStats(data.capacity(), 1); checkReadIOStats(0, 0); - byte[] expectedData = chunkManager.readChunk(keyValueContainer, blockID, + ByteBuffer expectedData = chunkManager.readChunk(keyValueContainer, blockID, chunkInfo, getDispatcherContext()); - assertEquals(expectedData.length, data.length); - assertTrue(Arrays.equals(expectedData, data)); - checkReadIOStats(data.length, 1); + assertEquals(expectedData.limit()-expectedData.position(), + chunkInfo.getLen()); + assertTrue(expectedData.rewind().equals(data.rewind())); + checkReadIOStats(expectedData.capacity(), 1); } @Test public void testDeleteChunk() throws Exception { File chunksPath = new File(keyValueContainerData.getChunksPath()); - chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, - ByteBuffer.wrap(data), getDispatcherContext()); + chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, data, + getDispatcherContext()); assertTrue(chunksPath.listFiles().length == 1); chunkManager.deleteChunk(keyValueContainer, blockID, chunkInfo); assertTrue(chunksPath.listFiles().length == 0); @@ -218,8 +218,8 @@ public void testDeleteChunk() throws Exception { @Test public void testDeleteChunkUnsupportedRequest() throws Exception { try { - chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, - ByteBuffer.wrap(data), getDispatcherContext()); + chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, data, + getDispatcherContext()); long randomLength = 200L; chunkInfo = new ChunkInfo(String.format("%d.data.%d", blockID .getLocalID(), 0), 0, randomLength); @@ -235,8 +235,8 @@ public void testDeleteChunkUnsupportedRequest() throws Exception { public void testReadChunkFileNotExists() throws Exception { try { // trying to read a chunk, where chunk file does not exist - byte[] expectedData = chunkManager.readChunk(keyValueContainer, blockID, - chunkInfo, getDispatcherContext()); + ByteBuffer expectedData = chunkManager.readChunk(keyValueContainer, + blockID, chunkInfo, getDispatcherContext()); fail("testReadChunkFileNotExists failed"); } catch (StorageContainerException ex) { GenericTestUtils.assertExceptionContains("Unable to find the chunk " + @@ -249,20 +249,21 @@ public void testReadChunkFileNotExists() throws Exception { public void testWriteAndReadChunkMultipleTimes() throws Exception { for (int i=0; i<100; i++) { chunkInfo = new ChunkInfo(String.format("%d.data.%d", blockID - .getLocalID(), i), 0, data.length); - chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, - ByteBuffer.wrap(data), getDispatcherContext()); + .getLocalID(), i), 0, data.capacity()); + chunkManager.writeChunk(keyValueContainer, blockID, chunkInfo, data, + getDispatcherContext()); + data.rewind(); } - checkWriteIOStats(data.length*100, 100); + checkWriteIOStats(data.capacity()*100, 100); assertTrue(hddsVolume.getVolumeIOStats().getWriteTime() > 0); for (int i=0; i<100; i++) { chunkInfo = new ChunkInfo(String.format("%d.data.%d", blockID - .getLocalID(), i), 0, data.length); + .getLocalID(), i), 0, data.capacity()); chunkManager.readChunk(keyValueContainer, blockID, chunkInfo, getDispatcherContext()); } - checkReadIOStats(data.length*100, 100); + checkReadIOStats(data.capacity()*100, 100); assertTrue(hddsVolume.getVolumeIOStats().getReadTime() > 0); } diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueBlockIterator.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueBlockIterator.java index 15d7b342d4a90..4fdd994fb1193 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueBlockIterator.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueBlockIterator.java @@ -33,8 +33,8 @@ import org.apache.hadoop.ozone.container.common.volume.VolumeSet; import org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils; import org.apache.hadoop.test.GenericTestUtils; -import org.apache.hadoop.utils.MetadataKeyFilters; -import org.apache.hadoop.utils.MetadataStore; +import org.apache.hadoop.hdds.utils.MetadataKeyFilters; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import org.junit.After; import org.junit.Before; import org.junit.Test; @@ -109,30 +109,31 @@ public void testKeyValueBlockIteratorWithMixedBlocks() throws Exception { createContainerWithBlocks(containerID, normalBlocks, deletedBlocks); String containerPath = new File(containerData.getMetadataPath()) .getParent(); - KeyValueBlockIterator keyValueBlockIterator = new KeyValueBlockIterator( - containerID, new File(containerPath)); - - int counter = 0; - while(keyValueBlockIterator.hasNext()) { - BlockData blockData = keyValueBlockIterator.nextBlock(); - assertEquals(blockData.getLocalID(), counter++); - } - - assertFalse(keyValueBlockIterator.hasNext()); - - keyValueBlockIterator.seekToFirst(); - counter = 0; - while(keyValueBlockIterator.hasNext()) { - BlockData blockData = keyValueBlockIterator.nextBlock(); - assertEquals(blockData.getLocalID(), counter++); - } - assertFalse(keyValueBlockIterator.hasNext()); - - try { - keyValueBlockIterator.nextBlock(); - } catch (NoSuchElementException ex) { - GenericTestUtils.assertExceptionContains("Block Iterator reached end " + - "for ContainerID " + containerID, ex); + try(KeyValueBlockIterator keyValueBlockIterator = new KeyValueBlockIterator( + containerID, new File(containerPath))) { + + int counter = 0; + while (keyValueBlockIterator.hasNext()) { + BlockData blockData = keyValueBlockIterator.nextBlock(); + assertEquals(blockData.getLocalID(), counter++); + } + + assertFalse(keyValueBlockIterator.hasNext()); + + keyValueBlockIterator.seekToFirst(); + counter = 0; + while (keyValueBlockIterator.hasNext()) { + BlockData blockData = keyValueBlockIterator.nextBlock(); + assertEquals(blockData.getLocalID(), counter++); + } + assertFalse(keyValueBlockIterator.hasNext()); + + try { + keyValueBlockIterator.nextBlock(); + } catch (NoSuchElementException ex) { + GenericTestUtils.assertExceptionContains("Block Iterator reached end " + + "for ContainerID " + containerID, ex); + } } } @@ -142,17 +143,18 @@ public void testKeyValueBlockIteratorWithNextBlock() throws Exception { createContainerWithBlocks(containerID, 2, 0); String containerPath = new File(containerData.getMetadataPath()) .getParent(); - KeyValueBlockIterator keyValueBlockIterator = new KeyValueBlockIterator( - containerID, new File(containerPath)); - long blockID = 0L; - assertEquals(blockID++, keyValueBlockIterator.nextBlock().getLocalID()); - assertEquals(blockID, keyValueBlockIterator.nextBlock().getLocalID()); - - try { - keyValueBlockIterator.nextBlock(); - } catch (NoSuchElementException ex) { - GenericTestUtils.assertExceptionContains("Block Iterator reached end " + - "for ContainerID " + containerID, ex); + try(KeyValueBlockIterator keyValueBlockIterator = new KeyValueBlockIterator( + containerID, new File(containerPath))) { + long blockID = 0L; + assertEquals(blockID++, keyValueBlockIterator.nextBlock().getLocalID()); + assertEquals(blockID, keyValueBlockIterator.nextBlock().getLocalID()); + + try { + keyValueBlockIterator.nextBlock(); + } catch (NoSuchElementException ex) { + GenericTestUtils.assertExceptionContains("Block Iterator reached end " + + "for ContainerID " + containerID, ex); + } } } @@ -162,42 +164,41 @@ public void testKeyValueBlockIteratorWithHasNext() throws Exception { createContainerWithBlocks(containerID, 2, 0); String containerPath = new File(containerData.getMetadataPath()) .getParent(); - KeyValueBlockIterator keyValueBlockIterator = new KeyValueBlockIterator( - containerID, new File(containerPath)); - long blockID = 0L; - - // Even calling multiple times hasNext() should not move entry forward. - assertTrue(keyValueBlockIterator.hasNext()); - assertTrue(keyValueBlockIterator.hasNext()); - assertTrue(keyValueBlockIterator.hasNext()); - assertTrue(keyValueBlockIterator.hasNext()); - assertTrue(keyValueBlockIterator.hasNext()); - assertEquals(blockID++, keyValueBlockIterator.nextBlock().getLocalID()); - - assertTrue(keyValueBlockIterator.hasNext()); - assertTrue(keyValueBlockIterator.hasNext()); - assertTrue(keyValueBlockIterator.hasNext()); - assertTrue(keyValueBlockIterator.hasNext()); - assertTrue(keyValueBlockIterator.hasNext()); - assertEquals(blockID, keyValueBlockIterator.nextBlock().getLocalID()); - - keyValueBlockIterator.seekToLast(); - assertTrue(keyValueBlockIterator.hasNext()); - assertEquals(blockID, keyValueBlockIterator.nextBlock().getLocalID()); - - keyValueBlockIterator.seekToFirst(); - blockID = 0L; - assertEquals(blockID++, keyValueBlockIterator.nextBlock().getLocalID()); - assertEquals(blockID, keyValueBlockIterator.nextBlock().getLocalID()); - - try { - keyValueBlockIterator.nextBlock(); - } catch (NoSuchElementException ex) { - GenericTestUtils.assertExceptionContains("Block Iterator reached end " + - "for ContainerID " + containerID, ex); + try(KeyValueBlockIterator keyValueBlockIterator = new KeyValueBlockIterator( + containerID, new File(containerPath))) { + long blockID = 0L; + + // Even calling multiple times hasNext() should not move entry forward. + assertTrue(keyValueBlockIterator.hasNext()); + assertTrue(keyValueBlockIterator.hasNext()); + assertTrue(keyValueBlockIterator.hasNext()); + assertTrue(keyValueBlockIterator.hasNext()); + assertTrue(keyValueBlockIterator.hasNext()); + assertEquals(blockID++, keyValueBlockIterator.nextBlock().getLocalID()); + + assertTrue(keyValueBlockIterator.hasNext()); + assertTrue(keyValueBlockIterator.hasNext()); + assertTrue(keyValueBlockIterator.hasNext()); + assertTrue(keyValueBlockIterator.hasNext()); + assertTrue(keyValueBlockIterator.hasNext()); + assertEquals(blockID, keyValueBlockIterator.nextBlock().getLocalID()); + + keyValueBlockIterator.seekToLast(); + assertTrue(keyValueBlockIterator.hasNext()); + assertEquals(blockID, keyValueBlockIterator.nextBlock().getLocalID()); + + keyValueBlockIterator.seekToFirst(); + blockID = 0L; + assertEquals(blockID++, keyValueBlockIterator.nextBlock().getLocalID()); + assertEquals(blockID, keyValueBlockIterator.nextBlock().getLocalID()); + + try { + keyValueBlockIterator.nextBlock(); + } catch (NoSuchElementException ex) { + GenericTestUtils.assertExceptionContains("Block Iterator reached end " + + "for ContainerID " + containerID, ex); + } } - - } @Test @@ -208,14 +209,15 @@ public void testKeyValueBlockIteratorWithFilter() throws Exception { createContainerWithBlocks(containerId, normalBlocks, deletedBlocks); String containerPath = new File(containerData.getMetadataPath()) .getParent(); - KeyValueBlockIterator keyValueBlockIterator = new KeyValueBlockIterator( + try(KeyValueBlockIterator keyValueBlockIterator = new KeyValueBlockIterator( containerId, new File(containerPath), MetadataKeyFilters - .getDeletingKeyFilter()); + .getDeletingKeyFilter())) { - int counter = 5; - while(keyValueBlockIterator.hasNext()) { - BlockData blockData = keyValueBlockIterator.nextBlock(); - assertEquals(blockData.getLocalID(), counter++); + int counter = 5; + while (keyValueBlockIterator.hasNext()) { + BlockData blockData = keyValueBlockIterator.nextBlock(); + assertEquals(blockData.getLocalID(), counter++); + } } } @@ -226,11 +228,12 @@ public void testKeyValueBlockIteratorWithOnlyDeletedBlocks() throws createContainerWithBlocks(containerId, 0, 5); String containerPath = new File(containerData.getMetadataPath()) .getParent(); - KeyValueBlockIterator keyValueBlockIterator = new KeyValueBlockIterator( - containerId, new File(containerPath)); - //As all blocks are deleted blocks, blocks does not match with normal key - // filter. - assertFalse(keyValueBlockIterator.hasNext()); + try(KeyValueBlockIterator keyValueBlockIterator = new KeyValueBlockIterator( + containerId, new File(containerPath))) { + //As all blocks are deleted blocks, blocks does not match with normal key + // filter. + assertFalse(keyValueBlockIterator.hasNext()); + } } /** @@ -251,27 +254,30 @@ private void createContainerWithBlocks(long containerId, int container = new KeyValueContainer(containerData, conf); container.create(volumeSet, new RoundRobinVolumeChoosingPolicy(), UUID .randomUUID().toString()); - MetadataStore metadataStore = BlockUtils.getDB(containerData, conf); - - List chunkList = new ArrayList<>(); - ChunkInfo info = new ChunkInfo("chunkfile", 0, 1024); - chunkList.add(info.getProtoBufMessage()); - - for (int i=0; i chunkList = new ArrayList<>(); + ChunkInfo info = new ChunkInfo("chunkfile", 0, 1024); + chunkList.add(info.getProtoBufMessage()); + + for (int i = 0; i < normalBlocks; i++) { + BlockID blockID = new BlockID(containerId, i); + BlockData blockData = new BlockData(blockID); + blockData.setChunks(chunkList); + metadataStore.getStore().put(Longs.toByteArray(blockID.getLocalID()), + blockData + .getProtoBufMessage().toByteArray()); + } + + for (int i = normalBlocks; i < deletedBlocks; i++) { + BlockID blockID = new BlockID(containerId, i); + BlockData blockData = new BlockData(blockID); + blockData.setChunks(chunkList); + metadataStore.getStore().put(DFSUtil.string2Bytes(OzoneConsts + .DELETING_KEY_PREFIX + blockID.getLocalID()), blockData + .getProtoBufMessage().toByteArray()); + } } } diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java index 8e2986cca6fb2..81d3065833ebe 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java @@ -36,7 +36,7 @@ import org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils; import org.apache.hadoop.test.GenericTestUtils; import org.apache.hadoop.util.DiskChecker; -import org.apache.hadoop.utils.MetadataStore; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import org.junit.Assert; import org.junit.Before; @@ -132,23 +132,24 @@ public void testBlockIterator() throws Exception{ private void addBlocks(int count) throws Exception { long containerId = keyValueContainerData.getContainerID(); - MetadataStore metadataStore = BlockUtils.getDB(keyValueContainer - .getContainerData(), conf); - for (int i=0; i < count; i++) { - // Creating BlockData - BlockID blockID = new BlockID(containerId, i); - BlockData blockData = new BlockData(blockID); - blockData.addMetadata("VOLUME", "ozone"); - blockData.addMetadata("OWNER", "hdfs"); - List chunkList = new ArrayList<>(); - ChunkInfo info = new ChunkInfo(String.format("%d.data.%d", blockID - .getLocalID(), 0), 0, 1024); - chunkList.add(info.getProtoBufMessage()); - blockData.setChunks(chunkList); - metadataStore.put(Longs.toByteArray(blockID.getLocalID()), blockData - .getProtoBufMessage().toByteArray()); + try(ReferenceCountedDB metadataStore = BlockUtils.getDB(keyValueContainer + .getContainerData(), conf)) { + for (int i = 0; i < count; i++) { + // Creating BlockData + BlockID blockID = new BlockID(containerId, i); + BlockData blockData = new BlockData(blockID); + blockData.addMetadata("VOLUME", "ozone"); + blockData.addMetadata("OWNER", "hdfs"); + List chunkList = new ArrayList<>(); + ChunkInfo info = new ChunkInfo(String.format("%d.data.%d", blockID + .getLocalID(), 0), 0, 1024); + chunkList.add(info.getProtoBufMessage()); + blockData.setChunks(chunkList); + metadataStore.getStore().put(Longs.toByteArray(blockID.getLocalID()), + blockData + .getProtoBufMessage().toByteArray()); + } } - } @SuppressWarnings("RedundantCast") @@ -191,9 +192,12 @@ public void testContainerImportExport() throws Exception { int numberOfKeysToWrite = 12; //write one few keys to check the key count after import - MetadataStore metadataStore = BlockUtils.getDB(keyValueContainerData, conf); - for (int i = 0; i < numberOfKeysToWrite; i++) { - metadataStore.put(("test" + i).getBytes(UTF_8), "test".getBytes(UTF_8)); + try(ReferenceCountedDB metadataStore = + BlockUtils.getDB(keyValueContainerData, conf)) { + for (int i = 0; i < numberOfKeysToWrite; i++) { + metadataStore.getStore().put(("test" + i).getBytes(UTF_8), + "test".getBytes(UTF_8)); + } } BlockUtils.removeDB(keyValueContainerData, conf); diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java index 0bc1bbc387b39..fe702fc693a26 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java @@ -19,23 +19,29 @@ package org.apache.hadoop.ozone.container.keyvalue; import com.google.common.primitives.Longs; -import org.apache.hadoop.conf.Configuration; +import org.apache.commons.lang3.RandomStringUtils; import org.apache.hadoop.conf.StorageUnit; import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.hdds.client.BlockID; import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; import org.apache.hadoop.hdfs.DFSUtil; +import org.apache.hadoop.hdfs.util.DataTransferThrottler; import org.apache.hadoop.ozone.OzoneConsts; +import org.apache.hadoop.ozone.common.Checksum; +import org.apache.hadoop.ozone.common.ChecksumData; import org.apache.hadoop.ozone.container.common.helpers.BlockData; import org.apache.hadoop.ozone.container.common.helpers.ChunkInfo; import org.apache.hadoop.ozone.container.common.transport.server.ratis.DispatcherContext; +import org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils; +import org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerLocationUtil; import org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerImpl; import org.apache.hadoop.ozone.container.common.volume.RoundRobinVolumeChoosingPolicy; import org.apache.hadoop.ozone.container.common.volume.VolumeSet; import org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils; +import org.apache.hadoop.ozone.container.ozoneimpl.ContainerScrubberConfiguration; import org.apache.hadoop.test.GenericTestUtils; -import org.apache.hadoop.utils.MetadataStore; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import org.junit.After; import org.junit.Before; import org.junit.Test; @@ -43,6 +49,7 @@ import org.junit.runners.Parameterized; import java.io.File; +import java.io.RandomAccessFile; import java.util.Arrays; import java.util.ArrayList; import java.nio.ByteBuffer; @@ -55,8 +62,11 @@ import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_LEVELDB; import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB; -import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertFalse; + /** * Basic sanity test for the KeyValueContainerCheck class. @@ -65,9 +75,8 @@ private final String storeImpl; private KeyValueContainer container; private KeyValueContainerData containerData; - private ChunkManagerImpl chunkManager; private VolumeSet volumeSet; - private Configuration conf; + private OzoneConfiguration conf; private File testRoot; public TestKeyValueContainerCheck(String metadataImpl) { @@ -94,104 +103,168 @@ public TestKeyValueContainerCheck(String metadataImpl) { /** * Sanity test, when there are no corruptions induced. - * @throws Exception */ - @Test public void testKeyValueContainerCheckNoCorruption() throws Exception { + @Test + public void testKeyValueContainerCheckNoCorruption() throws Exception { long containerID = 101; int deletedBlocks = 1; int normalBlocks = 3; int chunksPerBlock = 4; - boolean corruption = false; + ContainerScrubberConfiguration c = conf.getObject( + ContainerScrubberConfiguration.class); // test Closed Container - createContainerWithBlocks(containerID, normalBlocks, deletedBlocks, 65536, + createContainerWithBlocks(containerID, normalBlocks, deletedBlocks, chunksPerBlock); - File chunksPath = new File(containerData.getChunksPath()); - assertTrue(chunksPath.listFiles().length - == (deletedBlocks + normalBlocks) * chunksPerBlock); KeyValueContainerCheck kvCheck = new KeyValueContainerCheck(containerData.getMetadataPath(), conf, containerID); // first run checks on a Open Container - corruption = kvCheck.fastCheck(); - assertFalse(corruption); + boolean valid = kvCheck.fastCheck(); + assertTrue(valid); container.close(); // next run checks on a Closed Container - corruption = kvCheck.fullCheck(); - assertFalse(corruption); + valid = kvCheck.fullCheck(new DataTransferThrottler( + c.getBandwidthPerVolume()), null); + assertTrue(valid); + } + + /** + * Sanity test, when there are corruptions induced. + */ + @Test + public void testKeyValueContainerCheckCorruption() throws Exception { + long containerID = 102; + int deletedBlocks = 1; + int normalBlocks = 3; + int chunksPerBlock = 4; + ContainerScrubberConfiguration sc = conf.getObject( + ContainerScrubberConfiguration.class); + + // test Closed Container + createContainerWithBlocks(containerID, normalBlocks, deletedBlocks, + chunksPerBlock); + + container.close(); + + KeyValueContainerCheck kvCheck = + new KeyValueContainerCheck(containerData.getMetadataPath(), conf, + containerID); + + File metaDir = new File(containerData.getMetadataPath()); + File dbFile = KeyValueContainerLocationUtil + .getContainerDBFile(metaDir, containerID); + containerData.setDbFile(dbFile); + try (ReferenceCountedDB ignored = + BlockUtils.getDB(containerData, conf); + KeyValueBlockIterator kvIter = new KeyValueBlockIterator(containerID, + new File(containerData.getContainerPath()))) { + BlockData block = kvIter.nextBlock(); + assertFalse(block.getChunks().isEmpty()); + ContainerProtos.ChunkInfo c = block.getChunks().get(0); + File chunkFile = ChunkUtils.getChunkFile(containerData, + ChunkInfo.getFromProtoBuf(c)); + long length = chunkFile.length(); + assertTrue(length > 0); + // forcefully truncate the file to induce failure. + try (RandomAccessFile file = new RandomAccessFile(chunkFile, "rws")) { + file.setLength(length / 2); + } + assertEquals(length/2, chunkFile.length()); + } + + // metadata check should pass. + boolean valid = kvCheck.fastCheck(); + assertTrue(valid); + + // checksum validation should fail. + valid = kvCheck.fullCheck(new DataTransferThrottler( + sc.getBandwidthPerVolume()), null); + assertFalse(valid); } /** * Creates a container with normal and deleted blocks. * First it will insert normal blocks, and then it will insert * deleted blocks. - * @param containerId - * @param normalBlocks - * @param deletedBlocks - * @throws Exception */ private void createContainerWithBlocks(long containerId, int normalBlocks, - int deletedBlocks, long chunkLen, int chunksPerBlock) throws Exception { - long chunkCount; + int deletedBlocks, int chunksPerBlock) throws Exception { String strBlock = "block"; String strChunk = "-chunkFile"; - byte[] chunkData = new byte[(int) chunkLen]; - long totalBlks = normalBlocks + deletedBlocks; + long totalBlocks = normalBlocks + deletedBlocks; + int unitLen = 1024; + int chunkLen = 3 * unitLen; + int bytesPerChecksum = 2 * unitLen; + Checksum checksum = new Checksum(ContainerProtos.ChecksumType.SHA256, + bytesPerChecksum); + byte[] chunkData = RandomStringUtils.randomAscii(chunkLen).getBytes(); + ChecksumData checksumData = checksum.computeChecksum(chunkData); containerData = new KeyValueContainerData(containerId, (long) StorageUnit.BYTES.toBytes( - chunksPerBlock * chunkLen * totalBlks), + chunksPerBlock * chunkLen * totalBlocks), UUID.randomUUID().toString(), UUID.randomUUID().toString()); container = new KeyValueContainer(containerData, conf); container.create(volumeSet, new RoundRobinVolumeChoosingPolicy(), UUID.randomUUID().toString()); - MetadataStore metadataStore = BlockUtils.getDB(containerData, conf); - chunkManager = new ChunkManagerImpl(true); - - assertTrue(containerData.getChunksPath() != null); - File chunksPath = new File(containerData.getChunksPath()); - assertTrue(chunksPath.exists()); - // Initially chunks folder should be empty. - assertTrue(chunksPath.listFiles().length == 0); - - List chunkList = new ArrayList<>(); - for (int i = 0; i < (totalBlks); i++) { - BlockID blockID = new BlockID(containerId, i); - BlockData blockData = new BlockData(blockID); - - chunkList.clear(); - for (chunkCount = 0; chunkCount < chunksPerBlock; chunkCount++) { - String chunkName = strBlock + i + strChunk + chunkCount; - long offset = chunkCount * chunkLen; - ChunkInfo info = new ChunkInfo(chunkName, offset, chunkLen); - chunkList.add(info.getProtoBufMessage()); - chunkManager - .writeChunk(container, blockID, info, ByteBuffer.wrap(chunkData), - new DispatcherContext.Builder() - .setStage(DispatcherContext.WriteChunkStage.WRITE_DATA) - .build()); - chunkManager - .writeChunk(container, blockID, info, ByteBuffer.wrap(chunkData), - new DispatcherContext.Builder() - .setStage(DispatcherContext.WriteChunkStage.COMMIT_DATA) - .build()); - } - blockData.setChunks(chunkList); - - if (i >= normalBlocks) { - // deleted key - metadataStore.put(DFSUtil.string2Bytes( - OzoneConsts.DELETING_KEY_PREFIX + blockID.getLocalID()), - blockData.getProtoBufMessage().toByteArray()); - } else { - // normal key - metadataStore.put(Longs.toByteArray(blockID.getLocalID()), - blockData.getProtoBufMessage().toByteArray()); + try (ReferenceCountedDB metadataStore = BlockUtils.getDB(containerData, + conf)) { + ChunkManagerImpl chunkManager = new ChunkManagerImpl(true); + + assertNotNull(containerData.getChunksPath()); + File chunksPath = new File(containerData.getChunksPath()); + assertTrue(chunksPath.exists()); + // Initially chunks folder should be empty. + File[] chunkFilesBefore = chunksPath.listFiles(); + assertNotNull(chunkFilesBefore); + assertEquals(0, chunkFilesBefore.length); + + List chunkList = new ArrayList<>(); + for (int i = 0; i < totalBlocks; i++) { + BlockID blockID = new BlockID(containerId, i); + BlockData blockData = new BlockData(blockID); + + chunkList.clear(); + for (long chunkCount = 0; chunkCount < chunksPerBlock; chunkCount++) { + String chunkName = strBlock + i + strChunk + chunkCount; + ChunkInfo info = new ChunkInfo(chunkName, 0, chunkLen); + info.setChecksumData(checksumData); + chunkList.add(info.getProtoBufMessage()); + chunkManager + .writeChunk(container, blockID, info, ByteBuffer.wrap(chunkData), + new DispatcherContext.Builder() + .setStage(DispatcherContext.WriteChunkStage.WRITE_DATA) + .build()); + chunkManager + .writeChunk(container, blockID, info, ByteBuffer.wrap(chunkData), + new DispatcherContext.Builder() + .setStage(DispatcherContext.WriteChunkStage.COMMIT_DATA) + .build()); + } + blockData.setChunks(chunkList); + + if (i >= normalBlocks) { + // deleted key + metadataStore.getStore().put(DFSUtil.string2Bytes( + OzoneConsts.DELETING_KEY_PREFIX + blockID.getLocalID()), + blockData.getProtoBufMessage().toByteArray()); + } else { + // normal key + metadataStore.getStore().put(Longs.toByteArray(blockID.getLocalID()), + blockData.getProtoBufMessage().toByteArray()); + } } + + File[] chunkFilesAfter = chunksPath.listFiles(); + assertNotNull(chunkFilesAfter); + assertEquals((deletedBlocks + normalBlocks) * chunksPerBlock, + chunkFilesAfter.length); } } + } \ No newline at end of file diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerMarkUnhealthy.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerMarkUnhealthy.java index e11bca582d0f4..c3e67c7ae6b4e 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerMarkUnhealthy.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerMarkUnhealthy.java @@ -155,6 +155,9 @@ public void testMarkClosedContainerAsUnhealthy() throws IOException { */ @Test public void testMarkQuasiClosedContainerAsUnhealthy() throws IOException { + // We need to create the container so the sync-on-quasi-close operation + // does not NPE. + keyValueContainer.create(volumeSet, volumeChoosingPolicy, scmId); keyValueContainer.quasiClose(); keyValueContainer.markContainerUnhealthy(); assertThat(keyValueContainerData.getState(), is(UNHEALTHY)); diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java index 8ef9e19d53712..2c71fef11a646 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java @@ -229,14 +229,14 @@ public void testHandlerCommandHandling() throws Exception { @Test public void testVolumeSetInKeyValueHandler() throws Exception{ File path = GenericTestUtils.getRandomizedTestDir(); + Configuration conf = new OzoneConfiguration(); + conf.set(HDDS_DATANODE_DIR_KEY, path.getAbsolutePath()); + VolumeSet volumeSet = new VolumeSet(UUID.randomUUID().toString(), conf); try { - Configuration conf = new OzoneConfiguration(); - conf.set(HDDS_DATANODE_DIR_KEY, path.getAbsolutePath()); ContainerSet cset = new ContainerSet(); int[] interval = new int[1]; interval[0] = 2; ContainerMetrics metrics = new ContainerMetrics(interval); - VolumeSet volumeSet = new VolumeSet(UUID.randomUUID().toString(), conf); DatanodeDetails datanodeDetails = Mockito.mock(DatanodeDetails.class); DatanodeStateMachine stateMachine = Mockito.mock( DatanodeStateMachine.class); @@ -263,6 +263,7 @@ public void testVolumeSetInKeyValueHandler() throws Exception{ ex); } } finally { + volumeSet.shutdown(); FileUtil.fullyDelete(path); } } diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/helpers/TestChunkUtils.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/helpers/TestChunkUtils.java new file mode 100644 index 0000000000000..4a1637cb16914 --- /dev/null +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/helpers/TestChunkUtils.java @@ -0,0 +1,164 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ozone.container.keyvalue.helpers; + +import org.apache.commons.io.FileUtils; +import org.apache.hadoop.ozone.container.common.helpers.ChunkInfo; +import org.apache.hadoop.ozone.container.common.volume.VolumeIOStats; +import org.apache.hadoop.test.GenericTestUtils; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.nio.ByteBuffer; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Arrays; +import java.util.LinkedList; +import java.util.List; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; + +/** + * Tests for {@link ChunkUtils}. + */ +public class TestChunkUtils { + + private static final Logger LOG = + LoggerFactory.getLogger(TestChunkUtils.class); + + private static final String PREFIX = TestChunkUtils.class.getSimpleName(); + + @Test + public void concurrentReadOfSameFile() throws Exception { + String s = "Hello World"; + byte[] array = s.getBytes(); + ByteBuffer data = ByteBuffer.wrap(array); + Path tempFile = Files.createTempFile(PREFIX, "concurrent"); + try { + ChunkInfo chunkInfo = new ChunkInfo(tempFile.toString(), + 0, data.capacity()); + File file = tempFile.toFile(); + VolumeIOStats stats = new VolumeIOStats(); + ChunkUtils.writeData(file, chunkInfo, data, stats, true); + int threads = 10; + ExecutorService executor = new ThreadPoolExecutor(threads, threads, + 0, TimeUnit.SECONDS, new LinkedBlockingQueue<>()); + AtomicInteger processed = new AtomicInteger(); + AtomicBoolean failed = new AtomicBoolean(); + for (int i = 0; i < threads; i++) { + final int threadNumber = i; + executor.submit(() -> { + try { + ByteBuffer readBuffer = ChunkUtils.readData(file, chunkInfo, stats); + LOG.info("Read data ({}): {}", threadNumber, + new String(readBuffer.array())); + if (!Arrays.equals(array, readBuffer.array())) { + failed.set(true); + } + } catch (Exception e) { + LOG.error("Failed to read data ({})", threadNumber, e); + failed.set(true); + } + processed.incrementAndGet(); + }); + } + try { + GenericTestUtils.waitFor(() -> processed.get() == threads, + 100, (int) TimeUnit.SECONDS.toMillis(5)); + } finally { + executor.shutdownNow(); + } + assertEquals(threads * stats.getWriteBytes(), stats.getReadBytes()); + assertFalse(failed.get()); + } finally { + Files.deleteIfExists(tempFile); + } + } + + @Test + public void concurrentProcessing() throws Exception { + final int perThreadWait = 1000; + final int maxTotalWait = 5000; + int threads = 20; + List paths = new LinkedList<>(); + + try { + ExecutorService executor = new ThreadPoolExecutor(threads, threads, + 0, TimeUnit.SECONDS, new LinkedBlockingQueue<>()); + AtomicInteger processed = new AtomicInteger(); + for (int i = 0; i < threads; i++) { + Path path = Files.createTempFile(PREFIX, String.valueOf(i)); + paths.add(path); + executor.submit(() -> { + ChunkUtils.processFileExclusively(path, () -> { + try { + Thread.sleep(perThreadWait); + } catch (InterruptedException e) { + e.printStackTrace(); + } + processed.incrementAndGet(); + return null; + }); + }); + } + try { + GenericTestUtils.waitFor(() -> processed.get() == threads, + 100, maxTotalWait); + } finally { + executor.shutdownNow(); + } + } finally { + for (Path path : paths) { + FileUtils.deleteQuietly(path.toFile()); + } + } + } + + @Test + public void serialRead() throws Exception { + String s = "Hello World"; + byte[] array = s.getBytes(); + ByteBuffer data = ByteBuffer.wrap(array); + Path tempFile = Files.createTempFile(PREFIX, "serial"); + try { + ChunkInfo chunkInfo = new ChunkInfo(tempFile.toString(), + 0, data.capacity()); + File file = tempFile.toFile(); + VolumeIOStats stats = new VolumeIOStats(); + ChunkUtils.writeData(file, chunkInfo, data, stats, true); + ByteBuffer readBuffer = ChunkUtils.readData(file, chunkInfo, stats); + assertArrayEquals(array, readBuffer.array()); + assertEquals(stats.getWriteBytes(), stats.getReadBytes()); + } catch (Exception e) { + LOG.error("Failed to read data", e); + } finally { + Files.deleteIfExists(tempFile); + } + } + +} diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestContainerScrubberMetrics.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestContainerScrubberMetrics.java new file mode 100644 index 0000000000000..b9b1beabdbd11 --- /dev/null +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestContainerScrubberMetrics.java @@ -0,0 +1,110 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

    + * http://www.apache.org/licenses/LICENSE-2.0 + *

    + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ozone.container.ozoneimpl; + +import org.apache.hadoop.hdds.conf.OzoneConfiguration; +import org.apache.hadoop.hdfs.util.Canceler; +import org.apache.hadoop.hdfs.util.DataTransferThrottler; +import org.apache.hadoop.ozone.container.common.impl.ContainerData; +import org.apache.hadoop.ozone.container.common.interfaces.Container; +import org.apache.hadoop.ozone.container.common.volume.HddsVolume; +import org.junit.Assert; +import org.junit.Test; +import org.mockito.Mockito; + +import java.util.Arrays; +import java.util.Collection; + +/** + * This test verifies the container scrubber metrics functionality. + */ +public class TestContainerScrubberMetrics { + @Test + public void testContainerMetaDataScrubberMetrics() { + OzoneConfiguration conf = new OzoneConfiguration(); + ContainerScrubberConfiguration c = conf.getObject( + ContainerScrubberConfiguration.class); + c.setMetadataScanInterval(0); + HddsVolume vol = Mockito.mock(HddsVolume.class); + ContainerController cntrl = mockContainerController(vol); + + ContainerMetadataScanner mc = new ContainerMetadataScanner(c, cntrl); + mc.runIteration(); + + Assert.assertEquals(1, mc.getMetrics().getNumScanIterations()); + Assert.assertEquals(3, mc.getMetrics().getNumContainersScanned()); + Assert.assertEquals(1, mc.getMetrics().getNumUnHealthyContainers()); + } + + @Test + public void testContainerDataScrubberMetrics() { + OzoneConfiguration conf = new OzoneConfiguration(); + ContainerScrubberConfiguration c = conf.getObject( + ContainerScrubberConfiguration.class); + c.setDataScanInterval(0); + HddsVolume vol = Mockito.mock(HddsVolume.class); + ContainerController cntrl = mockContainerController(vol); + + ContainerDataScanner sc = new ContainerDataScanner(c, cntrl, vol); + sc.runIteration(); + + ContainerDataScrubberMetrics m = sc.getMetrics(); + Assert.assertEquals(1, m.getNumScanIterations()); + Assert.assertEquals(2, m.getNumContainersScanned()); + Assert.assertEquals(1, m.getNumUnHealthyContainers()); + } + + private ContainerController mockContainerController(HddsVolume vol) { + // healthy container + Container c1 = Mockito.mock(Container.class); + Mockito.when(c1.shouldScanData()).thenReturn(true); + Mockito.when(c1.scanMetaData()).thenReturn(true); + Mockito.when(c1.scanData( + Mockito.any(DataTransferThrottler.class), + Mockito.any(Canceler.class))).thenReturn(true); + + // unhealthy container (corrupt data) + ContainerData c2d = Mockito.mock(ContainerData.class); + Mockito.when(c2d.getContainerID()).thenReturn(101L); + Container c2 = Mockito.mock(Container.class); + Mockito.when(c2.scanMetaData()).thenReturn(true); + Mockito.when(c2.shouldScanData()).thenReturn(true); + Mockito.when(c2.scanData( + Mockito.any(DataTransferThrottler.class), + Mockito.any(Canceler.class))).thenReturn(false); + Mockito.when(c2.getContainerData()).thenReturn(c2d); + + // unhealthy container (corrupt metadata) + ContainerData c3d = Mockito.mock(ContainerData.class); + Mockito.when(c3d.getContainerID()).thenReturn(102L); + Container c3 = Mockito.mock(Container.class); + Mockito.when(c3.shouldScanData()).thenReturn(false); + Mockito.when(c3.scanMetaData()).thenReturn(false); + Mockito.when(c3.getContainerData()).thenReturn(c3d); + + Collection> containers = Arrays.asList(c1, c2, c3); + ContainerController cntrl = Mockito.mock(ContainerController.class); + Mockito.when(cntrl.getContainers(vol)) + .thenReturn(containers.iterator()); + Mockito.when(cntrl.getContainers()) + .thenReturn(containers.iterator()); + + return cntrl; + } + +} diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java index 003f26e7a4902..2d679a1cb45db 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java @@ -19,29 +19,45 @@ package org.apache.hadoop.ozone.container.ozoneimpl; +import com.google.common.base.Preconditions; +import com.google.common.primitives.Longs; import org.apache.hadoop.conf.StorageUnit; import org.apache.hadoop.hdds.HddsConfigKeys; +import org.apache.hadoop.hdds.client.BlockID; import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdds.protocol.DatanodeDetails; +import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; import org.apache.hadoop.hdds.scm.ScmConfigKeys; +import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException; +import org.apache.hadoop.ozone.container.common.helpers.BlockData; +import org.apache.hadoop.ozone.container.common.helpers.ChunkInfo; import org.apache.hadoop.ozone.container.common.impl.ContainerSet; import org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine; import org.apache.hadoop.ozone.container.common.statemachine.StateContext; +import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB; import org.apache.hadoop.ozone.container.common.volume.HddsVolume; import org.apache.hadoop.ozone.container.common.volume.RoundRobinVolumeChoosingPolicy; import org.apache.hadoop.ozone.container.common.volume.VolumeSet; import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer; import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerData; +import org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils; +import org.apache.hadoop.test.LambdaTestUtils; +import org.junit.After; import org.junit.Before; import org.junit.Rule; import org.junit.Test; import org.junit.rules.TemporaryFolder; import org.mockito.Mockito; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import java.util.Random; import java.util.UUID; +import java.util.HashMap; +import java.util.List; +import java.util.ArrayList; - +import static org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.Result.DISK_OUT_OF_SPACE; import static org.junit.Assert.assertEquals; /** @@ -49,10 +65,12 @@ */ public class TestOzoneContainer { + private static final Logger LOG = + LoggerFactory.getLogger(TestOzoneContainer.class); + @Rule public TemporaryFolder folder = new TemporaryFolder(); - private OzoneConfiguration conf; private String scmId = UUID.randomUUID().toString(); private VolumeSet volumeSet; @@ -60,6 +78,8 @@ public class TestOzoneContainer { private KeyValueContainerData keyValueContainerData; private KeyValueContainer keyValueContainer; private final DatanodeDetails datanodeDetails = createDatanodeDetails(); + private HashMap commitSpaceMap; //RootDir -> committed space + private final int numTestContainers = 10; @Before public void setUp() throws Exception { @@ -68,26 +88,50 @@ public void setUp() throws Exception { .getAbsolutePath()); conf.set(HddsConfigKeys.OZONE_METADATA_DIRS, folder.newFolder().getAbsolutePath()); + commitSpaceMap = new HashMap(); + volumeSet = new VolumeSet(datanodeDetails.getUuidString(), conf); + volumeChoosingPolicy = new RoundRobinVolumeChoosingPolicy(); + } + + @After + public void cleanUp() throws Exception { + if (volumeSet != null) { + volumeSet.shutdown(); + volumeSet = null; + } } @Test public void testBuildContainerMap() throws Exception { - volumeSet = new VolumeSet(datanodeDetails.getUuidString(), conf); - volumeChoosingPolicy = new RoundRobinVolumeChoosingPolicy(); - // Format the volumes for (HddsVolume volume : volumeSet.getVolumesList()) { volume.format(UUID.randomUUID().toString()); + commitSpaceMap.put(getVolumeKey(volume), Long.valueOf(0)); } // Add containers to disk - for (int i=0; i<10; i++) { + for (int i = 0; i < numTestContainers; i++) { + long freeBytes = 0; + long volCommitBytes; + long maxCap = (long) StorageUnit.GB.toBytes(1); + + HddsVolume myVolume; + keyValueContainerData = new KeyValueContainerData(i, - (long) StorageUnit.GB.toBytes(1), UUID.randomUUID().toString(), + maxCap, UUID.randomUUID().toString(), datanodeDetails.getUuidString()); keyValueContainer = new KeyValueContainer( keyValueContainerData, conf); keyValueContainer.create(volumeSet, volumeChoosingPolicy, scmId); + myVolume = keyValueContainer.getContainerData().getVolume(); + + freeBytes = addBlocks(keyValueContainer, 2, 3); + + // update our expectation of volume committed space in the map + volCommitBytes = commitSpaceMap.get(getVolumeKey(myVolume)).longValue(); + Preconditions.checkState(freeBytes >= 0); + commitSpaceMap.put(getVolumeKey(myVolume), + Long.valueOf(volCommitBytes + freeBytes)); } DatanodeStateMachine stateMachine = Mockito.mock( @@ -97,12 +141,93 @@ public void testBuildContainerMap() throws Exception { Mockito.when(context.getParent()).thenReturn(stateMachine); // When OzoneContainer is started, the containers from disk should be // loaded into the containerSet. + // Also expected to initialize committed space for each volume. OzoneContainer ozoneContainer = new OzoneContainer(datanodeDetails, conf, context, null); + ContainerSet containerset = ozoneContainer.getContainerSet(); - assertEquals(10, containerset.containerCount()); + assertEquals(numTestContainers, containerset.containerCount()); + + verifyCommittedSpace(ozoneContainer); } + @Test + public void testContainerCreateDiskFull() throws Exception { + long containerSize = (long) StorageUnit.MB.toBytes(100); + + // Format the volumes + for (HddsVolume volume : volumeSet.getVolumesList()) { + volume.format(UUID.randomUUID().toString()); + + // eat up all available space except size of 1 container + volume.incCommittedBytes(volume.getAvailable() - containerSize); + // eat up 10 bytes more, now available space is less than 1 container + volume.incCommittedBytes(10); + } + keyValueContainerData = new KeyValueContainerData(99, containerSize, + UUID.randomUUID().toString(), datanodeDetails.getUuidString()); + keyValueContainer = new KeyValueContainer(keyValueContainerData, conf); + + // we expect an out of space Exception + StorageContainerException e = LambdaTestUtils.intercept( + StorageContainerException.class, + () -> keyValueContainer.create(volumeSet, volumeChoosingPolicy, scmId) + ); + if (!DISK_OUT_OF_SPACE.equals(e.getResult())) { + LOG.info("Unexpected error during container creation", e); + } + assertEquals(DISK_OUT_OF_SPACE, e.getResult()); + } + + //verify committed space on each volume + private void verifyCommittedSpace(OzoneContainer oc) { + for (HddsVolume dnVol : oc.getVolumeSet().getVolumesList()) { + String key = getVolumeKey(dnVol); + long expectedCommit = commitSpaceMap.get(key).longValue(); + long volumeCommitted = dnVol.getCommittedBytes(); + assertEquals("Volume committed space not initialized correctly", + expectedCommit, volumeCommitted); + } + } + + private long addBlocks(KeyValueContainer container, + int blocks, int chunksPerBlock) throws Exception { + String strBlock = "block"; + String strChunk = "-chunkFile"; + int datalen = 65536; + long usedBytes = 0; + + long freeBytes = container.getContainerData().getMaxSize(); + long containerId = container.getContainerData().getContainerID(); + ReferenceCountedDB db = BlockUtils.getDB(container + .getContainerData(), conf); + + for (int bi = 0; bi < blocks; bi++) { + // Creating BlockData + BlockID blockID = new BlockID(containerId, bi); + BlockData blockData = new BlockData(blockID); + List chunkList = new ArrayList<>(); + + chunkList.clear(); + for (int ci = 0; ci < chunksPerBlock; ci++) { + String chunkName = strBlock + bi + strChunk + ci; + long offset = ci * datalen; + ChunkInfo info = new ChunkInfo(chunkName, offset, datalen); + usedBytes += datalen; + chunkList.add(info.getProtoBufMessage()); + } + blockData.setChunks(chunkList); + db.getStore().put(Longs.toByteArray(blockID.getLocalID()), + blockData.getProtoBufMessage().toByteArray()); + } + + // remaining available capacity of the container + return (freeBytes - usedBytes); + } + + private String getVolumeKey(HddsVolume volume) { + return volume.getHddsRootDir().getPath(); + } private DatanodeDetails createDatanodeDetails() { Random random = new Random(); diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/testutils/BlockDeletingServiceTestImpl.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/testutils/BlockDeletingServiceTestImpl.java index 115b5e2cf84d0..a136983415b7d 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/testutils/BlockDeletingServiceTestImpl.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/testutils/BlockDeletingServiceTestImpl.java @@ -19,9 +19,9 @@ import com.google.common.annotations.VisibleForTesting; import com.google.common.util.concurrent.ThreadFactoryBuilder; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.ozone.container.common.impl.ContainerSet; import org.apache.hadoop.ozone.container.keyvalue.statemachine.background .BlockDeletingService; +import org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer; import java.util.concurrent.CountDownLatch; import java.util.concurrent.Future; @@ -42,9 +42,9 @@ public class BlockDeletingServiceTestImpl private Thread testingThread; private AtomicInteger numOfProcessed = new AtomicInteger(0); - public BlockDeletingServiceTestImpl(ContainerSet containerSet, + public BlockDeletingServiceTestImpl(OzoneContainer container, int serviceInterval, Configuration conf) { - super(containerSet, serviceInterval, SERVICE_TIMEOUT_IN_MILLISECONDS, + super(container, serviceInterval, SERVICE_TIMEOUT_IN_MILLISECONDS, TimeUnit.MILLISECONDS, conf); } @@ -67,7 +67,8 @@ public int getTimesOfProcessed() { } // Override the implementation to start a single on-call control thread. - @Override public void start() { + @Override + public void start() { PeriodicalTask svc = new PeriodicalTask(); // In test mode, relies on a latch countdown to runDeletingTasks tasks. Runnable r = () -> { diff --git a/hadoop-hdds/dev-support/checkstyle/checkstyle-noframes-sorted.xsl b/hadoop-hdds/dev-support/checkstyle/checkstyle-noframes-sorted.xsl new file mode 100644 index 0000000000000..7f2aedf8675f3 --- /dev/null +++ b/hadoop-hdds/dev-support/checkstyle/checkstyle-noframes-sorted.xsl @@ -0,0 +1,189 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + +

    + + + + + + + +
    + +

    CheckStyle Audit

    Designed for use with CheckStyle and Ant.
    +
    + + + +
    + + + +
    + + + + +
    + + + + + + +

    Files

    + + + + + + + + + + + + + + +
    NameErrors
    +     + + + +

    File

    + + + + + + + + + + + + + + +
    Error DescriptionLine
    + Back to top +     + + +

    Summary

    + + + + + + + + + + + + +
    FilesErrors
    +     + + + + a + b + + + + + diff --git a/hadoop-hdds/dev-support/checkstyle/checkstyle.xml b/hadoop-hdds/dev-support/checkstyle/checkstyle.xml new file mode 100644 index 0000000000000..1c437418ccfa3 --- /dev/null +++ b/hadoop-hdds/dev-support/checkstyle/checkstyle.xml @@ -0,0 +1,196 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/hadoop-hdds/dev-support/checkstyle/suppressions.xml b/hadoop-hdds/dev-support/checkstyle/suppressions.xml new file mode 100644 index 0000000000000..7bc94797df856 --- /dev/null +++ b/hadoop-hdds/dev-support/checkstyle/suppressions.xml @@ -0,0 +1,21 @@ + + + + + + + diff --git a/hadoop-hdds/docs/config.yaml b/hadoop-hdds/docs/config.yaml index 2db1147cd5a54..7b75888fb2876 100644 --- a/hadoop-hdds/docs/config.yaml +++ b/hadoop-hdds/docs/config.yaml @@ -21,26 +21,6 @@ theme: "ozonedoc" pygmentsCodeFences: true uglyurls: true relativeURLs: true - -menu: - main: - - identifier: Starting - name: "Getting Started" - title: "Getting Started" - url: runningviadocker.html - weight: 1 - - identifier: Client - name: Client - title: Client - url: commandshell.html - weight: 2 - - identifier: Tools - name: Tools - title: Tools - url: dozone.html - weight: 3 - - identifier: Recipes - name: Recipes - title: Recipes - url: prometheus.html - weight: 4 \ No newline at end of file +disableKinds: +- taxonomy +- taxonomyTerm \ No newline at end of file diff --git a/hadoop-hdds/docs/content/BucketCommands.md b/hadoop-hdds/docs/content/BucketCommands.md deleted file mode 100644 index c726f6d82f70d..0000000000000 --- a/hadoop-hdds/docs/content/BucketCommands.md +++ /dev/null @@ -1,131 +0,0 @@ ---- -title: Bucket Commands -menu: - main: - parent: OzoneShell - weight: 2 ---- - - -Ozone shell supports the following bucket commands. - - * [create](#create) - * [delete](#delete) - * [info](#info) - * [list](#list) - * [update](#update) - -### Create - -The bucket create command allows users to create a bucket. - -***Params:*** - -| Arguments | Comment | -|--------------------------------|-----------------------------------------| -| Uri | The name of the bucket in **/volume/bucket** format. - -{{< highlight bash >}} -ozone sh bucket create /hive/jan -{{< /highlight >}} - -The above command will create a bucket called _jan_ in the _hive_ volume. -Since no scheme was specified this command defaults to O3 (RPC) protocol. - -### Delete - -The bucket delete command allows users to delete a bucket. If the -bucket is not empty then this command will fail. - -***Params:*** - -| Arguments | Comment | -|--------------------------------|-----------------------------------------| -| Uri | The name of the bucket - -{{< highlight bash >}} -ozone sh bucket delete /hive/jan -{{< /highlight >}} - -The above command will delete _jan_ bucket if it is empty. - -### Info - -The bucket info commands returns the information about the bucket. -***Params:*** - -| Arguments | Comment | -|--------------------------------|-----------------------------------------| -| Uri | The name of the bucket. - -{{< highlight bash >}} -ozone sh bucket info /hive/jan -{{< /highlight >}} - -The above command will print out the information about _jan_ bucket. - -### List - -The bucket list command allows users to list the buckets in a volume. - -***Params:*** - -| Arguments | Comment | -|--------------------------------|-----------------------------------------| -| -l, --length | Maximum number of results to return. Default: 100 -| -p, --prefix | Optional, Only buckets that match this prefix will be returned. -| -s, --start | The listing will start from key after the start key. -| Uri | The name of the _volume_. - -{{< highlight bash >}} -ozone sh bucket list /hive -{{< /highlight >}} - -This command will list all buckets on the volume _hive_. - - - -### Update - -The bucket update command allows changing access permissions on bucket. - -***Params:*** - -| Arguments | Comment | -|--------------------------------|-----------------------------------------| -| --addAcl | Optional, Comma separated ACLs that will added to bucket. -| --removeAcl | Optional, Comma separated list of acl to remove. -| Uri | The name of the bucket. - -{{< highlight bash >}} -ozone sh bucket update --addAcl=user:bilbo:rw /hive/jan -{{< /highlight >}} - -The above command gives user bilbo read/write permission to the bucket. - -### path -The bucket command to provide ozone mapping for s3 bucket (Created via aws cli) - -{{< highlight bash >}} -ozone s3 path <> -{{< /highlight >}} - -The above command will print VolumeName and the mapping created for s3Bucket. - -You can try out these commands from the docker instance of the [Alpha -Cluster](runningviadocker.html). diff --git a/hadoop-hdds/docs/content/BuildingSources.md b/hadoop-hdds/docs/content/BuildingSources.md deleted file mode 100644 index bdc99e9fa9a1f..0000000000000 --- a/hadoop-hdds/docs/content/BuildingSources.md +++ /dev/null @@ -1,66 +0,0 @@ ---- -title: Building from Sources -weight: 1 -menu: - main: - parent: Starting - weight: 6 ---- - - -***This is a guide on how to build the ozone sources. If you are not -planning to build sources yourself, you can safely skip this page.*** - -If you are a Hadoop ninja, and wise in the ways of Apache, you already know -that a real Apache release is a source release. - -If you want to build from sources, Please untar the source tarball and run -the ozone build command. This instruction assumes that you have all the -dependencies to build Hadoop on your build machine. If you need instructions -on how to build Hadoop, please look at the Apache Hadoop Website. - -```bash -mvn -f pom.ozone.xml clean package -DskipTests=true -``` - -This will build an ozone-\.tar.gz in your `hadoop-ozone/dist/target` directory. - -You can copy this tarball and use this instead of binary artifacts that are -provided along with the official release. - -## How to test the build -You can run the acceptance tests in the hadoop-ozone directory to make sure -that your build is functional. To launch the acceptance tests, please follow - the instructions in the **README.md** in the `smoketest` directory. - -```bash -cd smoketest -./test.sh -``` - - You can also execute only a minimal subset of the tests: - -```bash -cd smoketest -./test.sh --env ozone basic -``` - -Acceptance tests - will start a small ozone cluster and verify that ozone shell and ozone file - system is fully functional. \ No newline at end of file diff --git a/hadoop-hdds/docs/content/CommandShell.md b/hadoop-hdds/docs/content/CommandShell.md deleted file mode 100644 index d95d1e6dc6d82..0000000000000 --- a/hadoop-hdds/docs/content/CommandShell.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -title: Ozone CLI -menu: - main: - parent: Client - weight: 1 - identifier: OzoneShell ---- - - -Ozone has a set of command line tools that can be used to manage ozone. - -All these commands are invoked via the ```ozone``` script. - -The commands supported by ozone are: - - * **classpath** - Prints the class path needed to get the hadoop jar and the - required libraries. - * **dtutil** - Operations related to delegation tokens - * **fs** - Runs a command on ozone file system. - * **datanode** - Via daemon command, the HDDS data nodes can be started or - stopped. - * **envvars** - Display computed Hadoop environment variables. - * **freon** - Runs the ozone load generator. - * **genesis** - Developer Only, Ozone micro-benchmark application. - * **getconf** - Reads ozone config values from configuration. - * **jmxget** - Get JMX exported values from NameNode or DataNode. - * **om** - Ozone Manager, via daemon command can be started or stopped. - * **sh** - Primary command line interface for ozone. - * **scm** - Storage Container Manager service, via daemon can be - stated or stopped. - * **scmcli** - Developer only, Command Line Interface for the Storage - Container Manager. - * **version** - Prints the version of Ozone and HDDS. - * **genconf** - Generate minimally required ozone configs and output to - ozone-site.xml. - -## Understanding Ozone command shell -The most used command when working with Ozone is the Ozone command shell. -Ozone command shell gives a command shell interface to work against -Ozone. - -The Ozone shell commands take the following format. - -> _ozone sh object action url_ - -**ozone** script is used to invoke all Ozone sub-commands. The ozone shell is -invoked via ```sh``` command. - -The object can be a volume, bucket or a key. The action is various verbs like - create, list, delete etc. - - -Ozone URL can point to a volume, bucket or keys in the following format: - -_\[scheme\]\[server:port\]/volume/bucket/key_ - - -Where, - -1. Scheme - This should be `o3` which is the native RPC protocol to access - Ozone API. The usage of the schema is optional. - -2. Server:Port - This is the address of the Ozone Manager. This can be server - only, in that case, the default port is used. If this value is omitted -then the defaults specified in the ozone-site.xml will be used for Ozone -Manager address. - -Depending on the call, the volume/bucket/key names will be part of the URL. -Please see volume commands, bucket commands, and key commands section for more -detail. - -## Invoking help - -Ozone shell help can be invoked at _object_ level or at _action_ level. -For example: - -{{< highlight bash >}} -ozone sh volume --help -{{< /highlight >}} - -This will show all possible actions for volumes. - -or it can be invoked to explain a specific action like -{{< highlight bash >}} -ozone sh volume create --help -{{< /highlight >}} -This command will give you command line options of the create command. diff --git a/hadoop-hdds/docs/content/Concepts.md b/hadoop-hdds/docs/content/Concepts.md deleted file mode 100644 index 4cbd7c400d390..0000000000000 --- a/hadoop-hdds/docs/content/Concepts.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -title: Architecture -date: "2017-10-10" -menu: main ---- - - - -Ozone is a redundant, distributed object store build by -leveraging primitives present in HDFS. The primary design point of ozone is scalability, and it aims to scale to billions of objects. - -Ozone consists of volumes, buckets, and keys. A volume is similar to a home directory in the ozone world. Only an administrator can create it. Volumes are used to store buckets. Once a volume is created users can create as many buckets as needed. Ozone stores data as keys which live inside these buckets. - -Ozone namespace is composed of many storage volumes. Storage volumes are also used as the basis for storage accounting. - -To access a key, an Ozone URL has the following format: - -``` -http://servername:port/volume/bucket/key -``` - -Where the server name is the name of a data node, the port is the data node HTTP port. The volume represents the name of the ozone volume; bucket is an ozone bucket created by the user and key represents the file. - -Please look at the [command line interface]({{< ref "CommandShell.md#shell" >}}) for more info. - -Ozone supports both (S3 compatible) REST and RPC protocols. Clients can choose either of these protocols to communicate with Ozone. Please see the [client documentation]({{< ref "JavaApi.md" >}}) for more details. - -Ozone separates namespace management and block space management; this helps -ozone to scale much better. The namespace is managed by a daemon called -[Ozone Manager ]({{< ref "OzoneManager.md" >}}) (OM), and block space is -managed by [Storage Container Manager] ({{< ref "Hdds.md" >}}) (SCM). - -The data nodes provide replication and ability to store blocks; these blocks are stored in groups to reduce the metadata pressure on SCM. This groups of blocks are called storage containers. Hence the block manager is called storage container -manager. - -Ozone Overview --------------- - -
The following diagram is a high-level overview of the core components of Ozone.

 - -![Architecture diagram](../../OzoneOverview.svg) - -The main elements of Ozone are
: - -### Ozone Manager
 - -[Ozone Manager]({{< ref "OzoneManager.md" >}}) (OM) takes care of the Ozone's namespace. -All ozone objects like volumes, buckets, and keys are managed by OM. In Short, OM is the metadata manager for Ozone. -OM talks to blockManager(SCM) to get blocks and passes it on to the Ozone -client. Ozone client writes data to these blocks. -OM will eventually be replicated via Apache Ratis for High Availability.
 - -### Storage Container Manager - -[Storage Container Manager]({{< ref "Hdds.md" >}}) (SCM) is the block and cluster manager for Ozone. -SCM along with data nodes offer a service called 'storage containers'. -A storage container is a group unrelated of blocks that are managed together as a single entity. - -SCM offers the following abstractions.

 - -![SCM Abstractions](../../SCMBlockDiagram.png) - -### Blocks -Blocks are similar to blocks in HDFS. They are replicated store of data. Client writes data to blocks. - -### Containers -A collection of blocks replicated and managed together. - -### Pipelines -SCM allows each storage container to choose its method of replication. -For example, a storage container might decide that it needs only one copy of a block -and might choose a stand-alone pipeline. Another storage container might want to have a very high level of reliability and pick a RATIS based pipeline. In other words, SCM allows different kinds of replication strategies to co-exist. The client while writing data, chooses a storage container with required properties. - -### Pools -A group of data nodes is called a pool. For scaling purposes, -we define a pool as a set of machines. This makes management of data nodes easier. - -### Nodes -The data node where data is stored. SCM monitors these nodes via heartbeat. - -### Clients -Ozone ships with a set of clients. Ozone [CLI]({{< ref "CommandShell.md#shell" >}}) is the command line interface like 'hdfs' command.
 [Freon] ({{< ref "Freon.md" >}}) is a load generation tool for Ozone.
 - -## S3 gateway - -Ozone provides and [S3 compatible REST gateway server]({{< ref "S3.md">}}). All of the main S3 features are supported and any S3 compatible client library can be used. - -### Ozone File System -[Ozone file system]({{< ref "OzoneFS.md">}}) is a Hadoop compatible file system. This allows Hadoop services and applications like Hive and Spark to run against -Ozone without any change. (For example: you can use `hdfs dfs -ls o3fs://...` instead of `hdfs dfs -ls hdfs://...`) - -### Ozone Client -This is similar to DFSClient in HDFS. This is the standard client to talk to Ozone. All other components that we have discussed so far rely on Ozone client. Ozone client supports RPC protocol. diff --git a/hadoop-hdds/docs/content/Dozone.md b/hadoop-hdds/docs/content/Dozone.md deleted file mode 100644 index 559134b6e29ee..0000000000000 --- a/hadoop-hdds/docs/content/Dozone.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: "Dozone & Dev Tools" -date: 2017-08-10 -menu: - main: - parent: Tools ---- - - - -Dozone stands for docker for ozone. Ozone supports docker to make it easy to develop and test ozone. Starting a docker-based ozone container is simple. - -In the `compose/ozone` directory there are two files that define the docker and ozone settings. - -Developers can - -{{< highlight bash >}} -cd compose/ozone -{{< /highlight >}} - -and simply run - -{{< highlight bash >}} -docker-compose up -d -{{< /highlight >}} - -to run a ozone cluster on docker. - -This command will launch OM, SCM and a data node. - -To access the OM UI, one can view http://localhost:9874. - -_Please note_: dozone does not map the data node ports to the 9864. Instead, it maps to the ephemeral port range. So many examples in the command shell will not work if you run those commands from the host machine. To find out where the data node port is listening, you can run the `docker ps` command or always ssh into a container before running ozone commands. - -To shutdown a running docker-based ozone cluster, please run - -{{< highlight bash >}} -docker-compose down -{{< /highlight >}} - - -Adding more config settings ---------------------------- -The file called `docker-config` contains all ozone specific config settings. This file is processed to create the ozone-site.xml. - -Useful Docker & Ozone Commands ------------------------------- - -If you make any modifications to ozone, the simplest way to test it is to run freon and unit tests. - -Here are the instructions to run freon in a docker-based cluster. - -{{< highlight bash >}} -docker-compose exec datanode bash -{{< /highlight >}} - -This will open a bash shell on the data node container. -Now we can execute freon for load generation. - -{{< highlight bash >}} -ozone freon randomkeys --numOfVolumes=10 --numOfBuckets 10 --numOfKeys 10 -{{< /highlight >}} - -Here is a set of helpful commands for working with docker for ozone. -To check the status of the components: - -{{< highlight bash >}} -docker-compose ps -{{< /highlight >}} - -To get logs from a specific node/service: - -{{< highlight bash >}} -docker-compose logs scm -{{< /highlight >}} - - -As the WebUI ports are forwarded to the external machine, you can check the web UI: - -* For the Storage Container Manager: http://localhost:9876 -* For the Ozone Manager: http://localhost:9874 -* For the Datanode: check the port with `docker ps` (as there could be multiple data nodes, ports are mapped to the ephemeral port range) - -You can start multiple data nodes with: - -{{< highlight bash >}} -docker-compose scale datanode=3 -{{< /highlight >}} - -You can test the commands from the [Ozone CLI]({{< ref "CommandShell.md#shell" >}}) after opening a new bash shell in one of the containers: - -{{< highlight bash >}} -docker-compose exec datanode bash -{{< /highlight >}} diff --git a/hadoop-hdds/docs/content/Freon.md b/hadoop-hdds/docs/content/Freon.md deleted file mode 100644 index 6ef0280717e66..0000000000000 --- a/hadoop-hdds/docs/content/Freon.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: Freon -date: "2017-09-02T23:58:17-07:00" -menu: - main: - parent: Tools ---- - - -Overview --------- - -Freon is a load-generator for Ozone. This tool is used for testing the functionality of ozone. - -### Random keys - -In randomkeys mode, the data written into ozone cluster is randomly generated. -Each key will be of size 10 KB. - -The number of volumes/buckets/keys can be configured. The replication type and -factor (eg. replicate with ratis to 3 nodes) Also can be configured. - -For more information use - -`bin/ozone freon --help` - -### Example - -{{< highlight bash >}} -ozone freon randomkeys --numOfVolumes=10 --numOfBuckets 10 --numOfKeys 10 --replicationType=RATIS --factor=THREE -{{< /highlight >}} - -{{< highlight bash >}} -*************************************************** -Status: Success -Git Base Revision: 48aae081e5afacbb3240657556b26c29e61830c3 -Number of Volumes created: 10 -Number of Buckets created: 100 -Number of Keys added: 1000 -Ratis replication factor: THREE -Ratis replication type: RATIS -Average Time spent in volume creation: 00:00:00,035 -Average Time spent in bucket creation: 00:00:00,319 -Average Time spent in key creation: 00:00:03,659 -Average Time spent in key write: 00:00:10,894 -Total bytes written: 10240000 -Total Execution time: 00:00:16,898 -*********************** -{{< /highlight >}} diff --git a/hadoop-hdds/docs/content/Hdds.md b/hadoop-hdds/docs/content/Hdds.md deleted file mode 100644 index 35654cdf4d4b1..0000000000000 --- a/hadoop-hdds/docs/content/Hdds.md +++ /dev/null @@ -1,65 +0,0 @@ ---- -title: "Hadoop Distributed Data Store" -date: "2017-09-14" -menu: - main: - parent: Architecture -weight: 10 ---- - - -SCM Overview ------------- - -Storage Container Manager or SCM is a very important component of ozone. SCM -offers block and container-based services to Ozone Manager. A container is a -collection of unrelated blocks under ozone. SCM and data nodes work together -to maintain the replication levels needed by the cluster. - -It is easier to look at a putKey operation to understand the role that SCM plays. - -To put a key, a client makes a call to OM with the following arguments. - --- putKey(keyName, data, pipeline type, replication count) - -1. keyName - refers to the file name. -2. data - The data that the client wants to write. -3. pipeline type - Allows the client to select the pipeline type. A pipeline - refers to the replication strategy used for replicating a block. Ozone - currently supports Stand Alone and Ratis as two different pipeline types. -4. replication count - This specifies how many copies of the block replica should be maintained. - -In most cases, the client does not specify the pipeline type and replication - count. The default pipeline type and replication count are used. - - -Ozone Manager when it receives the putKey call, makes a call to SCM asking -for a pipeline instance with the specified property. So if the client asked -for RATIS replication strategy and a replication count of three, then OM -requests SCM to return a set of data nodes that meet this capability. - -If SCM can find this a pipeline ( that is a set of data nodes) that can meet -the requirement from the client, then those nodes are returned to OM. OM will -persist this info and return a tuple consisting of {BlockID, ContainerName, and Pipeline}. - -If SCM is not able to find a pipeline, then SCM creates a logical pipeline and then returns it. - - -SCM manages blocks, containers, and pipelines. To return healthy pipelines, -SCM also needs to understand the node health. So SCM listens to heartbeats -from data nodes and acts as the node manager too. diff --git a/hadoop-hdds/docs/content/OzoneFS.md b/hadoop-hdds/docs/content/OzoneFS.md deleted file mode 100644 index 7438aa233a365..0000000000000 --- a/hadoop-hdds/docs/content/OzoneFS.md +++ /dev/null @@ -1,127 +0,0 @@ ---- -title: Ozone File System -weight: 1 -date: 2017-09-14 -menu: main -menu: - main: - parent: Starting - weight: 4 ---- - - -There are many Hadoop compatible files systems under Hadoop. Hadoop compatible file systems ensures that storage backends like Ozone can easily be integrated into Hadoop eco-system. - -## Setting up the Ozone file system - -To create an ozone file system, we have to choose a bucket where the file system would live. This bucket will be used as the backend store for OzoneFileSystem. All the files and directories will be stored as keys in this bucket. - -Please run the following commands to create a volume and bucket, if you don't have them already. - -{{< highlight bash >}} -ozone sh volume create /volume -ozone sh bucket create /volume/bucket -{{< /highlight >}} - -Once this is created, please make sure that bucket exists via the listVolume or listBucket commands. - -Please add the following entry to the core-site.xml. - -{{< highlight xml >}} - - fs.o3fs.impl - org.apache.hadoop.fs.ozone.OzoneFileSystem - - - fs.defaultFS - o3fs://bucket.volume - -{{< /highlight >}} - -This will make this bucket to be the default file system for HDFS dfs commands and register the o3fs file system type. - -You also need to add the ozone-filesystem.jar file to the classpath: - -{{< highlight bash >}} -export HADOOP_CLASSPATH=/opt/ozone/share/ozonefs/lib/hadoop-ozone-filesystem-lib-current*.jar:$HADOOP_CLASSPATH -{{< /highlight >}} - -Once the default Filesystem has been setup, users can run commands like ls, put, mkdir, etc. -For example, - -{{< highlight bash >}} -hdfs dfs -ls / -{{< /highlight >}} - -or - -{{< highlight bash >}} -hdfs dfs -mkdir /users -{{< /highlight >}} - - -Or put command etc. In other words, all programs like Hive, Spark, and Distcp will work against this file system. -Please note that any keys created/deleted in the bucket using methods apart from OzoneFileSystem will show up as directories and files in the Ozone File System. - -Note: Bucket and volume names are not allowed to have a period in them. -Moreover, the filesystem URI can take a fully qualified form with the OM host and port as a part of the path following the volume name. -For example, - -{{< highlight bash>}} -hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:5678/key -{{< /highlight >}} - - -## Supporting older Hadoop version (Legacy jar, BasicOzoneFilesystem) - -There are two ozonefs files, both of them include all the dependencies: - - * share/ozone/lib/hadoop-ozone-filesystem-lib-current-VERSION.jar - * share/ozone/lib/hadoop-ozone-filesystem-lib-legacy-VERSION.jar - -The first one contains all the required dependency to use ozonefs with a - compatible hadoop version (hadoop 3.2). - -The second one contains all the dependency in an internal, separated directory, - and a special class loader is used to load all the classes from the location. - -With this method the hadoop-ozone-filesystem-lib-legacy.jar can be used from - any older hadoop version (eg. hadoop 3.1, hadoop 2.7 or spark+hadoop 2.7) - -Similar to the dependency jar, there are two OzoneFileSystem implementation. - -For hadoop 3.0 and newer, you can use `org.apache.hadoop.fs.ozone.OzoneFileSystem` - which is a full implementation of the Hadoop compatible File System API. - -For Hadoop 2.x you should use the Basic version: `org.apache.hadoop.fs.ozone.BasicOzoneFileSystem`. - -This is the same implementation but doesn't include the features/dependencies which are added with - Hadoop 3.0. (eg. FS statistics, encryption zones). - -### Summary - -The following table summarize which jar files and implementation should be used: - -Hadoop version | Required jar | OzoneFileSystem implementation ----------------|-------------------------|---------------------------------------------------- -3.2 | filesystem-lib-current | org.apache.hadoop.fs.ozone.OzoneFileSystem -3.1 | filesystem-lib-legacy | org.apache.hadoop.fs.ozone.OzoneFileSystem -2.9 | filesystem-lib-legacy | org.apache.hadoop.fs.ozone.BasicOzoneFileSystem -2.7 | filesystem-lib-legacy | org.apache.hadoop.fs.ozone.BasicOzoneFileSystem - With this method the hadoop-ozone-filesystem-lib-legacy.jar can be used from - any older hadoop version (eg. hadoop 2.7 or spark+hadoop 2.7) diff --git a/hadoop-hdds/docs/content/OzoneManager.md b/hadoop-hdds/docs/content/OzoneManager.md deleted file mode 100644 index 5eb8663b04aa0..0000000000000 --- a/hadoop-hdds/docs/content/OzoneManager.md +++ /dev/null @@ -1,77 +0,0 @@ ---- -title: "Ozone Manager" -date: "2017-09-14" -menu: - main: - parent: Architecture -weight: 11 ---- - - -OM Overview -------------- - -Ozone Manager or OM is the namespace manager for Ozone. The clients (RPC clients, Rest proxy, Ozone file system, etc.) communicate with OM to create and delete various ozone objects. - -Each ozone volume is the root of a namespace under OM. This is very different from HDFS which provides a single rooted file system. - -Ozone's namespace is a collection of volumes or is a forest instead of a -single rooted tree as in HDFS. This property makes it easy to deploy multiple - OMs for scaling, this feature is under development. - -OM Metadata ------------------ - -Conceptually, OM maintains a list of volumes, buckets, and keys. For each user, it maintains a list of volumes. For each volume, the list of buckets and for each bucket the list of keys. - -Right now, OM is a single instance service. Ozone already relies on Apache Ratis (A Replicated State Machine based on Raft protocol). OM will be extended to replicate all its metadata via Ratis. With that, OM will be highly available. - -OM UI ------------- - -OM supports a simple UI for the time being. The default port of OM is 9874. To access the OM UI, the user can connect to http://OM:port or for a concrete example, -``` -http://omserver:9874/ -``` -OM UI primarily tries to measure load and latency of OM. The first section of OM UI relates to the number of operations seen by the cluster broken down by the object, operation and whether the operation was successful. - -The latter part of the UI is focused on latency and number of operations that OM is performing. - -One of the hardest problems in HDFS world is discovering the numerous settings offered to tune HDFS. Ozone solves that problem by tagging the configs. To discover settings, click on "Common Tools"->Config. This will take you to the ozone config UI. - -Config UI ------------- - -The ozone config UI is a matrix with row representing the tags, and columns representing All, OM and SCM. - -Suppose a user wanted to discover the required settings for ozone. Then the user can tick the checkbox that says "Required." -This will filter out all "Required" settings along with the description of what each setting does. - -The user can combine different checkboxes and UI will combine the results. That is, If you have more than one row selected, then all keys for those chosen tags are displayed together. - -We are hopeful that this leads to a more straightforward way of discovering settings that manage ozone. - - -OM and SCM -------------------- -[Storage container manager]({{< ref "Hdds.md" >}}) or (SCM) is the block manager - for ozone. When a client requests OM for a set of data nodes to write data, OM talks to SCM and gets a block. - -A block returned by SCM contains a pipeline, which is a set of nodes that we participate in that block replication. - -So OM is dependent on SCM for reading and writing of Keys. However, OM is independent of SCM while doing metadata operations like ozone volume or bucket operations. diff --git a/hadoop-hdds/docs/content/OzoneSecurityArchitecture.md b/hadoop-hdds/docs/content/OzoneSecurityArchitecture.md deleted file mode 100644 index e5a81f79c80b0..0000000000000 --- a/hadoop-hdds/docs/content/OzoneSecurityArchitecture.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -title: "Ozone Security Overview" -date: "2019-April-03" -menu: - main: - parent: Architecture -weight: 11 ---- - - -# Security in Ozone # -Starting with badlands release (ozone-0.4.0-alpha) ozone cluster can be secured against external threats. Specifically it can be configured for following security features: - -1. Authentication -2. Authorization -3. Audit -4. Transparent Data Encryption (TDE) - -## Authentication ## - -### Kerberos ### -Similar to hadoop, Ozone allows kerberos-based authentication. So one way to setup identities for all the daemons and clients is to create kerberos keytabs and configure it like any other service in hadoop. - -### Tokens ### -Tokens are widely used in Hadoop to achieve lightweight authentication without compromising on security. Main motivation for using tokens inside Ozone is to prevent the unauthorized access while keeping the protocol lightweight and without sharing secret over the wire. Ozone utilizes three types of token: - -#### Delegation token #### - -Once client establishes their identity via kerberos they can request a delegation token from OzoneManager. This token can be used by a client to prove its identity until the token expires. Like Hadoop delegation tokens, an Ozone delegation token has 3 important fields: - - 1. **Renewer**: User responsible for renewing the token. - 2. **Issue date**: Time at which token was issued. - 3. **Max date**: Time after which token can’t be renewed. - -Token operations like get, renew and cancel can only be performed over an Kerberos authenticated connection. Clients can use delegation token to establish connection with OzoneManager and perform any file system/object store related operations like, listing the objects in a bucket or creating a volume etc. - -#### Block Tokens #### - -Block tokens are similar to delegation tokens in sense that they are signed by OzoneManager. Block tokens are created by OM (OzoneManager) when a client request involves interaction with DataNodes such as read/write Ozone keys. - -Unlike delegation tokens there is no client API to request block tokens. Instead, they are handed transparently to client along with key/block locations. Block tokens are validated by Datanodes when receiving read/write requests from clients. Block token can't be renewed explicitly by client. Client with expired block token will need to refetch the key/block locations to get new block tokens. - -#### S3Token #### - -Like block tokens S3Tokens are handled transparently for clients. It is signed by S3secret created by client. S3Gateway creates this token for every s3 client request. To create an S3Token user must have a S3 secret. - -### Certificates ### -Apart from kerberos and tokens Ozone utilizes certificate based authentication for Ozone service components. To enable this, SCM (StorageContainerManager) bootstraps itself as an Certificate Authority when security is enabled. This allows all daemons inside Ozone to have an SCM signed certificate. Below is brief descriptions of steps involved: - - 1. Datanodes and OzoneManagers submits a CSR (certificate signing request) to SCM. - 2. SCM verifies identity of DN (Datanode) or OM via Kerberos and generates a certificate. - 3. This certificate is used by OM and DN to prove their identities. - 4. Datanodes use OzoneManager certificate to validate block tokens. This is possible because both of them trust SCM signed certificates. (i.e OzoneManager and Datanodes) - -## Authorization ## -Ozone provides a pluggable API to control authorization of all client related operations. Default implementation allows every request. Clearly it is not meant for production environments. To configure a more fine grained policy one may configure Ranger plugin for Ozone. Since it is a pluggable module clients can also implement their own custom authorization policy and configure it using `ozone.acl.authorizer.class`. - -## Audit ## - -Ozone provides ability to audit all read & write operations to OM, SCM and Datanodes. Ozone audit leverages the Marker feature which enables user to selectively audit only READ or WRITE operations by a simple config change without restarting the service(s). - -To enable/disable audit of READ operations, set filter.read.onMatch to NEUTRAL or DENY respectively. Similarly, the audit of WRITE operations can be controlled using filter.write.onMatch. - -Generating audit logs is only half the job, so Ozone also provides AuditParser - a sqllite based command line utility to parse/query audit logs with predefined templates(ex. Top 5 commands) and options for custom query. Once the log file has been loaded to AuditParser, one can simply run a template as shown below: -ozone auditparser template top5cmds - -Similarly, users can also execute custom query using: - -```bash -ozone auditparser query "select * from audit where level=='FATAL'" -``` - -## Transparent Data Encryption ## - -Ozone TDE setup process and usage are very similar to HDFS TDE. The major difference is that Ozone TDE is enabled at Ozone bucket level when a bucket is created. - -To create an encrypted bucket, client need to - -* Create a bucket encryption key with hadoop key CLI (same as you do for HDFS encryption zone key) - -```bash -hadoop key create key1 -``` - -* Create an encrypted bucket with -k option - -```bash -ozone sh bucket create -k key1 /vol1/ez1 -``` - -After that the usage will be transparent to the client and end users, i.e., all data written to encrypted bucket are encrypted at datanodes. - -To know more about how to setup a secure Ozone cluster refer to [How to setup secure Ozone cluster]({{< ref "SetupSecureOzone.md" >}}) - -Ozone [security architecture document](https://issues.apache.org/jira/secure/attachment/12911638/HadoopStorageLayerSecurity.pdf) can be referred for a deeper dive into Ozone Security architecture. \ No newline at end of file diff --git a/hadoop-hdds/docs/content/Prometheus.md b/hadoop-hdds/docs/content/Prometheus.md deleted file mode 100644 index 1d19f0c7da6bb..0000000000000 --- a/hadoop-hdds/docs/content/Prometheus.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: Monitoring with Prometheus -menu: - main: - parent: Recipes ---- - - -[Prometheus](https://prometheus.io/) is an open-source monitoring server developed under under the [Cloud Native Computing Foundation](https://www.cncf.io/). - -Ozone supports Prometheus out of the box. The servers start a prometheus -compatible metrics endpoint where all the available hadoop metrics are published in prometheus exporter format. - -## Prerequisites - - 1. [Install the and start]({{< ref "RunningViaDocker.md" >}}) an Ozone cluster. - 2. [Download](https://prometheus.io/download/#prometheus) the prometheus binary. - -## Monitoring with prometheus - -(1) To enable the Prometheus metrics endpoint you need to add a new configuration to the `ozone-site.xml` file: - -``` - - hdds.prometheus.endpoint.enabled - true - -``` - -_Note_: for Docker compose based pseudo cluster put the `OZONE-SITE.XML_hdds.prometheus.endpoint.enabled=true` line to the `docker-config` file. - -(2) Restart the Ozone Manager and Storage Container Manager and check the prometheus endpoints: - - * http://scm:9874/prom - - * http://ozoneManager:9876/prom - -(3) Create a prometheus.yaml configuration with the previous endpoints: - -```yaml -global: - scrape_interval: 15s - -scrape_configs: - - job_name: ozone - metrics_path: /prom - static_configs: - - targets: - - "scm:9876" - - "ozoneManager:9874" -``` - -(4) Start with prometheus from the directory where you have the prometheus.yaml file: - -``` -prometheus -``` - -(5) Check the active targets in the prometheus web-ui: - -http://localhost:9090/targets - -![Prometheus target page example](prometheus.png) - - -(6) Check any metrics on the prometheus web ui. For example: - -http://localhost:9090/graph?g0.range_input=1h&g0.expr=om_metrics_num_key_allocate&g0.tab=1 - -![Prometheus target page example](prometheus-key-allocate.png) - -## Note - -The ozone distribution contains a ready-to-use, dockerized environment to try out ozone and prometheus. It can be found under `compose/ozoneperf` directory. - -```bash -cd compose/ozoneperf -docker-compose up -d -``` \ No newline at end of file diff --git a/hadoop-hdds/docs/content/RealCluster.md b/hadoop-hdds/docs/content/RealCluster.md deleted file mode 100644 index 78dd46ee031b7..0000000000000 --- a/hadoop-hdds/docs/content/RealCluster.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -title: Starting an Ozone Cluster -weight: 1 -menu: - main: - parent: Starting - weight: 3 ---- - - -Before we boot up the Ozone cluster, we need to initialize both SCM and Ozone Manager. - -{{< highlight bash >}} -ozone scm --init -{{< /highlight >}} -This allows SCM to create the cluster Identity and initialize its state. -The ```init``` command is similar to Namenode format. Init command is executed only once, that allows SCM to create all the required on-disk structures to work correctly. -{{< highlight bash >}} -ozone --daemon start scm -{{< /highlight >}} - -Once we know SCM is up and running, we can create an Object Store for our use. This is done by running the following command. - -{{< highlight bash >}} -ozone om --init -{{< /highlight >}} - - -Once Ozone manager has created the Object Store, we are ready to run the name -services. - -{{< highlight bash >}} -ozone --daemon start om -{{< /highlight >}} - -At this point Ozone's name services, the Ozone manager, and the block service SCM is both running. -**Please note**: If SCM is not running -```om --init``` command will fail. SCM start will fail if on-disk data structures are missing. So please make sure you have done both ```scm --init``` and ```om --init``` commands. - -Now we need to start the data nodes. Please run the following command on each datanode. -{{< highlight bash >}} -ozone --daemon start datanode -{{< /highlight >}} - -At this point SCM, Ozone Manager and data nodes are up and running. - -***Congratulations!, You have set up a functional ozone cluster.*** - -------- -If you want to make your life simpler, you can just run -{{< highlight bash >}} -ozone scm --init -ozone om --init -start-ozone.sh -{{< /highlight >}} -This assumes that you have set up the slaves file correctly and ssh -configuration that allows ssh-ing to all data nodes. This is the same as the -HDFS configuration, so please refer to HDFS documentation on how to set this -up. diff --git a/hadoop-hdds/docs/content/Settings.md b/hadoop-hdds/docs/content/Settings.md deleted file mode 100644 index 32a5884daa587..0000000000000 --- a/hadoop-hdds/docs/content/Settings.md +++ /dev/null @@ -1,142 +0,0 @@ ---- -title: Configuration -weight: 1 -menu: - main: - parent: Starting - weight: 2 ---- - - - - - -If you are feeling adventurous, you can setup ozone in a real cluster. -Setting up a real cluster requires us to understand the components of Ozone. -Ozone is designed to work concurrently with HDFS. However, Ozone is also -capable of running independently. The components of ozone are the same in both approaches. - -## Ozone Components - -1. Ozone Manager - Is the server that is in charge of the namespace of Ozone. Ozone Manager is responsible for all volume, bucket and key operations. -2. Storage Container Manager - Acts as the block manager. Ozone Manager -requests blocks from SCM, to which clients can write data. -3. Datanodes - Ozone data node code runs inside the HDFS datanode or in the independent deployment case runs an ozone datanode daemon. - - - - -## Setting up an Ozone only cluster - -* Please untar the ozone- to the directory where you are going -to run Ozone from. We need Ozone jars on all machines in the cluster. So you -need to do this on all machines in the cluster. - -* Ozone relies on a configuration file called ```ozone-site.xml```. To -generate a template that you can replace with proper values, please run the -following command. This will generate a template called ```ozone-site.xml``` at -the specified path (directory). - -{{< highlight bash >}} -ozone genconf -{{< /highlight >}} - -Let us look at the settings inside the generated file (ozone-site.xml) and -how they control ozone. Once the right values are defined, this file -needs to be copied to ```ozone directory/etc/hadoop```. - - -* **ozone.enabled** This is the most critical setting for ozone. -Ozone is a work in progress and users have to enable this service explicitly. -By default, Ozone is disabled. Setting this flag to `true` enables ozone in the -HDFS or Ozone cluster. - -Here is an example, - -{{< highlight xml >}} - - ozone.enabled - true - -{{< /highlight >}} - -* **ozone.metadata.dirs** Allows Administrators to specify where the - metadata must reside. Usually you pick your fastest disk (SSD if - you have them on your nodes). OzoneManager, SCM and datanode will write the - metadata to this path. This is a required setting, if this is missing Ozone - will fail to come up. - - Here is an example, - -{{< highlight xml >}} - - ozone.metadata.dirs - /data/disk1/meta - -{{< /highlight >}} - -* **ozone.scm.names** Storage container manager(SCM) is a distributed block - service which is used by ozone. This property allows data nodes to discover - SCM's address. Data nodes send heartbeat to SCM. - Until HA feature is complete, we configure ozone.scm.names to be a - single machine. - - Here is an example, - - {{< highlight xml >}} - - ozone.scm.names - scm.hadoop.apache.org - - {{< /highlight >}} - - * **ozone.scm.datanode.id.dir** Data nodes generate a Unique ID called Datanode - ID. This identity is written to the file datanode.id in a directory specified by this path. *Data nodes - will create this path if it doesn't exist already.* - -Here is an example, -{{< highlight xml >}} - - ozone.scm.datanode.id.dir - /data/disk1/meta/node/datanode.id - -{{< /highlight >}} - -* **ozone.om.address** OM server address. This is used by OzoneClient and -Ozone File System. - -Here is an example, -{{< highlight xml >}} - - ozone.om.address - ozonemanager.hadoop.apache.org - -{{< /highlight >}} - - -### Ozone Settings Summary - -| Setting | Value | Comment | -|--------------------------------|------------------------------|------------------------------------------------------------------| -| ozone.enabled | true | This enables SCM and containers in HDFS cluster. | -| ozone.metadata.dirs | file path | The metadata will be stored here. | -| ozone.scm.names | SCM server name | Hostname:port or IP:port address of SCM. | -| ozone.scm.block.client.address | SCM server name and port | Used by services like OM | -| ozone.scm.client.address | SCM server name and port | Used by client-side | -| ozone.scm.datanode.address | SCM server name and port | Used by datanode to talk to SCM | -| ozone.om.address | OM server name | Used by Ozone handler and Ozone file system. | diff --git a/hadoop-hdds/docs/content/SetupSecureOzone.md b/hadoop-hdds/docs/content/SetupSecureOzone.md deleted file mode 100644 index e67c1e76c0d36..0000000000000 --- a/hadoop-hdds/docs/content/SetupSecureOzone.md +++ /dev/null @@ -1,98 +0,0 @@ ---- -title: "Setup secure ozone cluster" -date: "2019-April-03" -menu: - main: - parent: Architecture -weight: 11 ---- - -# Setup secure ozone cluster # -To enable security in ozone cluster **ozone.security.enabled** should be set to true. - -Property|Value -----------------------|------ -ozone.security.enabled| true - -## Kerberos ## -Configuration for service daemons: - -Property|Description ---------|------------------------------------------------------------ -hdds.scm.kerberos.principal | The SCM service principal. Ex scm/_HOST@REALM.COM_ -hdds.scm.kerberos.keytab.file |The keytab file used by SCM daemon to login as its service principal. -ozone.om.kerberos.principal |The OzoneManager service principal. Ex om/_HOST@REALM.COM -ozone.om.kerberos.keytab.file |The keytab file used by SCM daemon to login as its service principal. -hdds.scm.http.kerberos.principal|SCM http server service principal. -hdds.scm.http.kerberos.keytab.file|The keytab file used by SCM http server to login as its service principal. -ozone.om.http.kerberos.principal|OzoneManager http server principal. -ozone.om.http.kerberos.keytab.file|The keytab file used by OM http server to login as its service principal. -ozone.s3g.keytab.file |The keytab file used by S3 gateway. Ex /etc/security/keytabs/HTTP.keytab -ozone.s3g.authentication.kerberos.principal|S3 Gateway principal. Ex HTTP/_HOST@EXAMPLE.COM -## Tokens ## - -## Delegation token ## - -Delegation tokens are enabled by default when security is enabled. - -## Block Tokens ## - -Property|Value ------------------------------|------ -hdds.block.token.enabled | true - -## S3Token ## - -S3 token are enabled by default when security is enabled. -To use S3 tokens users need to perform following steps: - -* S3 clients should get the secret access id and user secret from OzoneManager. - -``` -ozone s3 getsecret -``` - -* Setup secret in aws configs: - -``` -aws configure set default.s3.signature_version s3v4 -aws configure set aws_access_key_id ${accessId} -aws configure set aws_secret_access_key ${secret} -aws configure set region us-west-1 -``` - -## Certificates ## - -Certificates are used internally inside Ozone. Its enabled be default when security is enabled. - -## Authorization ## - -Default access authorizer for Ozone approves every request. It is not suitable for production environments. It is recommended that clients use ranger plugin for Ozone to manage authorizations. - -Property|Value ---------|------------------------------------------------------------ -ozone.acl.enabled | true -ozone.acl.authorizer.class| org.apache.ranger.authorization.ozone.authorizer.RangerOzoneAuthorizer - -## TDE ## - -To use TDE clients must set KMS URI. - -Property|Value ------------------------------------|----------------------------------------- -hadoop.security.key.provider.path | KMS uri. Ex kms://http@kms-host:9600/kms diff --git a/hadoop-hdds/docs/content/VolumeCommands.md b/hadoop-hdds/docs/content/VolumeCommands.md deleted file mode 100644 index 4cdb212e7d84d..0000000000000 --- a/hadoop-hdds/docs/content/VolumeCommands.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -title: Volume Commands -menu: - main: - parent: OzoneShell - weight: 1 ---- - - -Volume commands generally need administrator privileges. The ozone shell supports the following volume commands. - - * [create](#create) - * [delete](#delete) - * [info](#info) - * [list](#list) - * [update](#update) - -### Create - -The volume create command allows an administrator to create a volume and -assign it to a user. - -***Params:*** - -| Arguments | Comment | -|--------------------------------|-----------------------------------------| -| -q, --quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. | -| -u, --user | Required, The name of the user who owns this volume. This user can create, buckets and keys on this volume. | -| Uri | The name of the volume. | - -{{< highlight bash >}} -ozone sh volume create --quota=1TB --user=bilbo /hive -{{< /highlight >}} - -The above command will create a volume called _hive_ on the ozone cluster. This -volume has a quota of 1TB, and the owner is _bilbo_. - -### Delete - -The volume delete command allows an administrator to delete a volume. If the -volume is not empty then this command will fail. - -***Params:*** - -| Arguments | Comment | -|--------------------------------|-----------------------------------------| -| Uri | The name of the volume. - -{{< highlight bash >}} -ozone sh volume delete /hive -{{< /highlight >}} - -The above command will delete the volume hive, if the volume has no buckets -inside it. - -### Info - -The volume info commands returns the information about the volume including -quota and owner information. -***Params:*** - -| Arguments | Comment | -|--------------------------------|-----------------------------------------| -| Uri | The name of the volume. - -{{< highlight bash >}} -ozone sh volume info /hive -{{< /highlight >}} - -The above command will print out the information about hive volume. - -### List - -The volume list command will list the volumes owned by a user. - -{{< highlight bash >}} -ozone sh volume list --user hadoop -{{< /highlight >}} - -The above command will print out all the volumes owned by the user hadoop. - -### Update - -The volume update command allows changing of owner and quota on a given volume. - -***Params:*** - -| Arguments | Comment | -|--------------------------------|-----------------------------------------| -| -q, --quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. | -| -u, --user | Optional, The name of the user who owns this volume. This user can create, buckets and keys on this volume. | -| Uri | The name of the volume. | - -{{< highlight bash >}} -ozone sh volume update --quota=10TB /hive -{{< /highlight >}} - -The above command updates the volume quota to 10TB. - -You can try out these commands from the docker instance of the [Alpha -Cluster](runningviadocker.html). diff --git a/hadoop-hdds/docs/content/_index.md b/hadoop-hdds/docs/content/_index.md index 4c5c1ab128644..bb1bf9a744e78 100644 --- a/hadoop-hdds/docs/content/_index.md +++ b/hadoop-hdds/docs/content/_index.md @@ -1,5 +1,5 @@ --- -title: Ozone Overview +title: Overview menu: main weight: -10 --- @@ -22,18 +22,24 @@ weight: -10 # Apache Hadoop Ozone -Ozone is a scalable, distributed object store for Hadoop. Applications like -Apache Spark, Hive and YARN, can run against Ozone without any -modifications. Ozone comes with a [Java client library]({{< ref "JavaApi.md" ->}}), a [S3]({{< ref "S3.md" >}}) and a [command line interface] -({{< ref "CommandShell.md#shell" >}}) which makes it easy to use Ozone. + -Ozone consists of volumes, buckets, and Keys: +*_Ozone is a scalable, redundant, and distributed object store for Hadoop.

    +Apart from scaling to billions of objects of varying sizes, +Ozone can function effectively in containerized environments +like Kubernetes._*

    -* Volumes are similar to user accounts. Only administrators can create or delete volumes. -* Buckets are similar to directories. A bucket can contain any number of keys, but buckets cannot contain other buckets. -* Keys are similar to files. A bucket can contain any number of keys. +Applications like Apache Spark, Hive and YARN, work without any modifications when using Ozone. Ozone comes with a [Java client library]({{< +ref "JavaApi.md" +>}}), [S3 protocol support] ({{< ref "S3.md" >}}), and a [command line interface] +({{< ref "shell/_index.md" >}}) which makes it easy to use Ozone. +Ozone consists of volumes, buckets, and keys: +* Volumes are similar to user accounts. Only administrators can create or delete volumes. +* Buckets are similar to directories. A bucket can contain any number of keys, but buckets cannot contain other buckets. +* Keys are similar to files. -}}"> + }}"> + diff --git a/hadoop-hdds/docs/content/beyond/Containers.md b/hadoop-hdds/docs/content/beyond/Containers.md new file mode 100644 index 0000000000000..ea7e3b17c4377 --- /dev/null +++ b/hadoop-hdds/docs/content/beyond/Containers.md @@ -0,0 +1,235 @@ +--- +title: "Ozone Containers" +summary: Ozone uses containers extensively for testing. This page documents the usage and best practices of Ozone. +weight: 2 +--- + + +Docker heavily is used at the ozone development with three principal use-cases: + +* __dev__: + * We use docker to start local pseudo-clusters (docker provides unified environment, but no image creation is required) +* __test__: + * We create docker images from the dev branches to test ozone in kubernetes and other container orchestrator system + * We provide _apache/ozone_ images for each release to make it easier for evaluation of Ozone. + These images are __not__ created __for production__ usage. + +

    + +* __production__: + * We have documentation on how you can create your own docker image for your production cluster. + +Let's check out each of the use-cases in more detail: + +## Development + +Ozone artifact contains example docker-compose directories to make it easier to start Ozone cluster in your local machine. + +From distribution: + +```bash +cd compose/ozone +docker-compose up -d +``` + +After a local build: + +```bash +cd hadoop-ozone/dist/target/ozone-*/compose +docker-compose up -d +``` + +These environments are very important tools to start different type of Ozone clusters at any time. + +To be sure that the compose files are up-to-date, we also provide acceptance test suites which start +the cluster and check the basic behaviour. + +The acceptance tests are part of the distribution, and you can find the test definitions in `smoketest` directory. + +You can start the tests from any compose directory: + +For example: + +```bash +cd compose/ozone +./test.sh +``` + +### Implementation details + +`compose` tests are based on the apache/hadoop-runner docker image. The image itself does not contain +any Ozone jar file or binary just the helper scripts to start ozone. + +hadoop-runner provdes a fixed environment to run Ozone everywhere, but the ozone distribution itself +is mounted from the including directory: + +(Example docker-compose fragment) + +``` + scm: + image: apache/hadoop-runner:jdk11 + volumes: + - ../..:/opt/hadoop + ports: + - 9876:9876 + +``` + +The containers are configured based on environment variables, but because the same environment +variables should be set for each containers we maintain the list of the environment variables +in a separated file: + +``` + scm: + image: apache/hadoop-runner:jdk11 + #... + env_file: + - ./docker-config +``` + +The docker-config file contains the list of the required environment variables: + +``` +OZONE-SITE.XML_ozone.om.address=om +OZONE-SITE.XML_ozone.om.http-address=om:9874 +OZONE-SITE.XML_ozone.scm.names=scm +OZONE-SITE.XML_ozone.enabled=True +#... +``` + +As you can see we use naming convention. Based on the name of the environment variable, the +appropriate hadoop config XML (`ozone-site.xml` in our case) will be generated by a +[script](https://github.com/apache/hadoop/tree/docker-hadoop-runner-latest/scripts) which is +included in the `hadoop-runner` base image. + +The [entrypoint](https://github.com/apache/hadoop/blob/docker-hadoop-runner-latest/scripts/starter.sh) +of the `hadoop-runner` image contains a helper shell script which triggers this transformation and +can do additional actions (eg. initialize scm/om storage, download required keytabs, etc.) +based on environment variables. + +## Test/Staging + +The `docker-compose` based approach is recommended only for local test, not for multi node cluster. +To use containers on a multi-node cluster we need a Container Orchestrator like Kubernetes. + +Kubernetes example files are included in the `kubernetes` folder. + +*Please note*: all the provided images are based the `hadoop-runner` image which contains all the +required tool for testing in staging environments. For production we recommend to create your own, +hardened image with your own base image. + +### Test the release + +The release can be tested with deploying any of the example clusters: + +```bash +cd kubernetes/examples/ozone +kubectl apply -f +``` + +Plese note that in this case the latest released container will be downloaded from the dockerhub. + +### Test the development build + +To test a development build you can create your own image and upload it to your own docker registry: + + +```bash +mvn clean install -f pom.ozone.xml -DskipTests -Pdocker-build,docker-push -Ddocker.image=myregistry:9000/name/ozone +``` + +The configured image will be used in all the generated kubernetes resources files (`image:` keys are adjusted during the build) + +```bash +cd kubernetes/examples/ozone +kubectl apply -f +``` + +## Production + + + +You can use the source of our development images as an example: + + * [Base image] (https://github.com/apache/hadoop/blob/docker-hadoop-runner-jdk11/Dockerfile) + * [Docker image] (https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/dist/src/main/docker/Dockerfile) + + Most of the elements are optional and just helper function but to use the provided example + kubernetes resources you may need the scripts from + [here](https://github.com/apache/hadoop/tree/docker-hadoop-runner-jdk11/scripts) + + * The two python scripts convert environment variables to real hadoop XML config files + * The start.sh executes the python scripts (and other initialization) based on environment variables. + +## Containers + +Ozone related container images and source locations: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    #ContainerRepositoryBaseBranchTagsComments
    1apache/ozonehttps://github.com/apache/hadoop-docker-ozoneozone-... hadoop-runner0.3.0,0.4.0,0.4.1For each Ozone release we create new release tag.
    2apache/hadoop-runner https://github.com/apache/hadoopdocker-hadoop-runnercentosjdk11,jdk8,latestThis is the base image used for testing Hadoop Ozone. + This is a set of utilities that make it easy for us run ozone.
    diff --git a/hadoop-hdds/docs/content/beyond/DockerCheatSheet.md b/hadoop-hdds/docs/content/beyond/DockerCheatSheet.md new file mode 100644 index 0000000000000..f4f5492cf177a --- /dev/null +++ b/hadoop-hdds/docs/content/beyond/DockerCheatSheet.md @@ -0,0 +1,88 @@ +--- +title: "Docker Cheat Sheet" +date: 2017-08-10 +summary: Docker Compose cheat sheet to help you remember the common commands to control an Ozone cluster running on top of Docker. +weight: 4 +--- + + + +In the `compose` directory of the ozone distribution there are multiple pseudo-cluster setup which +can be used to run Ozone in different way (for example: secure cluster, with tracing enabled, +with prometheus etc.). + +If the usage is not document in a specific directory the default usage is the following: + +```bash +cd compose/ozone +docker-compose up -d +``` + +The data of the container is ephemeral and deleted together with the docker volumes. +```bash +docker-compose down +``` + +## Useful Docker & Ozone Commands + +If you make any modifications to ozone, the simplest way to test it is to run freon and unit tests. + +Here are the instructions to run freon in a docker-based cluster. + +{{< highlight bash >}} +docker-compose exec datanode bash +{{< /highlight >}} + +This will open a bash shell on the data node container. +Now we can execute freon for load generation. + +{{< highlight bash >}} +ozone freon randomkeys --numOfVolumes=10 --numOfBuckets 10 --numOfKeys 10 +{{< /highlight >}} + +Here is a set of helpful commands for working with docker for ozone. +To check the status of the components: + +{{< highlight bash >}} +docker-compose ps +{{< /highlight >}} + +To get logs from a specific node/service: + +{{< highlight bash >}} +docker-compose logs scm +{{< /highlight >}} + + +As the WebUI ports are forwarded to the external machine, you can check the web UI: + +* For the Storage Container Manager: http://localhost:9876 +* For the Ozone Manager: http://localhost:9874 +* For the Datanode: check the port with `docker ps` (as there could be multiple data nodes, ports are mapped to the ephemeral port range) + +You can start multiple data nodes with: + +{{< highlight bash >}} +docker-compose scale datanode=3 +{{< /highlight >}} + +You can test the commands from the [Ozone CLI]({{< ref "shell/_index.md" >}}) after opening a new bash shell in one of the containers: + +{{< highlight bash >}} +docker-compose exec datanode bash +{{< /highlight >}} diff --git a/hadoop-hdds/docs/content/RunningWithHDFS.md b/hadoop-hdds/docs/content/beyond/RunningWithHDFS.md similarity index 81% rename from hadoop-hdds/docs/content/RunningWithHDFS.md rename to hadoop-hdds/docs/content/beyond/RunningWithHDFS.md index 1d88cc547dc66..154be5332bf4f 100644 --- a/hadoop-hdds/docs/content/RunningWithHDFS.md +++ b/hadoop-hdds/docs/content/beyond/RunningWithHDFS.md @@ -1,10 +1,8 @@ --- title: Running concurrently with HDFS +linktitle: Runing with HDFS weight: 1 -menu: - main: - parent: Starting - weight: 5 +summary: Ozone is designed to run concurrently with HDFS. This page explains how to deploy Ozone in a exisiting HDFS cluster. --- + +{{}} + Beyond Basics pages go into custom configurations of Ozone, including how + to run Ozone concurrently with an existing HDFS cluster. These pages also + take deep into how to run profilers and leverage tracing support built into + Ozone. +{{}} diff --git a/hadoop-hdds/docs/content/concept/ContainerMetadata.png b/hadoop-hdds/docs/content/concept/ContainerMetadata.png new file mode 100644 index 0000000000000..48bd1c43c0361 Binary files /dev/null and b/hadoop-hdds/docs/content/concept/ContainerMetadata.png differ diff --git a/hadoop-hdds/docs/content/concept/Datanodes.md b/hadoop-hdds/docs/content/concept/Datanodes.md new file mode 100644 index 0000000000000..ea63fe46b146e --- /dev/null +++ b/hadoop-hdds/docs/content/concept/Datanodes.md @@ -0,0 +1,75 @@ +--- +title: "Datanodes" +date: "2017-09-14" +weight: 4 +summary: Ozone supports Amazon's Simple Storage Service (S3) protocol. In fact, You can use S3 clients and S3 SDK based applications without any modifications with Ozone. +--- + + +Datanodes are the worker bees of Ozone. All data is stored on data nodes. +Clients write data in terms of blocks. Datanode aggregates these blocks into +a storage container. A storage container is the data streams and metadata +about the blocks written by the clients. + +## Storage Containers + +![FunctionalOzone](ContainerMetadata.png) + +A storage container is a self-contained super block. It has a list of Ozone +blocks that reside inside it, as well as on-disk files which contain the +actual data streams. This is the default Storage container format. From +Ozone's perspective, container is a protocol spec, actual storage layouts +does not matter. In other words, it is trivial to extend or bring new +container layouts. Hence this should be treated as a reference implementation +of containers under Ozone. + +## Understanding Ozone Blocks and Containers + +When a client wants to read a key from Ozone, the client sends the name of +the key to the Ozone Manager. Ozone manager returns the list of Ozone blocks +that make up that key. + +An Ozone block contains the container ID and a local ID. The figure below +shows the logical layout out of Ozone block. + +![OzoneBlock](OzoneBlock.png) + +The container ID lets the clients discover the location of the container. The +authoritative information about where a container is located is with the +Storage Container Manager (SCM). In most cases, the container location will be +cached by Ozone Manager and will be returned along with the Ozone blocks. + + +Once the client is able to locate the contianer, that is, understand which +data nodes contain this container, the client will connect to the datanode +and read the data stream specified by _Container ID:Local ID_. In other +words, the local ID serves as index into the container which describes what +data stream we want to read from. + +### Discovering the Container Locations + +How does SCM know where the containers are located ? This is very similar to +what HDFS does; the data nodes regularly send container reports like block +reports. Container reports are far more concise than block reports. For +example, an Ozone deployment with a 196 TB data node will have around 40 +thousand containers. Compare that with HDFS block count of million and half +blocks that get reported. That is a 40x reduction in the block reports. + +This extra indirection helps tremendously with scaling Ozone. SCM has far +less block data to process and the name node is a different service are +critical to scaling Ozone. diff --git a/hadoop-hdds/docs/content/concept/FunctionalOzone.png b/hadoop-hdds/docs/content/concept/FunctionalOzone.png new file mode 100644 index 0000000000000..0bc75b5e1fdbb Binary files /dev/null and b/hadoop-hdds/docs/content/concept/FunctionalOzone.png differ diff --git a/hadoop-hdds/docs/content/concept/Hdds.md b/hadoop-hdds/docs/content/concept/Hdds.md new file mode 100644 index 0000000000000..ad17b54d01116 --- /dev/null +++ b/hadoop-hdds/docs/content/concept/Hdds.md @@ -0,0 +1,52 @@ +--- +title: "Storage Container Manager" +date: "2017-09-14" +weight: 3 +summary: Storage Container Manager or SCM is the core metadata service of Ozone. SCM provides a distributed block layer for Ozone. +--- + + +Storage container manager provides multiple critical functions for the Ozone +cluster. SCM acts as the cluster manager, Certificate authority, Block +manager and the Replica manager. + +{{}} +SCM is in charge of creating an Ozone cluster. When an SCM is booted up via init command, SCM creates the cluster identity and root certificates needed for the SCM certificate authority. SCM manages the life cycle of a data node in the cluster. +{{}} + +{{}} +SCM's Ceritificate authority is in +charge of issuing identity certificates for each and every +service in the cluster. This certificate infrastructre makes +it easy to enable mTLS at network layer and also the block +token infrastructure depends on this certificate infrastructure. +{{}} + +{{}} +SCM is the block manager. SCM +allocates blocks and assigns them to data nodes. Clients +read and write these blocks directly. +{{}} + + +{{}} +SCM keeps track of all the block +replicas. If there is a loss of data node or a disk, SCM +detects it and instructs data nodes make copies of the +missing blocks to ensure high avialablity. +{{}} diff --git a/hadoop-hdds/docs/content/concept/Overview.md b/hadoop-hdds/docs/content/concept/Overview.md new file mode 100644 index 0000000000000..9e5746d84617d --- /dev/null +++ b/hadoop-hdds/docs/content/concept/Overview.md @@ -0,0 +1,81 @@ +--- +title: Overview +date: "2017-10-10" +weight: 1 +summary: Ozone's overview and components that make up Ozone. +--- + + + +Ozone is a redundant, distributed object store optimized for Big data +workloads. The primary design point of ozone is scalability, and it aims to +scale to billions of objects. + +Ozone separates namespace management and block space management; this helps +ozone to scale much better. The namespace is managed by a daemon called +[Ozone Manager ]({{< ref "OzoneManager.md" >}}) (OM), and block space is +managed by [Storage Container Manager] ({{< ref "Hdds.md" >}}) (SCM). + + +Ozone consists of volumes, buckets, and keys. +A volume is similar to a home directory in the ozone world. +Only an administrator can create it. + +Volumes are used to store buckets. +Once a volume is created users can create as many buckets as needed. +Ozone stores data as keys which live inside these buckets. + +Ozone namespace is composed of many storage volumes. +Storage volumes are also used as the basis for storage accounting. + +The block diagram shows the core components of Ozone. + +![Architecture diagram](ozoneBlockDiagram.png) + +The Ozone Manager is the name space manager, Storage Container Manager +manages the physical and data layer and Recon is the management interface for +Ozone. + + +## Different Perspectives + +![FunctionalOzone](FunctionalOzone.png) + +Any distributed system can be viewed from different perspectives. One way to +look at Ozone is to imagine it as Ozone Manager as a name space service built on + top of HDDS, a distributed block store. + +Another way to visualize Ozone is to look at the functional layers; we have a + metadata data management layer, composed of Ozone Manager and Storage + Container Manager. + +We have a data storage layer, which is basically the data nodes and they are + managed by SCM. + +The replication layer, provided by Ratis is used to replicate metadata (OM and SCM) +and also used for consistency when data is modified at the +data nodes. + +We have a management server called Recon, that talks to all other components +of Ozone and provides a unified management API and UX for Ozone. + +We have a protocol bus that allows Ozone to be extended via other +protocols. We currently only have S3 protocol support built via Protocol bus. +Protocol Bus provides a generic notion that you can implement new file system + or object store protocols that call into O3 Native protocol. + diff --git a/hadoop-hdds/docs/content/concept/OzoneBlock.png b/hadoop-hdds/docs/content/concept/OzoneBlock.png new file mode 100644 index 0000000000000..9583bd5ee78f1 Binary files /dev/null and b/hadoop-hdds/docs/content/concept/OzoneBlock.png differ diff --git a/hadoop-hdds/docs/content/concept/OzoneManager.md b/hadoop-hdds/docs/content/concept/OzoneManager.md new file mode 100644 index 0000000000000..1ebdd4951d200 --- /dev/null +++ b/hadoop-hdds/docs/content/concept/OzoneManager.md @@ -0,0 +1,87 @@ +--- +title: "Ozone Manager" +date: "2017-09-14" +weight: 2 +summary: Ozone Manager is the principal name space service of Ozone. OM manages the life cycle of volumes, buckets and Keys. +--- + + +Ozone Manager (OM) is the namespace manager for Ozone. + +This means that when you want to write some data, you ask Ozone +Manager for a block and Ozone Manager gives you a block and remembers that +information. When you want to read that file back, you need to find the +address of the block and Ozone Manager returns it you. + +Ozone Manager also allows users to organize keys under a volume and bucket. +Volumes and buckets are part of the namespace and managed by Ozone Manager. + +Each ozone volume is the root of an independent namespace under OM. +This is very different from HDFS which provides a single rooted file system. + +Ozone's namespace is a collection of volumes or is a forest instead of a +single rooted tree as in HDFS. This property makes it easy to deploy multiple +OMs for scaling. + +## Ozone Manager Metadata + +OM maintains a list of volumes, buckets, and keys. +For each user, it maintains a list of volumes. +For each volume, the list of buckets and for each bucket the list of keys. + +Ozone Manager will use Apache Ratis(A Raft protocol implementation) to +replicate Ozone Manager state. This will ensure High Availability for Ozone. + + +## Ozone Manager and Storage Container Manager + +The relationship between Ozone Manager and Storage Container Manager is best +understood if we trace what happens during a key write and key read. + +### Key Write + +* To write a key to Ozone, a client tells Ozone manager that it would like to +write a key into a bucket that lives inside a specific volume. Once Ozone +Manager determines that you are allowed to write a key to the specified bucket, +OM needs to allocate a block for the client to write data. + +* To allocate a block, Ozone Manager sends a request to Storage Container +Manager (SCM); SCM is the manager of data nodes. SCM picks three data nodes +into which client can write data. SCM allocates the block and returns the +block ID to Ozone Manager. + +* Ozone manager records this block information in its metadata and returns the +block and a block token (a security permission to write data to the block) +to the client. + +* The client uses the block token to prove that it is allowed to write data to +the block and writes data to the data node. + +* Once the write is complete on the data node, the client will update the block +information on +Ozone manager. + + +### Key Reads + +* Key reads are simpler, the client requests the block list from the Ozone +Manager +* Ozone manager will return the block list and block tokens which +allows the client to read the data from data nodes. +* Client connects to the data node and presents the block token and reads +the data from the data node. diff --git a/hadoop-hdds/docs/content/concept/_index.md b/hadoop-hdds/docs/content/concept/_index.md new file mode 100644 index 0000000000000..8f0aeb07c965c --- /dev/null +++ b/hadoop-hdds/docs/content/concept/_index.md @@ -0,0 +1,33 @@ +--- +title: Concepts +date: "2017-10-10" +menu: main +weight: 6 + +--- + + + +{{}} + +Ozone's architectural elements are explained in the following pages. The +metadata layer, data layer, protocol bus, replication layer and Recon are +discussed in the following pages. These concepts are useful if you want to +understand how ozone works in depth. + +{{}} diff --git a/hadoop-hdds/docs/content/concept/ozoneBlockDiagram.png b/hadoop-hdds/docs/content/concept/ozoneBlockDiagram.png new file mode 100644 index 0000000000000..7fb738f68a9eb Binary files /dev/null and b/hadoop-hdds/docs/content/concept/ozoneBlockDiagram.png differ diff --git a/hadoop-hdds/docs/content/design/decommissioning.md b/hadoop-hdds/docs/content/design/decommissioning.md new file mode 100644 index 0000000000000..8d620be515ed3 --- /dev/null +++ b/hadoop-hdds/docs/content/design/decommissioning.md @@ -0,0 +1,624 @@ + + +--- +title: Decommissioning in Ozone +summary: Formal process to shut down machines in a safe way after the required replications. +date: 2019-07-31 +jira: HDDS-1881 +status: current +author: Anu Engineer, Marton Elek, Stephen O'Donnell +--- + +# Abstract + +The goal of decommissioning is to turn off a selected set of machines without data loss. It may or may not require to move the existing replicas of the containers to other nodes. + +There are two main classes of the decommissioning: + + * __Maintenance mode__: where the node is expected to be back after a while. It may not require replication of containers if enough replicas are available from other nodes (as we expect to have the current replicas after the restart.) + + * __Decommissioning__: where the node won't be started again. All the data should be replicated according to the current replication rules. + +Goals: + + * Decommissioning can be canceled any time + * The progress of the decommissioning should be trackable + * The nodes under decommissioning / maintenance mode should not been used for new pipelines / containers + * The state of the datanodes should be persisted / replicated by the SCM (in HDFS the decommissioning info exclude/include lists are replicated manually by the admin). If datanode is marked for decommissioning this state be available after SCM and/or Datanode restarts. + * We need to support validations before decommissioing (but the violations can be ignored by the admin). + * The administrator should be notified when a node can be turned off. + * The maintenance mode can be time constrained: if the node marked for maintenance for 1 week and the node is not up after one week, the containers should be considered as lost (DEAD node) and should be replicated. + +# Introduction + +Ozone is a highly available file system that relies on commodity hardware. In other words, Ozone is designed to handle failures of these nodes all the time. + +The Storage Container Manager(SCM) is designed to monitor the node health and replicate blocks and containers as needed. + +At times, Operators of the cluster can help the SCM by giving it hints. When removing a datanode, the operator can provide a hint. That is, a planned failure of the node is coming up, and SCM can make sure it reaches a safe state to handle this planned failure. + +Some times, this failure is transient; that is, the operator is taking down this node temporarily. In that case, we can live with lower replica counts by being optimistic. + +Both of these operations, __Maintenance__, and __Decommissioning__ are similar from the Replication point of view. In both cases, and the user instructs us on how to handle an upcoming failure. + +Today, SCM (*Replication Manager* component inside SCM) understands only one form of failure handling. This paper extends Replica Manager failure modes to allow users to request which failure handling model to be adopted(Optimistic or Pessimistic). + +Based on physical realities, there are two responses to any perceived failure, to heal the system by taking corrective actions or ignore the failure since the actions in the future will heal the system automatically. + +## User Experiences (Decommissioning vs Maintenance mode) + +From the user's point of view, there are two kinds of planned failures that the user would like to communicate to Ozone. + +The first kind is when a 'real' failure is going to happen in the future. This 'real' failure is the act of decommissioning. We denote this as "decommission" throughout this paper. The response that the user wants is SCM/Ozone to make replicas to deal with the planned failure. + +The second kind is when the failure is 'transient.' The user knows that this failure is temporary and cluster in most cases can safely ignore this issue. However, if the transient failures are going to cause a failure of availability; then the user would like the Ozone to take appropriate actions to address it. An example of this case, is if the user put 3 data nodes into maintenance mode and switched them off. + +The transient failure can violate the availability guarantees of Ozone; Since the user is telling us not to take corrective actions. Many times, the user does not understand the impact on availability while asking Ozone to ignore the failure. + +So this paper proposes the following definitions for Decommission and Maintenance of data nodes. + +__Decommission__ *of a data node is deemed to be complete when SCM/Ozone completes the replica of all containers on decommissioned data node to other data nodes.That is, the expected count matches the healthy count of containers in the cluster*. + +__Maintenance mode__ of a data node is complete if Ozone can guarantee at *least one copy of every container* is available in other healthy data nodes. + +## Examples + +Here are some illustrative examples: + +1. Let us say we have a container, which has only one copy and resides on Machine A. If the user wants to put machine A into maintenance mode; Ozone will make a replica before entering the maintenance mode. + +2. Suppose a container has two copies, and the user wants to put Machine A to maintenance mode. In this case; the Ozone understands that availability of the container is not affected and hence can decide to forgo replication. + +3. Suppose a container has two copies, and the user wants to put Machine A into maintenance mode. However, the user wants to put the machine into maintenance mode for one month. As the period of maintenance mode increases, the probability of data loss increases; hence, Ozone might choose to make a replica of the container even if we are entering maintenance mode. + +4. The semantics of decommissioning means that as long as we can find copies of containers in other machines, we can technically get away with calling decommission complete. Hence this clarification node; in the ordinary course of action; each decommission will create a replication flow for each container we have; however, it is possible to complete a decommission of a data node, even if we get a failure of the data node being decommissioned. As long as we can find the other datanodes to replicate from and get the number of replicas needed backup to expected count we are good. + +5. Let us say we have a copy of a container replica on Machine A, B, and C. It is possible to decommission all three machines at the same time, as decommissioning is just a status indicator of the data node and until we finish the decommissioning process. + + +The user-visible features for both of these are very similar: + +Both Decommission and Maintenance mode can be canceled any time before the operation is marked as completed by SCM. + +Decommissioned nodes, if and when added back, shall be treated as new data nodes; if they have blocks or containers on them, they can be used to reconstruct data. + + +## Maintenance mode in HDFS + +HDFS supports decommissioning and maintenance mode similar to Ozone. This is a quick description of the HDFS approach. + +The usage of HDFS maintenance mode: + + * First, you set a minimum replica count on the cluster, which can be zero, but defaults to 1. + * Then you can set a number of nodes into maintenance, with an expiry time or have them remain in maintenance forever, until they are manually removed. Nodes are put into maintenance in much the same way as nodes are decommissioned. + * When a set of nodes go into maintenance, all blocks hosted on them are scanned and if the node going into maintenance would cause the number of replicas to fall below the minimum replica count, the relevant nodes go into a decommissioning like state while new replicas are made for the blocks. + * Once the node goes into maintenance, it can be stopped etc and HDFS will not be concerned about the under-replicated state of the blocks. + * When the expiry time passes, the node is put back to normal state (if it is online and heartbeating) or marked as dead, at which time new replicas will start to be made. + +This is very similar to decommissioning, and the code to track maintenance mode and ensure the blocks are replicated etc, is effectively the same code as with decommissioning. The one area that differs is probably in the replication monitor as it must understand that the node is expected to be offline. + +The ideal way to use maintenance mode, is when you know there are a set of nodes you can stop without having to do any replications. In HDFS, the rack awareness states that all blocks should be on two racks, so that means a rack can be put into maintenance safely. + +There is another feature in HDFS called "upgrade Domain" which allows each datanode to be assigned a group. By default there should be at least 3 groups (domains) and then each of the 3 replicas will be stored on different group, allowing one full group to be put into maintenance at once. That is not yet supported in CDH, but is something we are targeting for CDPD I believe. + +One other difference with maintenance mode and decommissioning, is that you must have some sort of monitor thread checking for when maintenance is scheduled to end. HDFS solves this by having a class called the DatanodeAdminManager, and it tracks all nodes transitioning state, the under-replicated block count on them etc. + + +# Implementation + + +## Datanode state machine + +`NodeStateManager` maintains the state of the connected datanodes. The possible states: + + state | description + ---------------------|------------ + HEALTHY | The node is up and running. + STALE | Some heartbeats were missing for an already missing nodes. + DEAD | The stale node has not been recovered. + ENTERING_MAINTENANCE | The in-progress state, scheduling is disabled but the node can't not been turned off due to in-progress replication. + IN_MAINTENANCE | Node can be turned off but we expecteed to get it back and have all the replicas. + DECOMMISSIONING | The in-progress state, scheduling is disabled, all the containers should be replicated to other nodes. + DECOMMISSIONED | The node can be turned off, all the containers are replicated to other machine + + + +## High level algorithm + +The Algorithm is pretty simple from the Decommission or Maintenance point of view; + + 1. Mark a data node as DECOMMISSIONING or ENTERING_MAINTENANCE. This implies that node is NOT healthy anymore; we assume the use of a single flag and law of excluded middle. + + 2. Pipelines should be shut down and wait for confirmation that all pipelines are shutdown. So no new I/O or container creation can happen on a Datanode that is part of decomm/maint. + + 3. Once the Node has been marked as DECOMMISSIONING or ENTERING_MAINTENANCE; the Node will generate a list of containers that need replication. This list is generated by the Replica Count decisions for each container; the Replica Count will be computed by Replica Manager; + + 4. Replica Manager will check the stop condition for each node. The following should be true for all the containers to go from DECOMMISSIONING to DECOMMISSIONED or from ENTERING_MAINTENANCE to IN_MAINTENANCE. + + * Container is closed. + * We have at least one HEALTHY copy at all times. + * For entering DECOMMISSIONED mode `maintenance + healthy` must equal to `expectedeCount` + + 5. We will update the node state to DECOMMISSIONED or IN_MAINTENANCE reached state. + +_Replica count_ is a calculated number which represents the number of _missing_ replicas. The number can be negative in case of an over-replicated container. + +## Calculation of the _Replica count_ (required replicas) + +### Counters / Variables + +We have 7 different datanode state, but some of them can be combined. At high level we can group the existing state to three groups: + + * healthy state (when the container is available) + * maintenance (including IN_MAINTENANCE and ENTERING_MAINTENANCE) + * all the others. + +To calculate the required steps (required replication + stop condition) we need counter about the first two categories. + +Each counters should be calculated per container bases. + + Node state | Variable (# of containers) | + --------------------------------------|---------------------------------| + HEALTHY | `healthy` | + STALE + DEAD + DECOMMISSIONED | | + DECOMMISSIONING | | + ENTERING_MAINTENANCE + IN_MAINTENANCE | `maintenance` | + + +### The current replication model + +The current replication model in SCM/Ozone is very simplistic. We compute the replication count or the number of replications that we need to do as: + +``` +Replica count = expectedCount - currentCount +``` + +In case the _Replica count_ is positive, it means that we need to make more replicas. If the number is negative, it means that we are over replicated and we need to remove some replicas of this container. If the Replica count for a container is zero; it means that we have the expected number of containers in the cluster. + +To support idempontent placement strategies we should substract the in-fligt replications from the result: If there are one in-flight replication process and two replicas we won't start a new replication command unless the original command is timed out. The timeout is configured with `hdds.scm.replication.event.timeout` and the default value is 10 minutes. + +More preciously the current algorithm is the following: + +```java +replicaCount = expectedCount - healthy; + +if (replicaCount - inflight copies + inflight deletes) > 0 { + // container is over replicated +}.else if (replicaCount - inflight copies + inflight deletes) <0 { + // handle under replicated containers +} +``` + +The handling of inflight copies and deletes are independent from the decommissioning problem therefore here we focus only on the core mode: + +``` +replicaCount = expectedCount - healthy; +``` + +### The proposed solution + +To support the notion that a user can provide hints to the replication model, we propose to add two variables to the current model. + +In the new model, we propose to break the `currentCount` into the two separate groups. That is _Healthy nodes_ and _Maintenance nodes_. The new model replaces the currentCount with these two separate counts. The following function captures the code that drives the logic of computing Replica counts in the new model. The later section discusses the input and output of this model very extensively. + +```java +/** + * Calculate the number of the missing replicas. + * + * @return the number of the missing replicas. + If it's less than zero, the container is over replicated. + */ +int getReplicationCount(int expectedCount, int healthy, int maintenance) { + + //for over replication, count only with the healthy replicas + if (expectedCount <= healthy) { + return expectedCount - healthy; + } + + replicaCount = expectedCount - (healthy + maintenance); + + //at least one HEALTHY replicas should be guaranteed! + if (replicaCount == 0 && healthy < 1) { + replicaCount ++; + } + + //over replication is already handled. Always return with + // positive as over replication is already handled + return Math.max(0, replicaCount); +} + +``` + +To understand the reasing behind the two special `if` condition check the examples below. + +We also need to specify two end condition when the DECOMMISSIONING node can be moved to the DECOMMISSIONED state or the ENTERING_MAINTENANCE mode can be moved to the IN_MAINTENANCE state. + +The following conditions should be true for all the containers and all the containers on the specific node should be closed. + +**From DECOMMISSIONING to DECOMMISSIONED**: + + * There are at least one healthy replica + * We have three replicas (both helthy and maintenance) + +Which means that our stop condition can be formalized as: + +``` +(healthy >= 1) && (healthy + maintenance >= 3) +``` + +Both the numbers can be configurable: + + * 1 is the minimum number of healthy replicas (`decommissioning.minimum.healthy-replicas`) + * 3 is the minimum number of existing replicas (`decommissioning.minimum.replicas`) + +For example `decommissioning.minimum.healthy-replicas` can be set to two if administrator would like to survive an additional node failure during the maintenance period. + +**From ENTERING_MAINTENANCE to IN_MAINTENANCE:** + + * There are at least one healthy replicas + +This is the weaker version of the previous condition: + +``` +(healthy >= 1) +``` + +### Examples (normal cases) + +In this section we show example use cases together with the output of the proposed algorithm + +#### All healthy + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + HEALTHY | HEALTHY | HEALTHY| 3 | 3 | 0 | 0 + +The container C1 exists on machines A, B , and C. All the container reports tell us that the container is healthy. Running the above algorithm, we get: + +`expected - healthy + maint. = 3 - (3 + 0) = 0` + +It means, _"we don’t need no replication"._ + +#### One failure + + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + HEALTHY | HEALTHY | DEAD | 3 | 2 | 0 | 1 + + +The machine C has failed, and as a result, the healthy count has gone down from `3` to `2`. This means that we need to start one replication flow. + +`ReplicaCount = expected - healthy + maint. = 3 - (2 + 0) = 1.` + +This means that the new model will handle failure cases just like the current model. + +#### One decommissioning + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + HEALTHY | HEALTHY | DECOMM | 3 | 2 | 0 | 1 + + +In this case, machine C is being decommissioned. Therefore the healthy count has gone down to `2` , and decommission count is `1`. Since the + +```ReplicaCount = expected - healthy + maint`. we have `1 = 3 - (2 + 0)```, + +this gives us the decommission count implicitly. The trick here is to realize that incrementing decommission automatically causes a decrement in the healthy count, which allows us not to have _decommission_ in the equation explicitly. + +**Stop condition**: Not that if this containers is the only one on node C, node C can be moved to the DECOMMISSIONED state. + +#### Failure + decommissioning + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + HEALTHY | DEAD | DECOMM | 3 | 1 | 0 | 2 + + +Here is a case where we have a failure of a data node and a decommission of another data node. In this case, the container C1 needs two replica flows to heal itself. The equation is the same and we get + +`ReplicaCount(2) = ExpectecCount(3) - healthy(1)` + +The maintenance is still zero so ignored in this equation. + +#### 1 failure + 2 decommissioning + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + HEALTHY | DECOMM | DECOMM | 3 | 0 | 0 | 3 + +In this case, we have one failed data node and two data nodes being decommissioned. We need to get three replica flows in the system. This is achieved by: + +``` +ReplicaCount(3) = ExpectedCount(3) - (healthy(0) + maintenance(0)) +``` + +#### Maintenance mode + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + HEALTHY | DEAD | MAINT | 3 | 2 | 1 | 0 + +This represents the normal maintenance mode, where a single machine is marked as in maintenance mode. This means the following: + +``` +ReplicaCount(0) = ExpectedCount(3) - (healthy(2) + maintenance(1) +``` + +There are no replica flows since the user has asked us to move a single node into maintenance mode, and asked us explicitly not to worry about the single missing node. + +**Stop condition**: Not that if this containers is the only one on node C, node C can be moved to the IN_MAINTENANCE state. + +#### Maintenance + decommissioning + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + HEALTHY | DECOMM | MAINT | 3 | 1 | 1 | 1 + +*This is a fascinating case*; We have one good node; one decommissioned node and one node in maintenance mode. The expected result is that the replica manager will launch one replication flow to compensate for the node that is being decommissioned, and we also expect that there will be no replication for the node in maintenance mode. + +``` +Replica Count (1) = expectedCount(3) - (healthy(1) + maintenance(1)) +``` +So as expected we have one replication flow in the system. + +**Stop condition**: Not that if this containers is the only one in the system: + + * node C can be moved to the IN_MAINTENANCE state + * node B can not be decommissioned (we need the three replicas first) + +#### Decommissioning all the replicas + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + DECOMM | DECOMM | DECOMM | 3 | 0 | 0 | 3 + +In this case, we deal with all the data nodes being decommissioned. The number of healthy replicas for this container is 0, and hence: + +``` +replicaCount (3) = expectedCount (3)- (healthy(0) + maintenance(0)). +``` + +This provides us with all 3 independent replica flows in the system. + +#### Decommissioning the one remaining replicas + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + DEAD | DEAD | DECOMM | 3 | 0 | 0 | 3 + +We have two failed nodes and one node in Decomm. It is the opposite of case Line 5, where we have one failed node and 2 nodes in Decomm. The expected results are the same, we get 3 flows. + +#### Total failure + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + DEAD | DEAD | DEAD | 3 | 0 | 0 | 3 + +This is really an error condition. We have lost all 3 data nodes. The Replica Manager will compute that we need to rebuild 3 replicas, but we might not have a source to rebuild from. + +### Last replica is on ENTERING_MAINTENANCE + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + DEAD | MAINT | DEAD | 3 | 0 | 1 | 2 + +Is also an interesting case; we have lost 2 data nodes; and one node is being marked as Maint. Since we have 2 failed nodes, we need 2 replica flows in the system. However, the maintenance mode cannot be entered, since we will lose lone replica if we do that. + +### All maintenance + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|--------|--------|---------|---------|---------- + MAINT | MAINT | MAINT | 3 | 0 | 3 | *1* + +This is also a very special case; this is the case where the user is telling us to ignore the peril for all 3 replicas being offline. This means that the system will not be able to get to that container and would lead to potential I/O errors. Ozone will strive to avoid that case; this means that Ozone will hit the “if condition” and discover that we our ReplicCount is 0; since the user asked for it; but we are also going to lose all Replicas. At this point of time, we make a conscious decision to replicate one copy instead of obeying the user command and get to the situation where I/O can fail. + +**This brings us back to the semantics of Maintenance mode in Ozone**. If going into maintenance mode will not lead to a potential I/O failure, we will enter into the maintenance mode; Otherwise, we will replicate and enter into the maintenance mode after the replication is done. This is just the core replication algorithm, not the complete Decommission or Maintenance mode algorithms, just how the replica manager would behave. Once we define the behavior of Replica Manager, rest of the algorithm is easy to construct. + +Note: this is the case why we need the seconf `if` in the model (numbers in the brackets shows the actual value): + +``` + replicaCount(0) = expectedCount(3) - ( healthy(0) + maintenance(0) ); + + + //at least one HEALTHY replicas should be guaranteed! + if (replicaCount(0) == 0 && healthy(0) < 1) { + replicaCount ++; + } +``` + +### Over replication + +For over-replicated containers Ozone prefers to keep the replicas on the healthy nodes. We delete containers only if we have enough replicas on *healthy* nodes. + +``` +int getReplicationCount(int expectedCount, int healthy, int maintenance) { + + //for over replication, count only with the healthy replicas + if (expectedCount <= healthy) { + return expectedCount - healthy; + } + + replicaCount = ... //calculate missing replicas + + //over replication is already handled. Always return with + // positive as over replication is already handled + return Math.max(0, replicaCount); +} +``` + +Please note that we always assume that the the in-flight deletion are applied and the container is already deleted. + +There is a very rare case where the in-flight deletion is timed out (and as a result replication manager would assume the container is not deleted) BUT in the mean-time the container finally deleted. It can be survivied with including the creation timestamp in the ContainerDeleteCommand. + +### Over replication examples + +#### 4 replicas + + + Node A | Node B | Node C | Node D | expctd | healthy | mainten | repCount + --------|---------|---------|---------|--------|---------|---------|---------- + HEALTHY | HEALTHY | HEALTHY | HEALTHY | 3 | 4 | 0 | 0 + +This is an easy case as we have too many replicas we can safely remove on. + +``` +if (expectedCount <= healthy) { + return expectedCount - healthy +} +``` + +#### over replicated with IN_MAINTENANCE + + + Node A | Node B | Node C | Node D | expctd | healthy | mainten | repCount + --------|---------|---------|---------|--------|---------|---------|---------- + HEALTHY | HEALTHY | HEALTHY | MAINT | 3 | 3 | 1 | 0 + + +In this case we will delete the forth replica only after node D is restored and healthy again. (expectedCount is not less than healthy). As the `expectedCount (3) <= healthy (3)` the replicaCount will be calculated as `0`. + +#### over replicated with IN_MAINTENANCE + + + Node A | Node B | Node C | Node D | expctd | healthy | mainten | repCount + --------|---------|---------|---------|--------|---------|---------|---------- + HEALTHY | HEALTHY | MAINT | MAINT | 3 | 2 | 2 | 0 + +Here we are not over-repliacated as we don't have any healthy nodes. We will calculate the under-replication number as defined in the previous section: + +``` +replicaCount(-1) = expectedCount(3) - ( healthy(2) + maintenance(2) ); +``` + +The main algorithm would return with `replicaCount = -1` but as we return `Math.max(0,replicaCount)` the real response will be 0. Waiting for healthy nodes. + +### Handling in-flight replications + +In-flight replication requests and deletes are handled by the Replica Manager and the problem is orthogonal to the replication problem, but this section shows that the proposed model is not conflicted with the existing approach. + +Let's say we have an under-replicated container and we already selected a new datanode to copy a new replica to that specific node. + + + Node A | Node B | Node C | expctd | healthy | mainten | repCount + --------|---------|---------|--------|---------|---------|---------- + HEALTHY | HEALTHY | (copy) | 3 | 2 | 0 | 1 + + +Here the Replication Manager detects that one replica is missing but the real copy shouldn't been requested as it's alread inprogress. ReplicaManager must not select a new datanode based on the ContainerPlacementPolicy implementation as the policy may or may not be idempotent. + +For example if the placement policy would select a datanode randomly with each loop we would select a new datanode to replicate to. + +To avoid such a situation Replica Manager maintains a list of the in-flight copies (in-memory) on the SCM side. In this list we have all the sent replication requests but they are removed after a given amount of time (10 minutes) by default. + +With calculating the in-flight copy as a possible replication the Replication Manger doesn't need to request new replication. + +When a datanode is marked to be decommissioned there could be any in-flight replication copy process in that time. + + * At datanode we should stop all of the in-flight copy (datanodes should be notified about the DECOMMISSIONING/IN_MAINTENANCE state) + * We never ask any non-healthy nodes to replicate containers. + * In SCM, we don't need to do any special action + * In `ReplicationManager` we already have a map about the inflight replications (`Map>`). + * During a normal replication the number of in-flight replications are counted as real replication (2 real replicas + 1 inflight replication = replica count 3). During this calculation we need to check the current state of the datanodes and ignore the inflight replication if they are assigned to a node which is in decommissioning state. (Or we should update the inflight map, in case of node state change) + +### In-flight examples + +#### Maintenance + inflight + + Node A | Node B | Node C | expctd | healthy | mainten | repCount | copy | + --------|---------|---------|--------|---------|---------|----------|------| + HEALTHY | MAINT | copying | 3 | 1 | 1 | 1 | 1 | + +Here one have one node ENTERING_MAINTENANCE state, and one replica is missing and already started to be replicated. We don't need to start a new copy and node B can be moved to the IN_MAINTENANCE mode. + +``` +Replica Count (1) = expectedCount(3) - (healthy(1) + maintenance(1)) +Containers to copy (0) = Replica Count (1) - inflight copies (1) +``` + +#### Maintenance + inflight + + Node A | Node B | Node C | expctd | healthy | mainten | repCount | copy | + --------|---------|---------|--------|---------|---------|----------|------| + DECOMM | copying | copying | 3 | 0 | 0 | 3 | 1 | + + +Node A can not be DECOMMISSIONED as we have no HEALTHY replicas at all. + + +## Statefulness + +SCM stores all the node state in-memory. After a restart on the SCM side the datanode state can be lost. + +**Ozone doesn't guarantee that decommissioning/maintenance mode state survives the SCM restarts!!!** + + * If SCM restarts DECOMMISSIONED nodes will not report any more container reports and the nodes won't be registered. + * ENTERING_MAINTENANCE and DECOMMISSIONING nodes will became HEALTHY again and the decommissioning CLI command should be repeated. + * IN_MAINTENANCE nodes will become DEAD and all the containers will be replicated. + + *Ozone assumes that the maintenance mode is used short-term and SCM is not restarted during this specific period.* + +*Reasoning*: + +Neither of the node state nor the container state are persisted in SCM side. The decommissioned state can be stored on the SCM side (or on the SCM side and the datanode side) which can provide better user experience (and may be implemented). + +But to support maintenance mode after restart all the container information is required to be persisted (which is a too big architectural change). + +To make a replication decision replication manager needs the number of healthy replicas (they are reported via heartbeats) AND the number of containers on the node which is in maintenance mode. The later one is not available if the SCM is restarted as the container map exists only in the memory and the node which is turned off can't report any more container reports. Therefore the information about the existing containers on the node which is in the maintenance mode **can't be available'**. + +## Throttling + +SCM should avoid to request too many replication to live enough network bandwidth for the requests. + +Replication Manager can easily throttle the replication requests based on `inflightReplication` map, but this problem is independent from the handling of the decommissioning / maintenance mode because it should be solved for any kind of replication not just for this. + +## User interface + +The decommissioning and maintenance mode can be administered with a CLI interface. + +Required feature: + + * Set the state of a datanode (to DECOMMISSIONING or ENTERING_MAINTENANCE) + * Undo the decommissioning process + * check the current progress: + * This can be a table with the nodes, status of the nodes, number of containers, containers under replication and containers which doesn't much the stop condition yet (required replications) + * All the commands can support topology related filters (eg. display the nodes only for a specific rack or show the status of the nodes of s specific rack) + * Check current replication information of one specific container (required to debug why the decommissioning is stuck) + +## Checks before the decommissioning + +Decommissioning is requested via a new RPC call with the help of a new CLI tool. The server should check the current state of the cluster and deny the decommissioning if it's not possible. Possible violations: + + * Not enough space to store the new replicas. + * Not enough node to create all kind of pipelines + + In case of any violation, the request will fail, but any of theses rules can be turned off with a next request and the decommissioning can be forced. + +## Maintain progress + +We need to show the progress of the decommissioning process per node and cluster-wide. We already have the information about the under replicated containers, but we don't know the numbers of the containers before decommissioning. + +Instead of saving the original number of the required replications before (which is very fragile) we don't provide an absolute progress just the numbers of the remaining replication: + + +``` + Node | Status | # containers | in-progress replications| required replication + ----------------|------------------------|--------------|--------------------------|------------------------ + Node A | ENTERING_MAINTENANCE | 2837 | 12 | 402 + Node B | HEALTHY | 1239 | 0 | 0 + Node C | IN_MAINTENANCE | 2348 | 0 | 0 +``` + +`# containers` means the total number of the containers on the specific datanodes. To get the original number of the planned copies we can save the original 'container-to-node' map in memory and show some progress and provide more information for the users. diff --git a/hadoop-hdds/docs/content/design/ozone-enhancement-proposals.md b/hadoop-hdds/docs/content/design/ozone-enhancement-proposals.md new file mode 100644 index 0000000000000..cc7569eb2cfa0 --- /dev/null +++ b/hadoop-hdds/docs/content/design/ozone-enhancement-proposals.md @@ -0,0 +1,197 @@ +--- +title: Ozone Enhancement Proposals +summary: Definition of the process to share new technical proposals with the Ozone community. +date: 2019-06-07 +jira: HDDS-1659 +status: accepted +author: Anu Enginner, Marton Elek +--- + + +## Problem statement + +Some of the biggers features requires well defined plans before the implementation. Until now it was managed by uploading PDF design docs to selected JIRA. There are multiple problems with the current practice. + + 1. There is no easy way to find existing up-to-date and outdated design docs. + 2. Design docs usually have better description of the problem that the user docs + 3. We need better tools to discuss the design docs in the development phase of the doc + +We propose to follow the same process what we have now, but instead of uploading a PDF to the JIRA, create a PR to merge the proposal document to the documentation project. + +## Non-goals + + * Modify the existing workflow or approval process + * Migrate existing documents + * Make it harder to create design docs (it should be easy to support the creation of proposals for any kind of tasks) + * Define how the design docs are handled/created *before* the publication (this proposal is about the publishing process) + +## Proposed solution + + * Open a dedicated Jira (`HDDS-*` but with specific component) + * Use standard name prefix in the jira (easy to filter on the mailing list) `[OEP] + * Create a PR to add the design doc to the current documentation + * The content of the design can be added to the documentation (Recommended) + * Or can be added as external reference + * The design doc (or the summary with the reference) will be merged to the design doc folder of `hadoop-hdds/docs/content/design` (will be part of the docs) + * Discuss it as before (lazy consesus, except if somebody calls for a real vote) + * Design docs can be updated according to the changes during the implementation + * Only the implemented design docs will be visible as part of the design docs + + +As a result all the design docs can be listed under the documentation page. + +A good design doc has the following properties: + + 1. Publicly available for anybody (Please try to avoid services which are available only with registration, eg: google docs) + 2. Archived for the future (Commit it to the source OR use apache jira or wiki) + 3. Editable later (Best format is markdown, RTF is also good. PDF has a limitation, it's very hard to reuse the text, or create an updated design doc) + 4. Well structured to make it easy to comment any part of the document (Markdown files which are part of the pull request can be commented in the PR line by line) + + +### Example 1: Design doc as a markdown file + +The easiest way to create a design doc is to create a new markdown file in a PR and merge it to `hadoop-hdds/docs/content/design`. + + 1. Publicly available: YES, it can be linked from Apache git or github + 2. Archived: YES, and it's also versioned. All the change history can be tracked. + 3. Editable later: YES, as it's just a simple text file + 4. Commentable: YES, comment can be added to each line. + +### Example 2: Design doc as a PDF + +A very common practice of today is to create design doc on google docs and upload it to the JIRA. + + 1. Publicy available: YES, anybody can download it from the Jira. + 2. Archived: YES, it's available from Apache infra. + 3. Editable: NO, It's harder to reuse the text to import to the docs or create a new design doc. + 4. Commentable: PARTIAL, Not as easy as a text file or the original google docs, but a good structure with numbered section may help + + +### The format + +While the first version (markdown files) are the most powerful, the second version (the existing practice) is also acceptable. In this case we propose to create a PR with adding a reference page *without* the content but including the link. + +For example: + +```yaml +--- +title: Ozone Security Design +summary: A comprehensive description of the security flow between server and client components. +date: 2018-02-22 +jira: HDDS-4 +status: implemented +author: Sanjay Radia, Jitendra Pandey, Xiaoyu Yao, Anu Engineer + +## Summary + +Ozone security model is based on Kerberos and similar to the Hadoop security but some of the parts are improved: for example the SCM works as a Certificate Authority and PKI based solutions are wildely used. + +## Reference + +For more details please check the (uploaded design doc)[https://issues.apache.org/jira/secure/attachment/12911638/HadoopStorageLayerSecurity.pdf]. + +``` + +Obviously with the first approach the design doc itself can be included in this markdown file. + +## Migration + +It's not a hard requirement to migrate all the design doc. But process is always open: + + 1. To create reference pages for any of the old design docs + 2. To migrate any new design docs to markdown formats (by anybody not just by the author) + 3. To update any of the old design docs based on the current state of the code (We have versioning!) + +## Document template + +This the proposed template to document any proposal. It's recommended but not required the use exactly the some structure. Some proposal may require different structure, but we need the following information. + +1. Summary + +> Give a one sentence summary, like the jira title. It will be displayed on the documentation page. Should be enough to understand + +2. Status + +Defined in the markdown header. Proposed statuses: + + * `accepted`: (Use this as by default. If not accapted, won't be merged) + + * `implemented`: The discussed technical solution is implemented (maybe with some minor implementation difference) + + * `replaced`: Replaced by a new design doc + + * `outdated`: Code has been changed and design doc doesn't reflect any more the state of the current code. + + Note: the _accepted_ design docs won't be visible as part of the documentation or only under a dedicated section to clearly comminucate that it's not ready, yet. + +3. Problem statement (Motivation / Abstract) + +> What is the problem and how would you solve it? Think about an abstract of a paper: one paragraph overview. Why will the world better with this change? + +4. Non-goals + + > Very important to define what is outside of the scope of this proposal + +5. Technical Description (Architecture and implementation details) + + > Explain the problem in more details. How can it be reproduced? What is the current solution? What is the limitation of the current solution? + + > How the new proposed solution would solve the problem? Architectural design. + + > Implementation details. What should be changed in the code. Is it a huge change? Do we need to change wire protocol? Backward compatibility? + +6. Alternatives + + > What are the other alternatives you considered and why do yoy prefer the proposed solution The goal of this section is to help people understand why this is the best solution now, and also to prevent churn in the future when old alternatives are reconsidered. + +Note: In some cases 4/5 can be combined. For example if you have multiple proposals, the first version may include multiple solutions. At the end ot the discussion we can move the alternatives to 5. and explain why the community is decided to use the selected option. + +7. Plan + + > Planning to implement the feature. Estimated size of the work? Do we need feature branch? Any migration plan, dependency? If it's not a big new feature it can be one sentence or optional. + +8. References + +## Workflows form other projects + +There are similar process in other open source projects. This document and the template is inspired by the following projects: + + * [Apache Kafka Improvement Proposals](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals) + * [Apache Spark Project Improvement Proposals](https://spark.apache.org/improvement-proposals.html) + * [Kubernetes Enhancement Proposals](https://github.com/kubernetes/enhancements/tree/master/keps) + +Short summary of the processes: + +__Kafka__ process: + + * Create wiki page + * Start discussion on mail thread + * Vote on mail thread + +__Spark__ process: + + * Create JIRA (dedicated label) + * Discuss on the jira page + * Vote on dev list + +*Kubernetes*: + + * Deditaced git repository + * KEPs are committed to the repo + * Well defined approval process managed by SIGs (KEPs are assigned to SIGs) + diff --git a/hadoop-hdds/docs/content/gdpr/GDPR in Ozone.md b/hadoop-hdds/docs/content/gdpr/GDPR in Ozone.md new file mode 100644 index 0000000000000..dd23e04941690 --- /dev/null +++ b/hadoop-hdds/docs/content/gdpr/GDPR in Ozone.md @@ -0,0 +1,42 @@ +--- +title: "GDPR in Ozone" +date: "2019-September-17" +weight: 5 +summary: GDPR in Ozone +icon: user +--- + + + +Enabling GDPR compliance in Ozone is very straight forward. During bucket +creation, you can specify `--enforcegdpr=true` or `-g=true` and this will +ensure the bucket is GDPR compliant. Thus, any key created under this bucket +will automatically be GDPR compliant. + +GDPR can only be enabled on a new bucket. For existing buckets, you would +have to create a new GDPR compliant bucket and copy data from old bucket into + new bucket to take advantage of GDPR. + +Example to create a GDPR compliant bucket: + +`ozone sh bucket create --enforcegdpr=true /hive/jan` + +`ozone sh bucket create -g=true /hive/jan` + +If you want to create an ordinary bucket then you can skip `--enforcegdpr` +and `-g` flags. \ No newline at end of file diff --git a/hadoop-hdds/docs/content/gdpr/_index.md b/hadoop-hdds/docs/content/gdpr/_index.md new file mode 100644 index 0000000000000..9888369023bf5 --- /dev/null +++ b/hadoop-hdds/docs/content/gdpr/_index.md @@ -0,0 +1,38 @@ +--- +title: GDPR +name: GDPR +identifier: gdpr +menu: main +weight: 5 +--- + + +{{}} + The General Data Protection Regulation (GDPR) is a law that governs how personal data should be handled. This is an European Union law, but due to the nature of software oftentimes spills into other geographies. + Ozone supports GDPR's Right to Erasure(Right to be Forgotten). +{{}} + +     + +Once you create a GDPR compliant bucket, any key created in that bucket will +automatically by GDPR compliant. + + diff --git a/hadoop-hdds/docs/content/genconf.md b/hadoop-hdds/docs/content/genconf.md deleted file mode 100644 index 838a4c7f41359..0000000000000 --- a/hadoop-hdds/docs/content/genconf.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: "Generate Configurations" -date: 2018-12-18 -menu: - main: - parent: Tools ---- - - -Genconf tool generates a template ozone-site.xml file at the specified path. -This template file can be edited to replace with proper values. - -`ozone genconf ` diff --git a/hadoop-hdds/docs/content/JavaApi.md b/hadoop-hdds/docs/content/interface/JavaApi.md similarity index 92% rename from hadoop-hdds/docs/content/JavaApi.md rename to hadoop-hdds/docs/content/interface/JavaApi.md index 293ee07c2322f..bb18068f40006 100644 --- a/hadoop-hdds/docs/content/JavaApi.md +++ b/hadoop-hdds/docs/content/interface/JavaApi.md @@ -1,9 +1,8 @@ --- title: "Java API" date: "2017-09-14" -menu: - main: - parent: "Client" +weight: 1 +summary: Ozone has a set of Native RPC based APIs. This is the lowest level API's on which all other protocols are built. This is the most performant and feature-full of all Ozone protocols. --- -Introduction -------------- - -Ozone ships with its own client library that supports RPC. For generic use cases the S3 -compatible REST interface also can be used instead of the Ozone client. +Ozone ships with its own client library that supports RPC. For generic use cases the S3 +compatible REST interface also can be used instead of the Ozone client. ## Creating an Ozone client @@ -78,21 +74,21 @@ It is possible to pass an array of arguments to the createVolume by creating vol Once you have a volume, you can create buckets inside the volume. -{{< highlight bash >}} +{{< highlight java >}} // Let us create a bucket called videos. assets.createBucket("videos"); OzoneBucket video = assets.getBucket("videos"); {{< /highlight >}} -At this point we have a usable volume and a bucket. Our volume is called assets and bucket is called videos. +At this point we have a usable volume and a bucket. Our volume is called _assets_ and bucket is called _videos_. Now we can create a Key. ### Reading and Writing a Key -With a bucket object the users can now read and write keys. The following code reads a video called intro.mp4 from the local disk and stores in the video bucket that we just created. +With a bucket object the users can now read and write keys. The following code reads a video called intro.mp4 from the local disk and stores in the _video_ bucket that we just created. -{{< highlight bash >}} +{{< highlight java >}} // read data from the file, this is a user provided function. byte [] videoData = readFile("intro.mp4"); diff --git a/hadoop-hdds/docs/content/interface/OzoneFS.md b/hadoop-hdds/docs/content/interface/OzoneFS.md new file mode 100644 index 0000000000000..fcfef6dde3d6a --- /dev/null +++ b/hadoop-hdds/docs/content/interface/OzoneFS.md @@ -0,0 +1,155 @@ +--- +title: Ozone File System +date: 2017-09-14 +weight: 2 +summary: Hadoop Compatible file system allows any application that expects an HDFS like interface to work against Ozone with zero changes. Frameworks like Apache Spark, YARN and Hive work against Ozone without needing any change. +--- + + +The Hadoop compatible file system interface allows storage backends like Ozone +to be easily integrated into Hadoop eco-system. Ozone file system is an +Hadoop compatible file system. + +## Setting up the Ozone file system + +To create an ozone file system, we have to choose a bucket where the file system would live. This bucket will be used as the backend store for OzoneFileSystem. All the files and directories will be stored as keys in this bucket. + +Please run the following commands to create a volume and bucket, if you don't have them already. + +{{< highlight bash >}} +ozone sh volume create /volume +ozone sh bucket create /volume/bucket +{{< /highlight >}} + +Once this is created, please make sure that bucket exists via the _list volume_ or _list bucket_ commands. + +Please add the following entry to the core-site.xml. + +{{< highlight xml >}} + + fs.o3fs.impl + org.apache.hadoop.fs.ozone.OzoneFileSystem + + + fs.AbstractFileSystem.o3fs.impl + org.apache.hadoop.fs.ozone.OzFs + + + fs.defaultFS + o3fs://bucket.volume + +{{< /highlight >}} + +This will make this bucket to be the default file system for HDFS dfs commands and register the o3fs file system type. + +You also need to add the ozone-filesystem.jar file to the classpath: + +{{< highlight bash >}} +export HADOOP_CLASSPATH=/opt/ozone/share/ozonefs/lib/hadoop-ozone-filesystem-lib-current*.jar:$HADOOP_CLASSPATH +{{< /highlight >}} + +Once the default Filesystem has been setup, users can run commands like ls, put, mkdir, etc. +For example, + +{{< highlight bash >}} +hdfs dfs -ls / +{{< /highlight >}} + +or + +{{< highlight bash >}} +hdfs dfs -mkdir /users +{{< /highlight >}} + + +Or put command etc. In other words, all programs like Hive, Spark, and Distcp will work against this file system. +Please note that any keys created/deleted in the bucket using methods apart from OzoneFileSystem will show up as directories and files in the Ozone File System. + +Note: Bucket and volume names are not allowed to have a period in them. +Moreover, the filesystem URI can take a fully qualified form with the OM host and an optional port as a part of the path following the volume name. +For example, you can specify both host and port: + +{{< highlight bash>}} +hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:5678/key +{{< /highlight >}} + +When the port number is not specified, it will be retrieved from config key `ozone.om.address` +if defined; or it will fall back to the default port `9862`. +For example, we have `ozone.om.address` configured as following in `ozone-site.xml`: + +{{< highlight xml >}} + + ozone.om.address + 0.0.0.0:6789 + +{{< /highlight >}} + +When we run command: + +{{< highlight bash>}} +hdfs dfs -ls o3fs://bucket.volume.om-host.example.com/key +{{< /highlight >}} + +The above command is essentially equivalent to: + +{{< highlight bash>}} +hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:6789/key +{{< /highlight >}} + +Note: Only port number from the config is used in this case, +whereas the host name in the config `ozone.om.address` is ignored. + + +## Supporting older Hadoop version (Legacy jar, BasicOzoneFilesystem) + +There are two ozonefs files, both of them include all the dependencies: + + * share/ozone/lib/hadoop-ozone-filesystem-lib-current-VERSION.jar + * share/ozone/lib/hadoop-ozone-filesystem-lib-legacy-VERSION.jar + +The first one contains all the required dependency to use ozonefs with a + compatible hadoop version (hadoop 3.2). + +The second one contains all the dependency in an internal, separated directory, + and a special class loader is used to load all the classes from the location. + +With this method the hadoop-ozone-filesystem-lib-legacy.jar can be used from + any older hadoop version (eg. hadoop 3.1, hadoop 2.7 or spark+hadoop 2.7) + +Similar to the dependency jar, there are two OzoneFileSystem implementation. + +For hadoop 3.0 and newer, you can use `org.apache.hadoop.fs.ozone.OzoneFileSystem` + which is a full implementation of the Hadoop compatible File System API. + +For Hadoop 2.x you should use the Basic version: `org.apache.hadoop.fs.ozone.BasicOzoneFileSystem`. + +This is the same implementation but doesn't include the features/dependencies which are added with + Hadoop 3.0. (eg. FS statistics, encryption zones). + +### Summary + +The following table summarize which jar files and implementation should be used: + +Hadoop version | Required jar | OzoneFileSystem implementation +---------------|-------------------------|---------------------------------------------------- +3.2 | filesystem-lib-current | org.apache.hadoop.fs.ozone.OzoneFileSystem +3.1 | filesystem-lib-legacy | org.apache.hadoop.fs.ozone.OzoneFileSystem +2.9 | filesystem-lib-legacy | org.apache.hadoop.fs.ozone.BasicOzoneFileSystem +2.7 | filesystem-lib-legacy | org.apache.hadoop.fs.ozone.BasicOzoneFileSystem + With this method the hadoop-ozone-filesystem-lib-legacy.jar can be used from + any older hadoop version (eg. hadoop 2.7 or spark+hadoop 2.7) diff --git a/hadoop-hdds/docs/content/S3.md b/hadoop-hdds/docs/content/interface/S3.md similarity index 91% rename from hadoop-hdds/docs/content/S3.md rename to hadoop-hdds/docs/content/interface/S3.md index 5a9e4a512fe1a..6a8e2d7c53b02 100644 --- a/hadoop-hdds/docs/content/S3.md +++ b/hadoop-hdds/docs/content/interface/S3.md @@ -1,9 +1,7 @@ --- -title: S3 -menu: - main: - parent: Client - weight: 1 +title: S3 Protocol +weight: 3 +summary: Ozone supports Amazon's Simple Storage Service (S3) protocol. In fact, You can use S3 clients and S3 SDK based applications without any modifications with Ozone. --- + Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools. ## Getting started -S3 Gateway is a separated component which provides the S3 compatible. It should be started additional to the regular Ozone components. +S3 Gateway is a separated component which provides the S3 compatible APIs. It should be started additional to the regular Ozone components. You can start a docker based cluster, including the S3 gateway from the release package. @@ -94,7 +93,7 @@ If security is not enabled, you can *use* **any** AWS_ACCESS_KEY_ID and AWS_SECR If security is enabled, you can get the key and the secret with the `ozone s3 getsecret` command (*kerberos based authentication is required). -``` +```bash /etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM ozone s3 getsecret awsAccessKey=testuser/scm@EXAMPLE.COM @@ -104,7 +103,7 @@ awsSecret=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999 Now, you can use the key and the secret to access the S3 endpoint: -``` +```bash export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999 aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1 @@ -117,10 +116,10 @@ aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1 To show the storage location of a S3 bucket, use the `ozone s3 path ` command. -``` +```bash aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket=bucket1 -ozone sh bucket path bucket1 +ozone s3 path bucket1 Volume name for S3Bucket is : s3thisisakey Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey ``` @@ -129,23 +128,23 @@ Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey ### AWS Cli -`aws` CLI could be used with specifying the custom REST endpoint. +`aws` CLI could be used by specifying the custom REST endpoint. -``` +```bash aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest ``` Or -``` +```bash aws s3 ls --endpoint http://localhost:9878 s3://buckettest ``` ### S3 Fuse driver (goofys) -Goofys is a S3 FUSE driver. It could be used to mount any Ozone bucket as posix file system: +Goofys is a S3 FUSE driver. It could be used to mount any Ozone bucket as posix file system. -``` +```bash goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1 ``` diff --git a/hadoop-hdds/docs/content/interface/_index.md b/hadoop-hdds/docs/content/interface/_index.md new file mode 100644 index 0000000000000..254864732fb84 --- /dev/null +++ b/hadoop-hdds/docs/content/interface/_index.md @@ -0,0 +1,27 @@ +--- +title: "Programming Interfaces" +menu: + main: + weight: 4 +--- + + +{{}} +Ozone is a multi-protocol file system. There are different protocols by which + users can access data on Ozone. +{{}} diff --git a/hadoop-hdds/docs/content/recipe/Prometheus.md b/hadoop-hdds/docs/content/recipe/Prometheus.md new file mode 100644 index 0000000000000..310d078567b17 --- /dev/null +++ b/hadoop-hdds/docs/content/recipe/Prometheus.md @@ -0,0 +1,95 @@ +--- +title: Monitoring with Prometheus +summary: A Simple recipe to monitor Ozone using Prometheus +linktitle: Prometheus +--- + + +[Prometheus](https://prometheus.io/) is an open-source monitoring server developed under under the [Cloud Native Computing Foundation](https://www.cncf.io/). + +Ozone supports Prometheus out of the box. The servers start a prometheus +compatible metrics endpoint where all the available hadoop metrics are published in prometheus exporter format. + +## Prerequisites + + 1. [Install the and start]({{< ref "start/RunningViaDocker.md" >}}) an Ozone cluster. + 2. [Download](https://prometheus.io/download/#prometheus) the prometheus binary. + +## Monitoring with prometheus + +* To enable the Prometheus metrics endpoint you need to add a new configuration to the `ozone-site.xml` file. + + ```xml + + hdds.prometheus.endpoint.enabled + true + +``` + +_Note_: for Docker compose based pseudo cluster put the \ +`OZONE-SITE.XML_hdds.prometheus.endpoint.enabled=true` line to the `docker-config` file. + +* Restart the Ozone Manager and Storage Container Manager and check the prometheus endpoints: + + * http://scm:9874/prom + + * http://ozoneManager:9876/prom + +* Create a prometheus.yaml configuration with the previous endpoints: + +```yaml +global: + scrape_interval: 15s + +scrape_configs: + - job_name: ozone + metrics_path: /prom + static_configs: + - targets: + - "scm:9876" + - "ozoneManager:9874" +``` + +* Start with prometheus from the directory where you have the prometheus.yaml file: + +```bash +prometheus +``` + +* Check the active targets in the prometheus web-ui: + +http://localhost:9090/targets + +![Prometheus target page example](prometheus.png) + + +* Check any metrics on the prometheus web ui.\ +For example: + +http://localhost:9090/graph?g0.range_input=1h&g0.expr=om_metrics_num_key_allocate&g0.tab=1 + +![Prometheus metrics page example](prometheus-key-allocate.png) + +## Note + +The ozone distribution contains a ready-to-use, dockerized environment to try out ozone and prometheus. It can be found under `compose/ozoneperf` directory. + +```bash +cd compose/ozoneperf +docker-compose up -d +``` \ No newline at end of file diff --git a/hadoop-hdds/docs/content/SparkOzoneFSK8S.md b/hadoop-hdds/docs/content/recipe/SparkOzoneFSK8S.md similarity index 87% rename from hadoop-hdds/docs/content/SparkOzoneFSK8S.md rename to hadoop-hdds/docs/content/recipe/SparkOzoneFSK8S.md index fa6cacd6ed4ad..9f9d3478c9bca 100644 --- a/hadoop-hdds/docs/content/SparkOzoneFSK8S.md +++ b/hadoop-hdds/docs/content/recipe/SparkOzoneFSK8S.md @@ -1,8 +1,7 @@ --- title: Spark in Kubernetes with OzoneFS -menu: - main: - parent: Recipes +linktitle: Spark +summary: How to use Apache Spark with Ozone on K8s? --- -Using Ozone from Apache Spark -=== - This recipe shows how Ozone object store can be used from Spark using: - OzoneFS (Hadoop compatible file system) @@ -34,7 +30,7 @@ This recipe shows how Ozone object store can be used from Spark using: ## Requirements -Download latest Spark and Ozone distribution and extract them. This method is +Download latest Spark and Ozone distribution and extract them. This method is tested with the `spark-2.4.0-bin-hadoop2.7` distribution. You also need the following: @@ -47,16 +43,16 @@ You also need the following: ### Create the base Spark driver/executor image -First of all create a docker image with the Spark image creator. +First of all create a docker image with the Spark image creator. Execute the following from the Spark distribution -``` +```bash ./bin/docker-image-tool.sh -r myrepo -t 2.4.0 build ``` _Note_: if you use Minikube add the `-m` flag to use the docker daemon of the Minikube image: -``` +```bash ./bin/docker-image-tool.sh -m -r myrepo -t 2.4.0 build ``` @@ -68,18 +64,22 @@ Create a new directory for customizing the created docker image. Copy the `ozone-site.xml` from the cluster: -``` +```bash kubectl cp om-0:/opt/hadoop/etc/hadoop/ozone-site.xml . ``` -And create a custom `core-site.xml`: +And create a custom `core-site.xml`. -``` +```xml fs.o3fs.impl org.apache.hadoop.fs.ozone.BasicOzoneFileSystem + + fs.AbstractFileSystem.o3fs.impl + org.apache.hadoop.fs.ozone.OzFs + ``` @@ -102,13 +102,13 @@ ENV SPARK_EXTRA_CLASSPATH=/opt/hadoop/conf ADD hadoop-ozone-filesystem-lib-legacy-0.4.0-SNAPSHOT.jar /opt/hadoop-ozone-filesystem-lib-legacy.jar ``` -``` +```bash docker build -t myrepo/spark-ozone ``` For remote kubernetes cluster you may need to push it: -``` +```bash docker push myrepo/spark-ozone ``` @@ -116,7 +116,7 @@ docker push myrepo/spark-ozone Download any text file and put it to the `/tmp/alice.txt` first. -``` +```bash kubectl port-forward s3g-0 9878:9878 aws s3api --endpoint http://localhost:9878 create-bucket --bucket=test aws s3api --endpoint http://localhost:9878 put-object --bucket test --key alice.txt --body /tmp/alice.txt @@ -134,7 +134,7 @@ Write down the ozone filesystem uri as it should be used with the spark-submit c ## Create service account to use -``` +```bash kubectl create serviceaccount spark -n yournamespace kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=yournamespace:spark --namespace=yournamespace ``` @@ -142,13 +142,14 @@ kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount Execute the following spark-submit command, but change at least the following values: - * the kubernetes master url (you can check your ~/.kube/config to find the actual value) - * the kubernetes namespace (yournamespace in this example) - * serviceAccountName (you can use the _spark_ value if you folllowed the previous steps) - * container.image (in this example this is myrepo/spark-ozone. This is pushed to the registry in the previous steps) - * location of the input file (o3fs://...), use the string which is identified earlier with the `ozone s3 path ` command + * the kubernetes master url (you can check your _~/.kube/config_ to find the actual value) + * the kubernetes namespace (_yournamespace_ in this example) + * serviceAccountName (you can use the _spark_ value if you followed the previous steps) + * container.image (in this example this is _myrepo/spark-ozone_. This is pushed to the registry in the previous steps) + * location of the input file (o3fs://...), use the string which is identified earlier with the \ + `ozone s3 path ` command -``` +```bash bin/spark-submit \ --master k8s://https://kubernetes:6443 \ --deploy-mode cluster \ @@ -166,7 +167,8 @@ bin/spark-submit \ Check the available `spark-word-count-...` pods with `kubectl get pod` -Check the output of the calculation with `kubectl logs spark-word-count-1549973913699-driver` +Check the output of the calculation with \ +`kubectl logs spark-word-count-1549973913699-driver` You should see the output of the wordcount job. For example: diff --git a/hadoop-hdds/docs/content/recipe/_index.md b/hadoop-hdds/docs/content/recipe/_index.md new file mode 100644 index 0000000000000..47053ab6fbba8 --- /dev/null +++ b/hadoop-hdds/docs/content/recipe/_index.md @@ -0,0 +1,29 @@ +--- +title: Recipes +date: "2017-10-10" +menu: main +weight: 9 + +--- + + + +{{}} + Standard how-to documents which describe how to use Ozone with other Software. + For example, how to use Ozone with Apache Spark. +{{}} diff --git a/hadoop-hdds/docs/static/prometheus-key-allocate.png b/hadoop-hdds/docs/content/recipe/prometheus-key-allocate.png similarity index 100% rename from hadoop-hdds/docs/static/prometheus-key-allocate.png rename to hadoop-hdds/docs/content/recipe/prometheus-key-allocate.png diff --git a/hadoop-hdds/docs/static/prometheus.png b/hadoop-hdds/docs/content/recipe/prometheus.png similarity index 100% rename from hadoop-hdds/docs/static/prometheus.png rename to hadoop-hdds/docs/content/recipe/prometheus.png diff --git a/hadoop-hdds/docs/content/security/SecuityWithRanger.md b/hadoop-hdds/docs/content/security/SecuityWithRanger.md new file mode 100644 index 0000000000000..cbbd53ec7c128 --- /dev/null +++ b/hadoop-hdds/docs/content/security/SecuityWithRanger.md @@ -0,0 +1,43 @@ +--- +title: "Apache Ranger" +date: "2019-April-03" +weight: 5 +summary: Apache Ranger is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. +icon: user +--- + + + +Apache Ranger™ is a framework to enable, monitor and manage comprehensive data +security across the Hadoop platform. Any version of Apache Ranger which is greater +than 1.20 is aware of Ozone, and can manage an Ozone cluster. + + +To use Apache Ranger, you must have Apache Ranger installed in your Hadoop +Cluster. For installation instructions of Apache Ranger, Please take a look +at the [Apache Ranger website](https://ranger.apache.org/index.html). + +If you have a working Apache Ranger installation that is aware of Ozone, then +configuring Ozone to work with Apache Ranger is trivial. You have to enable +the ACLs support and set the acl authorizer class inside Ozone to be Ranger +authorizer. Please add the following properties to the ozone-site.xml. + +Property|Value +--------|------------------------------------------------------------ +ozone.acl.enabled | true +ozone.acl.authorizer.class| org.apache.ranger.authorization.ozone.authorizer.RangerOzoneAuthorizer diff --git a/hadoop-hdds/docs/content/security/SecureOzone.md b/hadoop-hdds/docs/content/security/SecureOzone.md new file mode 100644 index 0000000000000..d4d836fcf7fe9 --- /dev/null +++ b/hadoop-hdds/docs/content/security/SecureOzone.md @@ -0,0 +1,178 @@ +--- +title: "Securing Ozone" +date: "2019-April-03" +summary: Overview of Ozone security concepts and steps to secure Ozone Manager and SCM. +weight: 1 +icon: tower +--- + + + +# Kerberos + +Ozone depends on [Kerberos](https://web.mit.edu/kerberos/) to make the +clusters secure. Historically, HDFS has supported running in an isolated +secure networks where it is possible to deploy without securing the cluster. + +This release of Ozone follows that model, but soon will move to _secure by +default._ Today to enable security in ozone cluster, we need to set the +configuration **ozone.security.enabled** to _true_ and **hadoop.security.authentication** +to _kerberos_. + +Property|Value +----------------------|--------- +ozone.security.enabled| _true_ +hadoop.security.authentication| _kerberos_ + +# Tokens # + +Ozone uses a notion of tokens to avoid overburdening the Kerberos server. +When you serve thousands of requests per second, involving Kerberos might not +work well. Hence once an authentication is done, Ozone issues delegation +tokens and block tokens to the clients. These tokens allow applications to do +specified operations against the cluster, as if they have kerberos tickets +with them. Ozone supports following kinds of tokens. + +### Delegation Token ### +Delegation tokens allow an application to impersonate a users kerberos +credentials. This token is based on verification of kerberos identity and is +issued by the Ozone Manager. Delegation tokens are enabled by default when +security is enabled. + +### Block Token ### + +Block tokens allow a client to read or write a block. This is needed so that +data nodes know that the user/client has permission to read or make +modifications to the block. + +### S3Token ### + +S3 uses a very different shared secret security scheme. Ozone supports the AWS Signature Version 4 protocol, +and from the end users perspective Ozone's s3 feels exactly like AWS S3. + +The S3 credential tokens are called S3 tokens in the code. These tokens are +also enabled by default when security is enabled. + + +Each of the service daemons that make up Ozone needs a Kerberos service +principal name and a corresponding [kerberos key tab](https://web.mit.edu/kerberos/krb5-latest/doc/basic/keytab_def.html) file. + +All these settings should be made in ozone-site.xml. + +
    +
    +
    +

    Storage Container Manager

    +

    +
    + SCM requires two Kerberos principals, and the corresponding key tab files + for both of these principals. +
    + + + + + + + + + + + + + + + + + + + + + +
    PropertyDescription
    hdds.scm.kerberos.principal + The SCM service principal.
    e.g. scm/_HOST@REALM.COM
    hdds.scm.kerberos.keytab.file + The keytab file used by SCM daemon to login as its service principal.
    hdds.scm.http.kerberos.principal + SCM http server service principal.
    hdds.scm.http.kerberos.keytab + The keytab file used by SCM http server to login as its service principal.
    +

    +
    +
    +
    +

    Ozone Manager

    +

    +
    + Like SCM, OM also requires two Kerberos principals, and the + corresponding key tab files for both of these principals. +
    + + + + + + + + + + + + + + + + + + + + + +
    PropertyDescription
    ozone.om.kerberos.principal + The OzoneManager service principal.
    e.g. om/_HOST@REALM.COM
    ozone.om.kerberos.keytab.file + TThe keytab file used by SCM daemon to login as its service principal.
    ozone.om.http.kerberos.principal + Ozone Manager http server service principal.
    ozone.om.http.kerberos.keytab + The keytab file used by OM http server to login as its service principal.
    +

    +
    +
    +
    +

    S3 Gateway

    +

    +
    + S3 gateway requires one service principal and here the configuration values + needed in the ozone-site.xml. +
    + + + + + + + + + + + + + + + +
    PropertyDescription
    ozone.s3g.authentication.kerberos.principal + S3 Gateway principal.
    e.g. HTTP/_HOST@EXAMPLE.COM
    ozone.s3g.keytab.file + The keytab file used by S3 gateway
    +

    +
    +
    diff --git a/hadoop-hdds/docs/content/security/SecuringDatanodes.md b/hadoop-hdds/docs/content/security/SecuringDatanodes.md new file mode 100644 index 0000000000000..6b7d82365cbfe --- /dev/null +++ b/hadoop-hdds/docs/content/security/SecuringDatanodes.md @@ -0,0 +1,73 @@ +--- +title: "Securing Datanodes" +date: "2019-April-03" +weight: 2 +summary: Explains different modes of securing data nodes. These range from kerberos to auto approval. +icon: th +--- + + + +Datanodes under Hadoop is traditionally secured by creating a Keytab file on +the data nodes. With Ozone, we have moved away to using data node +certificates. That is, Kerberos on data nodes is not needed in case of a +secure Ozone cluster. + +However, we support the legacy Kerberos based Authentication to make it easy +for the current set of users.The HDFS configuration keys are the following +that is setup in hdfs-site.xml. + +Property|Description +--------|-------------- +dfs.datanode.kerberos.principal|The datanode service principal.
    e.g. dn/_HOST@REALM.COM +dfs.datanode.keytab.file| The keytab file used by datanode daemon to login as its service principal. +hdds.datanode.http.kerberos.principal| Datanode http server service principal. +hdds.datanode.http.kerberos.keytab| The keytab file used by datanode http server to login as its service principal. + + +## How a data node becomes secure. + +Under Ozone, when a data node boots up and discovers SCM's address, the first +thing that data node does is to create a private key and send a certificate +request to the SCM. + +

    Certificate Approval via Kerberos Current Model

    +SCM has a built-in CA, and SCM has to approve this request. If the data node +already has a Kerberos key tab, then SCM will trust Kerberos credentials and +issue a certificate automatically. + + +

    Manual Approval In Progress

    +If these are band new data nodes and Kerberos key tabs are not present at the +data nodes, then this request for the data nodes identity certificate is +queued up for approval from the administrator(This is work in progress, +not committed in Ozone yet). In other words, the web of trust is established +by the administrator of the cluster. + +

    Automatic Approval In Progress

    +If you running under an container orchestrator like Kubernetes, we rely on +Kubernetes to create a one-time token that will be given to data node during +boot time to prove the identity of the data node container (This is also work +in progress.) + + +Once a certificate is issued, a data node is secure and Ozone manager can +issue block tokens. If there is no data node certificates or the SCM's root +certificate is not present in the data node, then data node will register +itself and down load the SCM's root certificate as well get the certificates +for itself. diff --git a/hadoop-hdds/docs/content/security/SecuringS3.md b/hadoop-hdds/docs/content/security/SecuringS3.md new file mode 100644 index 0000000000000..1cb0c809e6116 --- /dev/null +++ b/hadoop-hdds/docs/content/security/SecuringS3.md @@ -0,0 +1,61 @@ +--- +title: "Securing S3" +date: "2019-April-03" +summary: Ozone supports S3 protocol, and uses AWS Signature Version 4 protocol which allows a seamless S3 experience. +weight: 4 +icon: cloud +--- + + +To access an S3 bucket, users need AWS access key ID and AWS secret. Both of +these are generated by going to AWS website. When you use Ozone's S3 +protocol, you need the same AWS access key and secret. + +Under Ozone, the clients can download the access key directly from Ozone. +The user needs to `kinit` first and once they have authenticated via kerberos + they can download the S3 access key ID and AWS secret. Just like AWS S3, + both of these are secrets that needs to be protected by the client since it + gives full access to the S3 buckets. + + +* S3 clients can get the secret access id and user secret from OzoneManager. + +```bash +ozone s3 getsecret +``` +This command will talk to ozone, validate the user via kerberos and generate +the AWS credentials. The values will be printed out on the screen. You can +set these values up in your _.aws_ file for automatic access while working +against Ozone S3 buckets. + + + + +* Now you can proceed to setup these secrets in aws configs: + +```bash +aws configure set default.s3.signature_version s3v4 +aws configure set aws_access_key_id ${accessId} +aws configure set aws_secret_access_key ${secret} +aws configure set region us-west-1 +``` +Please refer to AWS S3 documentation on how to use S3 via command line or via +S3 API. diff --git a/hadoop-hdds/docs/content/security/SecuringTDE.md b/hadoop-hdds/docs/content/security/SecuringTDE.md new file mode 100644 index 0000000000000..3e8f2d16819fe --- /dev/null +++ b/hadoop-hdds/docs/content/security/SecuringTDE.md @@ -0,0 +1,65 @@ +--- +title: "Transparent Data Encryption" +date: "2019-April-03" +summary: TDE allows data on the disks to be encrypted-at-rest and automatically decrypted during access. You can enable this per key or per bucket. +weight: 3 +icon: lock +--- + + +Ozone TDE setup process and usage are very similar to HDFS TDE. +The major difference is that Ozone TDE is enabled at Ozone bucket level +when a bucket is created. + +### Setting up the Key Management Server + +To use TDE, clients must setup a Key Management Server and provide that URI to +Ozone/HDFS. Since Ozone and HDFS can use the same Key Management Server, this + configuration can be provided via *hdfs-site.xml*. + +Property| Value +-----------------------------------|----------------------------------------- +hadoop.security.key.provider.path | KMS uri.
    e.g. kms://http@kms-host:9600/kms + +### Using Transparent Data Encryption +If this is already configured for your cluster, then you can simply proceed +to create the encryption key and enable encrypted buckets. + +To create an encrypted bucket, client need to: + + * Create a bucket encryption key with hadoop key CLI, which is similar to + how you would use HDFS encryption zones. + + ```bash + hadoop key create encKey + ``` + The above command creates an encryption key for the bucket you want to protect. + Once the key is created, you can tell Ozone to use that key when you are + reading and writing data into a bucket. + + * Assign the encryption key to a bucket. + + ```bash + ozone sh bucket create -k encKey /vol/encryptedBucket + ``` + +After this command, all data written to the _encryptedBucket_ will be encrypted +via the encKey and while reading the clients will talk to Key Management +Server and read the key and decrypt it. In other words, the data stored +inside Ozone is always encrypted. The fact that data is encrypted at rest +will be completely transparent to the clients and end users. diff --git a/hadoop-hdds/docs/content/security/SecurityAcls.md b/hadoop-hdds/docs/content/security/SecurityAcls.md new file mode 100644 index 0000000000000..31bbb0a95cc2a --- /dev/null +++ b/hadoop-hdds/docs/content/security/SecurityAcls.md @@ -0,0 +1,85 @@ +--- +title: "Ozone ACLs" +date: "2019-April-03" +weight: 6 +summary: Native Ozone Authorizer provides Access Control List (ACL) support for Ozone without Ranger integration. +icon: transfer +--- + + +Ozone supports a set of native ACLs. These ACLs can be used independently or +along with Ranger. If Apache Ranger is enabled, then ACL will be checked +first with Ranger and then Ozone's internal ACLs will be evaluated. + +Ozone ACLs are a super set of Posix and S3 ACLs. + +The general format of an ACL is _object_:_who_:_rights_. + +Where an _object_ can be: + +1. **Volume** - An Ozone volume. e.g. _/volume_ +2. **Bucket** - An Ozone bucket. e.g. _/volume/bucket_ +3. **Key** - An object key or an object. e.g. _/volume/bucket/key_ +4. **Prefix** - A path prefix for a specific key. e.g. _/volume/bucket/prefix1/prefix2_ + +Where a _who_ can be: + +1. **User** - A user in the Kerberos domain. User like in Posix world can be +named or unnamed. +2. **Group** - A group in the Kerberos domain. Group also like in Posix world +can +be named or unnamed. +3. **World** - All authenticated users in the Kerberos domain. This maps to +others in the Posix domain. +4. **Anonymous** - Ignore the user field completely. This is an extension to +the Posix semantics, This is needed for S3 protocol, where we express that +we have no way of knowing who the user is or we don't care. + + + + +Where a _right_ can be: + +1. **Create** – This ACL provides a user the ability to create buckets in a +volume and keys in a bucket. Please note: Under Ozone, Only admins can create volumes. +2. **List** – This ACL allows listing of buckets and keys. This ACL is attached + to the volume and buckets which allow listing of the child objects. Please note: The user and admins can list the volumes owned by the user. +3. **Delete** – Allows the user to delete a volume, bucket or key. +4. **Read** – Allows the user to read the metadata of a Volume and Bucket and +data stream and metadata of a key. +5. **Write** - Allows the user to write the metadata of a Volume and Bucket and +allows the user to overwrite an existing ozone key. +6. **Read_ACL** – Allows a user to read the ACL on a specific object. +7. **Write_ACL** – Allows a user to write the ACL on a specific object. + +

    Ozone Native ACL APIs

    + +The ACLs can be manipulated by a set of APIs supported by Ozone. The APIs +supported are: + +1. **SetAcl** – This API will take user principal, the name, type +of the ozone object and a list of ACLs. +2. **GetAcl** – This API will take the name and type of the ozone object +and will return a list of ACLs. +3. **AddAcl** - This API will take the name, type of the ozone object, the +ACL, and add it to existing ACL entries of the ozone object. +4. **RemoveAcl** - This API will take the name, type of the +ozone object and the ACL that has to be removed. diff --git a/hadoop-hdds/docs/content/security/_index.md b/hadoop-hdds/docs/content/security/_index.md new file mode 100644 index 0000000000000..20967e3343be0 --- /dev/null +++ b/hadoop-hdds/docs/content/security/_index.md @@ -0,0 +1,36 @@ +--- +title: Security +name: Security +identifier: SecureOzone +menu: main +weight: 5 +--- + + +{{}} + Ozone is an enterprise class, secure storage system. There are many + optional security features in Ozone. Following pages discuss how + you can leverage the security features of Ozone. +{{}} + + + +Depending on your needs, there are multiple optional steps in securing ozone. diff --git a/hadoop-hdds/docs/content/shell/BucketCommands.md b/hadoop-hdds/docs/content/shell/BucketCommands.md new file mode 100644 index 0000000000000..e81734924fb58 --- /dev/null +++ b/hadoop-hdds/docs/content/shell/BucketCommands.md @@ -0,0 +1,99 @@ +--- +title: Bucket Commands +summary: Bucket commands help you to manage the life cycle of a volume. +weight: 3 +--- + + +Ozone shell supports the following bucket commands. + + * [create](#create) + * [delete](#delete) + * [info](#info) + * [list](#list) + +### Create + +The `bucket create` command allows users to create a bucket. + +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| -g, \-\-enforcegdpr | Optional, if set to true it creates a GDPR compliant bucket, if not specified or set to false, it creates an ordinary bucket. +| Uri | The name of the bucket in **/volume/bucket** format. + + +{{< highlight bash >}} +ozone sh bucket create /hive/jan +{{< /highlight >}} + +The above command will create a bucket called _jan_ in the _hive_ volume. +Since no scheme was specified this command defaults to O3 (RPC) protocol. + +### Delete + +The `bucket delete` command allows users to delete a bucket. If the +bucket is not empty then this command will fail. + +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| Uri | The name of the bucket + +{{< highlight bash >}} +ozone sh bucket delete /hive/jan +{{< /highlight >}} + +The above command will delete _jan_ bucket if it is empty. + +### Info + +The `bucket info` commands returns the information about the bucket. + +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| Uri | The name of the bucket. + +{{< highlight bash >}} +ozone sh bucket info /hive/jan +{{< /highlight >}} + +The above command will print out the information about _jan_ bucket. + +### List + +The `bucket list` command allows users to list the buckets in a volume. + +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| -l, \-\-length | Maximum number of results to return. Default: 100 +| -p, \-\-prefix | Optional, Only buckets that match this prefix will be returned. +| -s, \-\-start | The listing will start from key after the start key. +| Uri | The name of the _volume_. + +{{< highlight bash >}} +ozone sh bucket list /hive +{{< /highlight >}} + +This command will list all buckets on the volume _hive_. diff --git a/hadoop-hdds/docs/content/shell/Format.md b/hadoop-hdds/docs/content/shell/Format.md new file mode 100644 index 0000000000000..72174c9ae9a49 --- /dev/null +++ b/hadoop-hdds/docs/content/shell/Format.md @@ -0,0 +1,69 @@ +--- +title: Shell Overview +summary: Explains the command syntax used by shell command. +weight: 1 +--- + + +Ozone shell help can be invoked at _object_ level or at _action_ level. +For example: + +{{< highlight bash >}} +ozone sh volume --help +{{< /highlight >}} + +This will show all possible actions for volumes. + +or it can be invoked to explain a specific action like +{{< highlight bash >}} +ozone sh volume create --help +{{< /highlight >}} +This command will give you command line options of the create command. + +

    + + +### General Command Format + +The Ozone shell commands take the following format. + +> _ozone sh object action url_ + +**ozone** script is used to invoke all Ozone sub-commands. The ozone shell is +invoked via ```sh``` command. + +The object can be a volume, bucket or a key. The action is various verbs like +create, list, delete etc. + + +Ozone URL can point to a volume, bucket or keys in the following format: + +_\[scheme\]\[server:port\]/volume/bucket/key_ + + +Where, + +1. **Scheme** - This should be `o3` which is the native RPC protocol to access + Ozone API. The usage of the schema is optional. + +2. **Server:Port** - This is the address of the Ozone Manager. If the port is +omitted the default port from ozone-site.xml will be used. + +Depending on the call, the volume/bucket/key names will be part of the URL. +Please see volume commands, bucket commands, and key commands section for more +detail. diff --git a/hadoop-hdds/docs/content/KeyCommands.md b/hadoop-hdds/docs/content/shell/KeyCommands.md similarity index 79% rename from hadoop-hdds/docs/content/KeyCommands.md rename to hadoop-hdds/docs/content/shell/KeyCommands.md index b9d096d39f85b..b4a38c8b1b521 100644 --- a/hadoop-hdds/docs/content/KeyCommands.md +++ b/hadoop-hdds/docs/content/shell/KeyCommands.md @@ -1,9 +1,8 @@ --- title: Key Commands -menu: - main: - parent: OzoneShell - weight: 3 +summary: Key commands help you to manage the life cycle of + Keys / Objects. +weight: 4 --- + Ozone shell supports the following key commands. * [get](#get) @@ -34,7 +34,7 @@ Ozone shell supports the following key commands. ### Get -The key get command downloads a key from Ozone cluster to local file system. +The `key get` command downloads a key from Ozone cluster to local file system. ***Params:*** @@ -52,7 +52,7 @@ local file sales.orc. ### Put -Uploads a file from the local file system to the specified bucket. +The `key put` command uploads a file from the local file system to the specified bucket. ***Params:*** @@ -61,7 +61,7 @@ Uploads a file from the local file system to the specified bucket. |--------------------------------|-----------------------------------------| | Uri | The name of the key in **/volume/bucket/key** format. | FileName | Local file to upload. -| -r, --replication | Optional, Number of copies, ONE or THREE are the options. Picks up the default from cluster configuration. +| -r, \-\-replication | Optional, Number of copies, ONE or THREE are the options. Picks up the default from cluster configuration. {{< highlight bash >}} ozone sh key put /hive/jan/corrected-sales.orc sales.orc @@ -70,7 +70,7 @@ The above command will put the sales.orc as a new key into _/hive/jan/corrected- ### Delete -The key delete command removes the key from the bucket. +The `key delete` command removes the key from the bucket. ***Params:*** @@ -87,7 +87,8 @@ The above command deletes the key _/hive/jan/corrected-sales.orc_. ### Info -The key info commands returns the information about the key. +The `key info` commands returns the information about the key. + ***Params:*** | Arguments | Comment | @@ -103,15 +104,15 @@ key. ### List -The key list command allows user to list all keys in a bucket. +The `key list` command allows user to list all keys in a bucket. ***Params:*** | Arguments | Comment | |--------------------------------|-----------------------------------------| -| -l, --length | Maximum number of results to return. Default: 1000 -| -p, --prefix | Optional, Only buckets that match this prefix will be returned. -| -s, --start | The listing will start from key after the start key. +| -l, \-\-length | Maximum number of results to return. Default: 1000 +| -p, \-\-prefix | Optional, Only buckets that match this prefix will be returned. +| -s, \-\-start | The listing will start from key after the start key. | Uri | The name of the _volume_. {{< highlight bash >}} @@ -135,10 +136,4 @@ The `key rename` command changes the name of an existing key in the specified bu {{< highlight bash >}} ozone sh key rename /hive/jan sales.orc new_name.orc {{< /highlight >}} -The above command will rename `sales.orc` to `new_name.orc` in the bucket `/hive/jan`. - - - - -You can try out these commands from the docker instance of the [Alpha -Cluster](runningviadocker.html). +The above command will rename _sales.orc_ to _new\_name.orc_ in the bucket _/hive/jan_. diff --git a/hadoop-hdds/docs/content/shell/VolumeCommands.md b/hadoop-hdds/docs/content/shell/VolumeCommands.md new file mode 100644 index 0000000000000..47fb9852b863e --- /dev/null +++ b/hadoop-hdds/docs/content/shell/VolumeCommands.md @@ -0,0 +1,112 @@ +--- +title: Volume Commands +weight: 2 +summary: Volume commands help you to manage the life cycle of a volume. +--- + + +Volume commands generally need administrator privileges. The ozone shell supports the following volume commands. + + * [create](#create) + * [delete](#delete) + * [info](#info) + * [list](#list) + * [update](#update) + +### Create + +The `volume create` command allows an administrator to create a volume and +assign it to a user. + +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| -q, \-\-quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. | +| -u, \-\-user | Required, The name of the user who owns this volume. This user can create, buckets and keys on this volume. | +| Uri | The name of the volume. | + +{{< highlight bash >}} +ozone sh volume create --quota=1TB --user=bilbo /hive +{{< /highlight >}} + +The above command will create a volume called _hive_ on the ozone cluster. This +volume has a quota of 1TB, and the owner is _bilbo_. + +### Delete + +The `volume delete` command allows an administrator to delete a volume. If the +volume is not empty then this command will fail. + +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| Uri | The name of the volume. + +{{< highlight bash >}} +ozone sh volume delete /hive +{{< /highlight >}} + +The above command will delete the volume hive, if the volume has no buckets +inside it. + +### Info + +The `volume info` commands returns the information about the volume including +quota and owner information. + +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| Uri | The name of the volume. + +{{< highlight bash >}} +ozone sh volume info /hive +{{< /highlight >}} + +The above command will print out the information about hive volume. + +### List + +The `volume list` command will list the volumes owned by a user. + +{{< highlight bash >}} +ozone sh volume list --user hadoop +{{< /highlight >}} + +The above command will print out all the volumes owned by the user hadoop. + +### Update + +The volume update command allows changing of owner and quota on a given volume. + +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| -q, \-\-quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. | +| -u, \-\-user | Optional, The name of the user who owns this volume. This user can create, buckets and keys on this volume. | +| Uri | The name of the volume. | + +{{< highlight bash >}} +ozone sh volume update --quota=10TB /hive +{{< /highlight >}} + +The above command updates the volume quota to 10TB. diff --git a/hadoop-hdds/docs/content/shell/_index.md b/hadoop-hdds/docs/content/shell/_index.md new file mode 100644 index 0000000000000..3cb1a9f61672b --- /dev/null +++ b/hadoop-hdds/docs/content/shell/_index.md @@ -0,0 +1,28 @@ +--- +title: Command Line Interface +menu: + main: + weight: 3 +--- + + + +{{}} + Ozone shell is the primary interface to interact with Ozone. + It provides a command shell interface to work against Ozone. +{{}} diff --git a/hadoop-hdds/docs/content/start/FromSource.md b/hadoop-hdds/docs/content/start/FromSource.md new file mode 100644 index 0000000000000..1e920d97cfc1f --- /dev/null +++ b/hadoop-hdds/docs/content/start/FromSource.md @@ -0,0 +1,68 @@ +--- +title: From Source +weight: 30 +--- + + +{{< requirements >}} + * Java 1.8 + * Maven + * Protoc (2.5) +{{< /requirements >}} + + + +If you are a Hadoop ninja, and wise in the ways of Apache, you already know +that a real Apache release is a source release. + +If you want to build from sources, Please untar the source tarball and run +the ozone build command. This instruction assumes that you have all the +dependencies to build Hadoop on your build machine. If you need instructions +on how to build Hadoop, please look at the Apache Hadoop Website. + +```bash +mvn -f pom.ozone.xml clean package -DskipTests=true +``` + +This will build an ozone-\.tar.gz in your `hadoop-ozone/dist/target` directory. + +You can copy this tarball and use this instead of binary artifacts that are +provided along with the official release. + +## How to test the build + +You can run the acceptance tests in the hadoop-ozone directory to make sure +that your build is functional. To launch the acceptance tests, please follow + the instructions in the **README.md** in the `smoketest` directory. + +```bash +cd smoketest +./test.sh +``` + + You can also execute only a minimal subset of the tests: + +```bash +cd smoketest +./test.sh --env ozone basic +``` + +Acceptance tests will start a small ozone cluster and verify that ozone shell and ozone file + system is fully functional. diff --git a/hadoop-hdds/docs/content/start/Kubernetes.md b/hadoop-hdds/docs/content/start/Kubernetes.md new file mode 100644 index 0000000000000..ad855341aa075 --- /dev/null +++ b/hadoop-hdds/docs/content/start/Kubernetes.md @@ -0,0 +1,53 @@ +--- +title: Ozone on Kubernetes +weight: 22 +--- + + + +{{< requirements >}} + * Working kubernetes cluster (LoadBalancer, PersistentVolume are not required) + * kubectl +{{< /requirements >}} + + +As the _apache/ozone_ docker images are available from the dockerhub the deployment process is very similar to Minikube deployment. The only big difference is that we have dedicated set of k8s files for hosted clusters (for example we can use one datanode per host) +Deploy to kubernetes + +`kubernetes/examples` folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases. + +To deploy to a hosted cluster use the ozone subdirectory: + +``` +cd kubernetes/examples/ozone +kubectl apply -f . +``` + +And you can check the results with + +``` +kubectl get pod +Access the services +``` + +Now you can access any of the services. By default the services are not published but you can access them with port-foward rules. + +``` +kubectl port-forward s3g-0 9878:9878 +kubectl port-forward scm-0 9876:9876 +``` diff --git a/hadoop-hdds/docs/content/start/Minikube.md b/hadoop-hdds/docs/content/start/Minikube.md new file mode 100644 index 0000000000000..ebb249d1337d2 --- /dev/null +++ b/hadoop-hdds/docs/content/start/Minikube.md @@ -0,0 +1,70 @@ +--- +title: Minikube & Ozone +weight: 21 +--- + + + +{{< requirements >}} + * Working minikube setup + * kubectl +{{< /requirements >}} + +`kubernetes/examples` folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases. By default the kubernetes resource files are configured to use `apache/ozone` image from the dockerhub. + +To deploy it to minikube use the minikube configuration set: + +``` +cd kubernetes/examples/minikube +kubectl apply -f . +``` + +And you can check the results with + +``` +kubectl get pod +``` + +Note: the kubernetes/examples/minikube resource set is optimized for minikube usage: + + * You can have multiple datanodes even if you have only one host (in a real production cluster usually you need one datanode per physical host) + * The services are published with node port + +## Access the services + +Now you can access any of the services. For each web endpoint an additional NodeType service is defined in the minikube k8s resource set. NodeType services are available via a generated port of any of the host nodes: + +```bash +kubectl get svc +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +datanode ClusterIP None 27s +kubernetes ClusterIP 10.96.0.1 443/TCP 118m +om ClusterIP None 9874/TCP 27s +om-public NodePort 10.108.48.148 9874:32649/TCP 27s +s3g ClusterIP None 9878/TCP 27s +s3g-public NodePort 10.97.133.137 9878:31880/TCP 27s +scm ClusterIP None 9876/TCP 27s +scm-public NodePort 10.105.231.28 9876:32171/TCP 27s +``` + +Minikube contains a convenience command to access any of the NodePort services: + +``` +minikube service s3g-public +Opening kubernetes service default/s3g-public in default browser... +``` \ No newline at end of file diff --git a/hadoop-hdds/docs/content/start/OnPrem.md b/hadoop-hdds/docs/content/start/OnPrem.md new file mode 100644 index 0000000000000..3bf40a6a767fe --- /dev/null +++ b/hadoop-hdds/docs/content/start/OnPrem.md @@ -0,0 +1,187 @@ +--- +title: Ozone On Premise Installation +weight: 20 + +--- + + +If you are feeling adventurous, you can setup ozone in a real cluster. +Setting up a real cluster requires us to understand the components of Ozone. +Ozone is designed to work concurrently with HDFS. However, Ozone is also +capable of running independently. The components of ozone are the same in both approaches. + +## Ozone Components + +1. Ozone Manager - Is the server that is in charge of the namespace of Ozone. Ozone Manager is responsible for all volume, bucket and key operations. +2. Storage Container Manager - Acts as the block manager. Ozone Manager +requests blocks from SCM, to which clients can write data. +3. Datanodes - Ozone data node code runs inside the HDFS datanode or in the independent deployment case runs an ozone datanode daemon. + +## Setting up an Ozone only cluster + +* Please untar the ozone-\ to the directory where you are going +to run Ozone from. We need Ozone jars on all machines in the cluster. So you +need to do this on all machines in the cluster. + +* Ozone relies on a configuration file called ```ozone-site.xml```. To +generate a template that you can replace with proper values, please run the +following command. This will generate a template called ```ozone-site.xml``` at +the specified path (directory). + +{{< highlight bash >}} +ozone genconf +{{< /highlight >}} + +Let us look at the settings inside the generated file (ozone-site.xml) and +how they control ozone. Once the right values are defined, this file +needs to be copied to ```ozone directory/etc/hadoop```. + + +* **ozone.enabled** This is the most critical setting for ozone. +Ozone is a work in progress and users have to enable this service explicitly. +By default, Ozone is disabled. Setting this flag to `true` enables ozone in the +HDFS or Ozone cluster. + +Here is an example, + +{{< highlight xml >}} + + ozone.enabled + true + +{{< /highlight >}} + +* **ozone.metadata.dirs** Allows Administrators to specify where the + metadata must reside. Usually you pick your fastest disk (SSD if + you have them on your nodes). OzoneManager, SCM and datanode will write the + metadata to this path. This is a required setting, if this is missing Ozone + will fail to come up. + + Here is an example, + +{{< highlight xml >}} + + ozone.metadata.dirs + /data/disk1/meta + +{{< /highlight >}} + +* **ozone.scm.names** Storage container manager(SCM) is a distributed block + service which is used by ozone. This property allows data nodes to discover + SCM's address. Data nodes send heartbeat to SCM. + Until HA feature is complete, we configure ozone.scm.names to be a + single machine. + + Here is an example, + + {{< highlight xml >}} + + ozone.scm.names + scm.hadoop.apache.org + + {{< /highlight >}} + + * **ozone.scm.datanode.id.dir** Data nodes generate a Unique ID called Datanode + ID. This identity is written to the file datanode.id in a directory specified by this path. *Data nodes + will create this path if it doesn't exist already.* + +Here is an example, +{{< highlight xml >}} + + ozone.scm.datanode.id.dir + /data/disk1/meta/node + +{{< /highlight >}} + +* **ozone.om.address** OM server address. This is used by OzoneClient and +Ozone File System. + +Here is an example, +{{< highlight xml >}} + + ozone.om.address + ozonemanager.hadoop.apache.org + +{{< /highlight >}} + + +## Ozone Settings Summary + +| Setting | Value | Comment | +|--------------------------------|------------------------------|------------------------------------------------------------------| +| ozone.enabled | true | This enables SCM and containers in HDFS cluster. | +| ozone.metadata.dirs | file path | The metadata will be stored here. | +| ozone.scm.names | SCM server name | Hostname:port or IP:port address of SCM. | +| ozone.scm.block.client.address | SCM server name and port | Used by services like OM | +| ozone.scm.client.address | SCM server name and port | Used by client-side | +| ozone.scm.datanode.address | SCM server name and port | Used by datanode to talk to SCM | +| ozone.om.address | OM server name | Used by Ozone handler and Ozone file system. | + + +## Startup the cluster + +Before we boot up the Ozone cluster, we need to initialize both SCM and Ozone Manager. + +{{< highlight bash >}} +ozone scm --init +{{< /highlight >}} +This allows SCM to create the cluster Identity and initialize its state. +The ```init``` command is similar to Namenode format. Init command is executed only once, that allows SCM to create all the required on-disk structures to work correctly. +{{< highlight bash >}} +ozone --daemon start scm +{{< /highlight >}} + +Once we know SCM is up and running, we can create an Object Store for our use. This is done by running the following command. + +{{< highlight bash >}} +ozone om --init +{{< /highlight >}} + + +Once Ozone manager is initialized, we are ready to run the name service. + +{{< highlight bash >}} +ozone --daemon start om +{{< /highlight >}} + +At this point Ozone's name services, the Ozone manager, and the block service SCM is both running.\ +**Please note**: If SCM is not running +```om --init``` command will fail. SCM start will fail if on-disk data structures are missing. So please make sure you have done both ```scm --init``` and ```om --init``` commands. + +Now we need to start the data nodes. Please run the following command on each datanode. +{{< highlight bash >}} +ozone --daemon start datanode +{{< /highlight >}} + +At this point SCM, Ozone Manager and data nodes are up and running. + +***Congratulations!, You have set up a functional ozone cluster.*** + +## Shortcut + +If you want to make your life simpler, you can just run +{{< highlight bash >}} +ozone scm --init +ozone om --init +start-ozone.sh +{{< /highlight >}} + +This assumes that you have set up the slaves file correctly and ssh +configuration that allows ssh-ing to all data nodes. This is the same as the +HDFS configuration, so please refer to HDFS documentation on how to set this +up. diff --git a/hadoop-hdds/docs/content/RunningViaDocker.md b/hadoop-hdds/docs/content/start/RunningViaDocker.md similarity index 76% rename from hadoop-hdds/docs/content/RunningViaDocker.md rename to hadoop-hdds/docs/content/start/RunningViaDocker.md index e60886f0c8559..9e1e361122538 100644 --- a/hadoop-hdds/docs/content/RunningViaDocker.md +++ b/hadoop-hdds/docs/content/start/RunningViaDocker.md @@ -1,10 +1,7 @@ --- -title: Alpha Cluster -weight: 1 -menu: - main: - parent: Starting - weight: 1 +title: Pseudo-cluster +weight: 23 + --- +{{< requirements >}} + * docker and docker-compose +{{< /requirements >}} -***This is an alpha release of Ozone. Please don't use this release in -production.*** Please check the road map page for features under -development. - -The easiest way to run ozone is to download the release tarball and launch -ozone via Docker. Docker will create a small ozone cluster on your machine, -including the data nodes and ozone services. - -## Running Ozone via Docker - - -**This assumes that you have Docker installed on the machine.** - -* Download the Ozone tarball and untar it. +* Download the Ozone binary tarball and untar it. * Go to the directory where the docker compose files exist and tell `docker-compose` to start Ozone in the background. This will start a small @@ -70,4 +57,5 @@ While you are there, please don't forget to check out the ozone configuration ex To shutdown the cluster, please run {{< highlight bash >}} docker-compose down -{{< /highlight >}} \ No newline at end of file +{{< /highlight >}} + diff --git a/hadoop-hdds/docs/content/start/StartFromDockerHub.md b/hadoop-hdds/docs/content/start/StartFromDockerHub.md new file mode 100644 index 0000000000000..e3e7d41cce68b --- /dev/null +++ b/hadoop-hdds/docs/content/start/StartFromDockerHub.md @@ -0,0 +1,111 @@ +--- +title: Simple Single Ozone +weight: 10 + +--- + + +{{< requirements >}} + * Working docker setup + * AWS CLI (optional) +{{< /requirements >}} + +# Ozone in a Single Container + +The easiest way to start up an all-in-one ozone container is to use the latest +docker image from docker hub: + +```bash +docker run -p 9878:9878 -p 9876:9876 apache/ozone +``` +This command will pull down the ozone image from docker hub and start all +ozone services in a single container.
    +This container will run the required metadata servers (Ozone Manager, Storage +Container Manager) one data node and the S3 compatible REST server +(S3 Gateway). + +# Local multi-container cluster + +If you would like to use a more realistic pseudo-cluster where each components +run in own containers, you can start it with a docker-compose file. + +We have shipped a docker-compose and an enviorment file as part of the +container image that is uploaded to docker hub. + +The following commands can be used to extract these files from the image in the docker hub. +```bash +docker run apache/ozone cat docker-compose.yaml > docker-compose.yaml +docker run apache/ozone cat docker-config > docker-config +``` + + Now you can start the cluster with docker-compose: + +```bash +docker-compose up -d +``` + +If you need multiple datanodes, we can just scale it up: + +```bash + docker-compose scale datanode=3 + ``` +# Running S3 Clients + +Once the cluster is booted up and ready, you can verify its status by +connecting to the SCM's UI at [http://localhost:9876](http://localhost:9876). + +The S3 gateway endpoint will be exposed at port 9878. You can use Ozone's S3 +support as if you are working against the real S3. + + +Here is how you create buckets from command line: + +```bash +aws s3api --endpoint http://localhost:9878/ create-bucket --bucket=bucket1 +``` + +Only notable difference in the above command line is the fact that you have +to tell the _endpoint_ address to the aws s3api command. + +Now let us put a simple file into the S3 Bucket hosted by Ozone. We will +start by creating a temporary file that we can upload to Ozone via S3 support. +```bash +ls -1 > /tmp/testfile + ``` + This command creates a temporary file that + we can upload to Ozone. The next command actually uploads to Ozone's S3 + bucket using the standard aws s3 command line interface. + +```bash +aws s3 --endpoint http://localhost:9878 cp --storage-class REDUCED_REDUNDANCY /tmp/testfile s3://bucket1/testfile +``` + +We can now verify that file got uploaded by running the list command against +our bucket. + +```bash +aws s3 --endpoint http://localhost:9878 ls s3://bucket1/testfile +``` + + +http://localhost:9878/bucket1?browser diff --git a/hadoop-hdds/docs/content/start/_index.md b/hadoop-hdds/docs/content/start/_index.md new file mode 100644 index 0000000000000..5529661b0a21f --- /dev/null +++ b/hadoop-hdds/docs/content/start/_index.md @@ -0,0 +1,88 @@ +--- +title: Getting Started +name: Getting Started +identifier: Starting +menu: main +weight: 1 +cards: "false" +--- + + + +{{}} +There are many ways to install and run Ozone. Starting from simple docker +deployments on +local nodes, to full scale multi-node cluster deployment on +Kubernetes or bare-metal. +{{}} + +
    + +Easy Start + +

    Running Ozone from Docker Hub

    + +You can try out Ozone using docker hub without downloading the official release. This makes it easy to explore Ozone. +
    + {{}} + The simplest and easiest way to start an ozone cluster + to explore what it can do is to start ozone via docker. + {{}} + +
    + +
    + +Recommended + + +

    Running Ozone from an Official Release

    + + Apache Ozone can also be run from the official release packages. Along with the official source releases, we also release a set of convenience binary packages. It is easy to run these binaries in different configurations. +
    + {{}} +Ozone is designed to work concurrently with HDFS. The physical cluster instructions explain each component of Ozone and how to deploy with maximum control. + {{}} + + {{}} +Ozone is designed to work well under Kubernetes. These are instructions to deploy Ozone on K8s. Ozone provides a replicated storage solution for K8s based apps. + {{}} + + {{}} +Ozone comes with a standard set of K8s resources. You can deploy them to MiniKube and experiment with the K8s based deployments. + {{}} + + {{}} + We also ship standard docker files with official release. These are part of official release and not depend upon Docker Hub. + {{}} + +
    + +
    + +Hadoop Ninja + +

    Building From Sources

    + + Instructions to build Ozone from source to create deployment packages. + + {{}} +If you are a Hadoop ninja, and wise in the ways of Apache, you already know that a real Apache release is a source release. We believe that even ninjas need help at times. + {{}} + +
    diff --git a/hadoop-hdds/docs/content/start/docker.png b/hadoop-hdds/docs/content/start/docker.png new file mode 100644 index 0000000000000..048730b23d0c3 Binary files /dev/null and b/hadoop-hdds/docs/content/start/docker.png differ diff --git a/hadoop-hdds/docs/content/start/hadoop.png b/hadoop-hdds/docs/content/start/hadoop.png new file mode 100644 index 0000000000000..183867ca0e96e Binary files /dev/null and b/hadoop-hdds/docs/content/start/hadoop.png differ diff --git a/hadoop-hdds/docs/content/start/k8s.png b/hadoop-hdds/docs/content/start/k8s.png new file mode 100644 index 0000000000000..5fa2e9a90a547 Binary files /dev/null and b/hadoop-hdds/docs/content/start/k8s.png differ diff --git a/hadoop-hdds/docs/content/start/minikube.png b/hadoop-hdds/docs/content/start/minikube.png new file mode 100644 index 0000000000000..0609eccc74fef Binary files /dev/null and b/hadoop-hdds/docs/content/start/minikube.png differ diff --git a/hadoop-hdds/docs/content/AuditParser.md b/hadoop-hdds/docs/content/tools/AuditParser.md similarity index 97% rename from hadoop-hdds/docs/content/AuditParser.md rename to hadoop-hdds/docs/content/tools/AuditParser.md index fa2a0f7f22be3..e4da208ed8515 100644 --- a/hadoop-hdds/docs/content/AuditParser.md +++ b/hadoop-hdds/docs/content/tools/AuditParser.md @@ -1,9 +1,7 @@ --- title: "Audit Parser" date: 2018-12-17 -menu: - main: - parent: Tools +summary: Audit Parser tool can be used for querying the ozone audit logs. --- + +Genconf tool generates a template ozone-site.xml file at the specified path. +This template file can be edited to replace with proper values. + +`ozone genconf ` diff --git a/hadoop-hdds/docs/content/SCMCLI.md b/hadoop-hdds/docs/content/tools/SCMCLI.md similarity index 97% rename from hadoop-hdds/docs/content/SCMCLI.md rename to hadoop-hdds/docs/content/tools/SCMCLI.md index bd6086c796e2a..04950c2c05685 100644 --- a/hadoop-hdds/docs/content/SCMCLI.md +++ b/hadoop-hdds/docs/content/tools/SCMCLI.md @@ -1,9 +1,7 @@ --- title: "SCMCLI" date: 2017-08-10 -menu: - main: - parent: Tools +summary: Admin tool for managing SCM --- + +Testing is one of the most important part during the development of a distributed system. We have the following type of test. + +This page includes our existing test tool which are part of the Ozone source base. + +Note: we have more tests (like TCP-DS, TCP-H tests via Spark or Hive) which are not included here because they use external tools only. + +## Unit test + +As every almost every java project we have the good old unit tests inside each of our projects. + +## Integration test (JUnit) + +Traditional unit tests are supposed to test only one unit, but we also have higher level unit tests. They use `MiniOzoneCluster` which is a helper method to start real daemons (scm,om,datanodes) during the unit test. + +From maven/java point of view they are just simple unit tests (JUnit library is used) but to separate them (and solve some dependency problems) we moved all of these tests to `hadoop-ozone/integration-test` + +## Smoketest + +We use docker-compose based pseudo-cluster to run different configuration of Ozone. To be sure that the different configuration can be started we implemented _acceptance_ tests with the help of https://robotframework.org/. + +The smoketests are available from the distribution (`./smoketest`) but the robot files defines only the tests: usually they start CLI and check the output. + +To run the tests in different environmente (docker-compose, kubernetes) you need a definition to start the containers and execute the right tests in the right containers. + +These definition of the tests are included in the `compose` directory (check `./compose/*/test.sh` or `./compose/test-all.sh`). + +For example a simple way to test the distribution packege: + +``` +cd compose/ozonze +./test.sh +``` + +## Blockade + +[Blockade](https://github.com/worstcase/blockade) is a tool to test network failures and partitions (it's inspired by the legendary [Jepsen tests](https://jepsen.io/analyses)). + +Blockade tests are implemented with the help of tests and can be started from the `./blockade` directory of the distrubution. + +``` +cd blocakde +pip install pytest==2.8.7,blockade +python -m pytest -s . +``` + +See the README in the blockade directory for more details. + +## MiniChaosOzoneCluster + +This is a way to get [chaos](https://en.wikipedia.org/wiki/Chaos_engineering) in your machine. It can be started from the source code and a MiniOzoneCluster (which starts real daemons) will be started and killed randomly. + +## Freon + +Freon is a command line application which is included in the Ozone distribution. It's a load generator which is used in our stress tests. + +For example: + +``` +ozone freon randomkeys --numOfVolumes=10 --numOfBuckets 10 --numOfKeys 10 --replicationType=RATIS --factor=THREE +``` + +``` +*************************************************** +Status: Success +Git Base Revision: 48aae081e5afacbb3240657556b26c29e61830c3 +Number of Volumes created: 10 +Number of Buckets created: 100 +Number of Keys added: 1000 +Ratis replication factor: THREE +Ratis replication type: RATIS +Average Time spent in volume creation: 00:00:00,035 +Average Time spent in bucket creation: 00:00:00,319 +Average Time spent in key creation: 00:00:03,659 +Average Time spent in key write: 00:00:10,894 +Total bytes written: 10240000 +Total Execution time: 00:00:16,898 +*********************** +``` + +For more information check the [documentation page](https://hadoop.apache.org/ozone/docs/0.4.0-alpha/freon.html) + +## Genesis + +Genesis is a microbenchmarking tool. It's also included in the distribution (`ozone genesis`) but it doesn't require real cluster. It measures different part of the code in an isolated way (eg. the code which saves the data to the local RocksDB based key value stores) + +Example run: + +``` + ozone genesis -benchmark=BenchMarkRocksDbStore +# JMH version: 1.19 +# VM version: JDK 11.0.1, VM 11.0.1+13-LTS +# VM invoker: /usr/lib/jvm/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/bin/java +# VM options: -Dproc_genesis -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender +# Warmup: 2 iterations, 1 s each +# Measurement: 20 iterations, 1 s each +# Timeout: 10 min per iteration +# Threads: 4 threads, will synchronize iterations +# Benchmark mode: Throughput, ops/time +# Benchmark: org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test +# Parameters: (backgroundThreads = 4, blockSize = 8, maxBackgroundFlushes = 4, maxBytesForLevelBase = 512, maxOpenFiles = 5000, maxWriteBufferNumber = 16, writeBufferSize = 64) + +# Run progress: 0.00% complete, ETA 00:00:22 +# Fork: 1 of 1 +# Warmup Iteration 1: 213775.360 ops/s +# Warmup Iteration 2: 32041.633 ops/s +Iteration 1: 196342.348 ops/s + ?stack: + +Iteration 2: 41926.816 ops/s + ?stack: + +Iteration 3: 210433.231 ops/s + ?stack: + +Iteration 4: 46941.951 ops/s + ?stack: + +Iteration 5: 212825.884 ops/s + ?stack: + +Iteration 6: 145914.351 ops/s + ?stack: + +Iteration 7: 141838.469 ops/s + ?stack: + +Iteration 8: 205334.438 ops/s + ?stack: + +Iteration 9: 163709.519 ops/s + ?stack: + +Iteration 10: 162494.608 ops/s + ?stack: + +Iteration 11: 199155.793 ops/s + ?stack: + +Iteration 12: 209679.298 ops/s + ?stack: + +Iteration 13: 193787.574 ops/s + ?stack: + +Iteration 14: 127004.147 ops/s + ?stack: + +Iteration 15: 145511.080 ops/s + ?stack: + +Iteration 16: 223433.864 ops/s + ?stack: + +Iteration 17: 169752.665 ops/s + ?stack: + +Iteration 18: 165217.191 ops/s + ?stack: + +Iteration 19: 191038.476 ops/s + ?stack: + +Iteration 20: 196335.579 ops/s + ?stack: + + + +Result "org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test": + 167433.864 ?(99.9%) 43530.883 ops/s [Average] + (min, avg, max) = (41926.816, 167433.864, 223433.864), stdev = 50130.230 + CI (99.9%): [123902.981, 210964.748] (assumes normal distribution) + +Secondary result "org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test:?stack": +Stack profiler: + +....[Thread state distributions].................................................................... + 78.9% RUNNABLE + 20.0% TIMED_WAITING + 1.1% WAITING + +....[Thread state: RUNNABLE]........................................................................ + 59.8% 75.8% org.rocksdb.RocksDB.put + 16.5% 20.9% org.rocksdb.RocksDB.get + 0.7% 0.9% java.io.UnixFileSystem.delete0 + 0.7% 0.9% org.rocksdb.RocksDB.disposeInternal + 0.3% 0.4% java.lang.Long.formatUnsignedLong0 + 0.1% 0.2% org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test + 0.1% 0.1% java.lang.Long.toUnsignedString0 + 0.1% 0.1% org.apache.hadoop.ozone.genesis.generated.BenchMarkRocksDbStore_test_jmhTest.test_thrpt_jmhStub + 0.0% 0.1% java.lang.Object.clone + 0.0% 0.0% java.lang.Thread.currentThread + 0.4% 0.5% + +....[Thread state: TIMED_WAITING]................................................................... + 20.0% 100.0% java.lang.Object.wait + +....[Thread state: WAITING]......................................................................... + 1.1% 100.0% jdk.internal.misc.Unsafe.park + + + +# Run complete. Total time: 00:00:38 + +Benchmark (backgroundThreads) (blockSize) (maxBackgroundFlushes) (maxBytesForLevelBase) (maxOpenFiles) (maxWriteBufferNumber) (writeBufferSize) Mode Cnt Score Error Units +BenchMarkRocksDbStore.test 4 8 4 512 5000 16 64 thrpt 20 167433.864 ? 43530.883 ops/s +BenchMarkRocksDbStore.test:?stack 4 8 4 512 5000 16 64 thrpt NaN --- +``` diff --git a/hadoop-hdds/docs/content/tools/_index.md b/hadoop-hdds/docs/content/tools/_index.md new file mode 100644 index 0000000000000..d7c92704ba339 --- /dev/null +++ b/hadoop-hdds/docs/content/tools/_index.md @@ -0,0 +1,65 @@ +--- +title: "Tools" +date: "2017-10-10" +summary: Ozone supports a set of tools that are handy for developers.Here is a quick list of command line tools. +menu: + main: + weight: 8 +--- + + + +Ozone has a set of command line tools that can be used to manage ozone. + +All these commands are invoked via the ```ozone``` script. + +Daemon commands: + + * **scm** - Storage Container Manager service, via daemon can be started + or stopped. + * **om** - Ozone Manager, via daemon command can be started or stopped. + * **datanode** - Via daemon command, the HDDS data nodes can be started or + stopped. + * **s3g** - + +Client commands: + + * **sh** - Primary command line interface for ozone to manage volumes/buckets/keys. + * **fs** - Runs a command on ozone file system (similar to `hdfs dfs`) + * **version** - Prints the version of Ozone and HDDS. + + +Admin commands: + + * **classpath** - Prints the class path needed to get the hadoop jar and the + required libraries. + * **dtutil** - Operations related to delegation tokens + * **envvars** - Display computed Hadoop environment variables. + * **getconf** - Reads ozone config values from configuration. + * **jmxget** - Get JMX exported values from NameNode or DataNode. + * **scmcli** - Developer only, Command Line Interface for the Storage + Container Manager. + * **genconf** - Generate minimally required ozone configs and output to + ozone-site.xml. + +Test tools: + + * **freon** - Runs the ozone load generator. + * **genesis** - Developer Only, Ozone micro-benchmark application. + + For more information see the following subpages: \ No newline at end of file diff --git a/hadoop-hdds/docs/pom.xml b/hadoop-hdds/docs/pom.xml index edb4665091ea8..6c6d77f3677fa 100644 --- a/hadoop-hdds/docs/pom.xml +++ b/hadoop-hdds/docs/pom.xml @@ -15,7 +15,7 @@ +https://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 org.apache.hadoop @@ -55,7 +55,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd"> themes/ozonedoc/static/js/bootstrap.min.js - themes/ozonedoc/static/js/jquery.min.js + themes/ozonedoc/static/js/jquery-3.4.1.min.js themes/ozonedoc/static/css/bootstrap-theme.min.css themes/ozonedoc/static/css/bootstrap.min.css.map diff --git a/hadoop-hdds/docs/static/ozone-logo.png b/hadoop-hdds/docs/static/ozone-logo-small.png similarity index 100% rename from hadoop-hdds/docs/static/ozone-logo.png rename to hadoop-hdds/docs/static/ozone-logo-small.png diff --git a/hadoop-hdds/docs/static/ozone-usage.png b/hadoop-hdds/docs/static/ozone-usage.png new file mode 100644 index 0000000000000..adcbdcf6e7b9b Binary files /dev/null and b/hadoop-hdds/docs/static/ozone-usage.png differ diff --git a/hadoop-hdds/docs/themes/ozonedoc/layouts/_default/section.html b/hadoop-hdds/docs/themes/ozonedoc/layouts/_default/section.html new file mode 100644 index 0000000000000..5c01241501de8 --- /dev/null +++ b/hadoop-hdds/docs/themes/ozonedoc/layouts/_default/section.html @@ -0,0 +1,71 @@ + +{{ partial "header.html" . }} + + + +{{ partial "navbar.html" . }} + +
    +
    + {{ partial "sidebar.html" . }} +
    +
    +

    {{ .Title }}

    +
    +
    + {{ .Content }} + {{.Params.card}} + {{ if not (eq .Params.cards "false")}} + {{ range $page_index, $page_val := .Pages }} + + {{ $page_count := len .Pages }} + {{if (eq (mod $page_index 2) 0)}} +
    + {{end}} +
    +
    +
    +

    + {{ with .Params.Icon}} + + {{end}} + {{ .LinkTitle }} +

    +

    {{.Summary}}

    + {{.LinkTitle}} +
    +
    +
    + + {{if (or (eq (mod $page_index 2) 1) (eq $page_index (sub $page_count 1)))}} +
    + {{end}} + {{ end }} + {{end}} +
    +
    +
    +
    + +{{ partial "footer.html" . }} + + + + \ No newline at end of file diff --git a/hadoop-hdds/docs/themes/ozonedoc/layouts/_default/single.html b/hadoop-hdds/docs/themes/ozonedoc/layouts/_default/single.html index ca2e1c4a65e00..3679ddbe2c053 100644 --- a/hadoop-hdds/docs/themes/ozonedoc/layouts/_default/single.html +++ b/hadoop-hdds/docs/themes/ozonedoc/layouts/_default/single.html @@ -16,23 +16,42 @@ --> {{ partial "header.html" . }} - - -{{ partial "navbar.html" . }} - -
    -
    - {{ partial "sidebar.html" . }} -
    -

    {{ .Title }}

    -
    - {{ .Content }} -
    + + + {{ partial "navbar.html" . }} + +
    +
    + {{ partial "sidebar.html" . }} +
    + + + +
    + + +
    +

    {{.Title}}

    +
    + + {{ .Content }} + + {{ with .PrevInSection }} + Next >> + {{ end }}
    +
    + + {{ partial "footer.html" . }} -{{ partial "footer.html" . }} + - - + \ No newline at end of file diff --git a/hadoop-hdds/docs/themes/ozonedoc/layouts/index.html b/hadoop-hdds/docs/themes/ozonedoc/layouts/index.html index 17f0246abf403..045c69224cf73 100644 --- a/hadoop-hdds/docs/themes/ozonedoc/layouts/index.html +++ b/hadoop-hdds/docs/themes/ozonedoc/layouts/index.html @@ -23,7 +23,7 @@
    {{ partial "sidebar.html" . }} -
    +
    {{ .Content }} diff --git a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/footer.html b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/footer.html index 5aaeed9e1ed4a..0e5ca0fec2b1d 100644 --- a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/footer.html +++ b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/footer.html @@ -17,6 +17,6 @@ - + diff --git a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/header.html b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/header.html index 35ba4c8042f94..a4e24c9513304 100644 --- a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/header.html +++ b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/header.html @@ -26,9 +26,9 @@ Documentation for Apache Hadoop Ozone - + - + diff --git a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html index 316f2cc11cc7c..0f26571fad5ea 100644 --- a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html +++ b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html @@ -23,7 +23,12 @@ - + + + + Hadoop Ozone