Merge

.github/pull_request_template.md

@@ -0,0 +1,6 @@
+## NOTICE
+
+Please create an issue in ASF JIRA before opening a pull request,
+and you need to set the title of the pull request which starts with
+the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
+For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

BUILDING.txt

@@ -6,7 +6,7 @@ Requirements:
 * Unix System
 * JDK 1.8
 * Maven 3.3 or later
-* ProtocolBuffer 2.5.0
+* Protocol Buffers 3.7.1 (if compiling native code)
 * CMake 3.1 or newer (if compiling native code)
 * Zlib devel (if compiling native code)
 * Cyrus SASL devel (if compiling native code)
@@ -62,8 +62,16 @@ Installing required packages for clean install of Ubuntu 14.04 LTS Desktop:
   $ sudo apt-get -y install maven
 * Native libraries
   $ sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev libsasl2-dev
-* ProtocolBuffer 2.5.0 (required)
-  $ sudo apt-get -y install protobuf-compiler
+* Protocol Buffers 3.7.1 (required to build native code)
+  $ mkdir -p /opt/protobuf-3.7-src \
+        && curl -L -s -S \
+          https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \
+          -o /opt/protobuf-3.7.1.tar.gz \
+        && tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C /opt/protobuf-3.7-src \
+        && cd /opt/protobuf-3.7-src \
+        && ./configure\
+        && make install \
+        && rm -rf /opt/protobuf-3.7-src
 
 Optional packages:
 
@@ -78,6 +86,8 @@ Optional packages:
   $ sudo apt-get install fuse libfuse-dev
 * ZStandard compression
     $ sudo apt-get install zstd
+* PMDK library for storage class memory(SCM) as HDFS cache backend
+  Please refer to http://pmem.io/ and https://github.com/pmem/pmdk
 
 ----------------------------------------------------------------------------------
 Maven main modules:
@@ -262,6 +272,32 @@ Maven build goals:
    invoke, run 'mvn dependency-check:aggregate'. Note that this plugin
    requires maven 3.1.1 or greater.
 
+ PMDK library build options:
+
+   The Persistent Memory Development Kit (PMDK), formerly known as NVML, is a growing
+   collection of libraries which have been developed for various use cases, tuned,
+   validated to production quality, and thoroughly documented. These libraries are built
+   on the Direct Access (DAX) feature available in both Linux and Windows, which allows
+   applications directly load/store access to persistent memory by memory-mapping files
+   on a persistent memory aware file system.
+
+   It is currently an optional component, meaning that Hadoop can be built without
+   this dependency. Please Note the library is used via dynamic module. For getting
+   more details please refer to the official sites:
+   http://pmem.io/ and https://github.com/pmem/pmdk.
+
+  * -Drequire.pmdk is used to build the project with PMDK libraries forcibly. With this
+    option provided, the build will fail if libpmem library is not found. If this option
+    is not given, the build will generate a version of Hadoop with libhadoop.so.
+    And storage class memory(SCM) backed HDFS cache is still supported without PMDK involved.
+    Because PMDK can bring better caching write/read performance, it is recommended to build
+    the project with this option if user plans to use SCM backed HDFS cache.
+  * -Dpmdk.lib is used to specify a nonstandard location for PMDK libraries if they are not
+    under /usr/lib or /usr/lib64.
+  * -Dbundle.pmdk is used to copy the specified libpmem libraries into the distribution tar
+    package. This option requires that -Dpmdk.lib is specified. With -Dbundle.pmdk provided,
+    the build will fail if -Dpmdk.lib is not specified.
+
 ----------------------------------------------------------------------------------
 Building components separately
 
@@ -274,16 +310,6 @@ level once; and then work from the submodule. Keep in mind that SNAPSHOTs
 time out after a while, using the Maven '-nsu' will stop Maven from trying
 to update SNAPSHOTs from external repos.
 
-----------------------------------------------------------------------------------
-Protocol Buffer compiler
-
-The version of Protocol Buffer compiler, protoc, must match the version of the
-protobuf JAR.
-
-If you have multiple versions of protoc in your system, you can set in your
-build shell the HADOOP_PROTOC_PATH environment variable to point to the one you
-want to use for the Hadoop build. If you don't define this environment variable,
-protoc is looked up in the PATH.
 ----------------------------------------------------------------------------------
 Importing projects to eclipse
 
@@ -352,6 +378,49 @@ export MAVEN_OPTS="-Xms256m -Xmx1536m"
 
 ----------------------------------------------------------------------------------
 
+Building on macOS (without Docker)
+
+----------------------------------------------------------------------------------
+Installing required dependencies for clean install of macOS 10.14:
+
+* Install Xcode Command Line Tools
+  $ xcode-select --install
+* Install Homebrew
+  $ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
+* Install OpenJDK 8
+  $ brew tap AdoptOpenJDK/openjdk
+  $ brew cask install adoptopenjdk8
+* Install maven and tools
+  $ brew install maven autoconf automake cmake wget
+* Install native libraries, only openssl is required to compile native code,
+you may optionally install zlib, lz4, etc.
+  $ brew install openssl
+* Protocol Buffers 3.7.1 (required to compile native code)
+  $ wget https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz
+  $ mkdir -p protobuf-3.7 && tar zxvf protobuf-java-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7
+  $ cd protobuf-3.7
+  $ ./configure
+  $ make
+  $ make check
+  $ make install
+  $ protoc --version
+
+Note that building Hadoop 3.1.1/3.1.2/3.2.0 native code from source is broken
+on macOS. For 3.1.1/3.1.2, you need to manually backport YARN-8622. For 3.2.0,
+you need to backport both YARN-8622 and YARN-9487 in order to build native code.
+
+----------------------------------------------------------------------------------
+Building command example:
+
+* Create binary distribution with native code but without documentation:
+  $ mvn package -Pdist,native -DskipTests -Dmaven.javadoc.skip \
+    -Dopenssl.prefix=/usr/local/opt/openssl
+
+Note that the command above manually specified the openssl library and include
+path. This is necessary at least for Homebrewed OpenSSL.
+
+----------------------------------------------------------------------------------
+
 Building on Windows
 
 ----------------------------------------------------------------------------------
@@ -360,7 +429,7 @@ Requirements:
 * Windows System
 * JDK 1.8
 * Maven 3.0 or later
-* ProtocolBuffer 2.5.0
+* Protocol Buffers 3.7.1
 * CMake 3.1 or newer
 * Visual Studio 2010 Professional or Higher
 * Windows SDK 8.1 (if building CPU rate control for the container executor)