From 9cdd2aff1f422f753c224c6e9254129d1d6e3696 Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Tue, 16 Jun 2015 00:29:54 -0500 Subject: [PATCH 01/27] HBASE-13915 remove EOL versions from config prereq section (cherry picked from commit 51b606cd185437802f0a7a4620f1434e8e2d9c74) Conflicts: src/main/asciidoc/_chapters/configuration.adoc --- src/main/asciidoc/_chapters/configuration.adoc | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 5e7d16d5e54c..4fe0d84a0e07 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -116,11 +116,6 @@ deprecated `remove()` method of the `PoolMap` class and is under consideration. link:https://issues.apache.org/jira/browse/HBASE-7608[HBASE-7608] for more information about JDK 8 support. -|0.96 -|yes -|yes -|N/A - |0.94 |yes |yes From 7d96b7ed5f4a9aba56de91097ce82c3dac181069 Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Tue, 16 Jun 2015 01:22:36 -0500 Subject: [PATCH 02/27] HBASE-13911 update java/hadoop prereqs for 1.2 (cherry picked from commit 348a11ad55dc93c40b7f6e3ff8cb71d2e7c6162c) Conflicts: src/main/asciidoc/_chapters/configuration.adoc --- src/main/asciidoc/_chapters/configuration.adoc | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 4fe0d84a0e07..f1fc79ca63e6 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -98,6 +98,11 @@ This section lists required services and some required system configuration. |JDK 7 |JDK 8 +|1.2 +|link:http://search-hadoop.com/m/DHED4Zlz0R1[Not Supported] +|yes +|yes + |1.1 |link:http://search-hadoop.com/m/DHED4Zlz0R1[Not Supported] |yes From e4a911bc443d5dc61e82447eed262072f639f8cc Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Mon, 31 Aug 2015 23:03:53 -0500 Subject: [PATCH 03/27] HBASE-13221 List Hadoop 2.6 as unsupported due to HADOOP-11710. (cherry picked from commit e54cb8abebb2971da68852c6dcef456dbc689afb) Conflicts: src/main/asciidoc/_chapters/configuration.adoc --- src/main/asciidoc/_chapters/configuration.adoc | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index f1fc79ca63e6..03b81ef34bf5 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -226,6 +226,21 @@ Use the following legend to interpret this table: |Hadoop-3.0.0 | NT | NT | NT | NT | NT |=== +.Hadoop 2.6.x +[TIP] +==== +Hadoop distributions based on the 2.6.x line *must* have +link:https://issues.apache.org/jira/browse/HADOOP-11710[HADOOP-11710] applied if you plan to run +HBase on top of an HDFS Encryption Zone. Failure to do so will result in cluster failure and +data loss. +==== + +.Hadoop 2.7.x +[TIP] +==== +Hadoop version 2.7.0 is not tested or supported as the Hadoop PMC has explicitly labeled that release as not being stable. +==== + .Replace the Hadoop Bundled With HBase! [NOTE] ==== From 162f54472754a23cc48188f52c87c478a2e25757 Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Tue, 29 Dec 2015 11:46:17 -0600 Subject: [PATCH 04/27] HBASE-15007 update docs to list Hadoop 2.6.1+ as cool. (cherry picked from commit 5e2c2e1780d57f2e0086b10cd28d5212d6a3d5d0) Conflicts: src/main/asciidoc/_chapters/configuration.adoc --- src/main/asciidoc/_chapters/configuration.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 03b81ef34bf5..b3f843fe6a3b 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -232,7 +232,7 @@ Use the following legend to interpret this table: Hadoop distributions based on the 2.6.x line *must* have link:https://issues.apache.org/jira/browse/HADOOP-11710[HADOOP-11710] applied if you plan to run HBase on top of an HDFS Encryption Zone. Failure to do so will result in cluster failure and -data loss. +data loss. This patch is present in Apache Hadoop releases 2.6.1+. ==== .Hadoop 2.7.x From e91e9751a24f6c99b190506d8ab872e1d196b6ca Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Thu, 28 Jan 2016 09:54:21 -0800 Subject: [PATCH 05/27] HBASE-14810 Update Hadoop support description to explain "not tested" vs "not supported" (cherry picked from commit 9cd487129d5a0048216ff00ef15fdb8effc525ae) --- .../asciidoc/_chapters/configuration.adoc | 4 ++- .../asciidoc/_chapters/getting_started.adoc | 1 + src/main/asciidoc/_chapters/preface.adoc | 35 +++++++++++++++++++ src/main/asciidoc/_chapters/upgrading.adoc | 2 +- 4 files changed, 40 insertions(+), 2 deletions(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index b3f843fe6a3b..3c2864f10d7d 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -28,7 +28,9 @@ :experimental: This chapter expands upon the <> chapter to further explain configuration of Apache HBase. -Please read this chapter carefully, especially the <> to ensure that your HBase testing and deployment goes smoothly, and prevent data loss. +Please read this chapter carefully, especially the <> +to ensure that your HBase testing and deployment goes smoothly, and prevent data loss. +Familiarize yourself with <> as well. == Configuration Files Apache HBase uses the same configuration system as Apache Hadoop. diff --git a/src/main/asciidoc/_chapters/getting_started.adoc b/src/main/asciidoc/_chapters/getting_started.adoc index 374c9463b080..324ece36defb 100644 --- a/src/main/asciidoc/_chapters/getting_started.adoc +++ b/src/main/asciidoc/_chapters/getting_started.adoc @@ -19,6 +19,7 @@ */ //// +[[getting_started]] = Getting Started :doctype: book :numbered: diff --git a/src/main/asciidoc/_chapters/preface.adoc b/src/main/asciidoc/_chapters/preface.adoc index 960fcc4a5c8e..859211d70bbb 100644 --- a/src/main/asciidoc/_chapters/preface.adoc +++ b/src/main/asciidoc/_chapters/preface.adoc @@ -61,4 +61,39 @@ Please use link:https://issues.apache.org/jira/browse/hbase[JIRA] to report non- To protect existing HBase installations from new vulnerabilities, please *do not* use JIRA to report security-related bugs. Instead, send your report to the mailing list private@apache.org, which allows anyone to send messages, but restricts who can read them. Someone on that list will contact you to follow up on your report. +[hbase_supported_tested_definitions] +.Support and Testing Expectations + +The phrases /supported/, /not supported/, /tested/, and /not tested/ occur several +places throughout this guide. In the interest of clarity, here is a brief explanation +of what is generally meant by these phrases, in the context of HBase. + +NOTE: Commercial technical support for Apache HBase is provided by many Hadoop vendors. +This is not the sense in which the term /support/ is used in the context of the +Apache HBase project. The Apache HBase team assumes no responsibility for your +HBase clusters, your configuration, or your data. + +Supported:: + In the context of Apache HBase, /supported/ means that HBase is designed to work + in the way described, and deviation from the defined behavior or functionality should + be reported as a bug. + +Not Supported:: + In the context of Apache HBase, /not supported/ means that a use case or use pattern + is not expected to work and should be considered an antipattern. If you think this + designation should be reconsidered for a given feature or use pattern, file a JIRA + or start a discussion on one of the mailing lists. + +Tested:: + In the context of Apache HBase, /tested/ means that a feature is covered by unit + or integration tests, and has been proven to work as expected. + +Not Tested:: + In the context of Apache HBase, /not tested/ means that a feature or use pattern + may or may notwork in a given way, and may or may not corrupt your data or cause + operational issues. It is an unknown, and there are no guarantees. If you can provide + proof that a feature designated as /not tested/ does work in a given way, please + submit the tests and/or the metrics so that other users can gain certainty about + such features or use patterns. + :numbered: diff --git a/src/main/asciidoc/_chapters/upgrading.adoc b/src/main/asciidoc/_chapters/upgrading.adoc index 1cb9bfdbe5a6..47c80a12018d 100644 --- a/src/main/asciidoc/_chapters/upgrading.adoc +++ b/src/main/asciidoc/_chapters/upgrading.adoc @@ -31,7 +31,7 @@ You cannot skip major versions when upgrading. If you are upgrading from version NOTE: It may be possible to skip across versions -- for example go from 0.92.2 straight to 0.98.0 just following the 0.96.x upgrade instructions -- but these scenarios are untested. -Review <>, in particular <>. +Review <>, in particular <>. Familiarize yourself with <>. [[hbase.versioning]] == HBase version number and compatibility From 59925b7cc08e395cd6a5cf6a1a13fa581d8c9ab8 Mon Sep 17 00:00:00 2001 From: Jerry He Date: Sat, 21 Jan 2017 13:24:33 -0800 Subject: [PATCH 06/27] HBASE-17502 Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support matrix (cherry picked from commit 9a9e3df8565a8ba36779a442907a38772ab59a72) --- src/main/asciidoc/_chapters/configuration.adoc | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 3c2864f10d7d..977afa5ee810 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -228,6 +228,15 @@ Use the following legend to interpret this table: |Hadoop-3.0.0 | NT | NT | NT | NT | NT |=== +.Hadoop Pre-2.6.1 and JDK 1.8 Kerberos +[TIP] +==== +When using pre-2.6.1 Hadoop versions and JDK 1.8 in a Kerberos environment, HBase server can fail +and abort due to Kerberos keytab relogin error. Late version of JDK 1.7 (1.7.0_80) has the problem too. +Refer to link:https://issues.apache.org/jira/browse/HADOOP-10786[HADOOP-10786] for additional details. +Consider upgrading to Hadoop 2.6.1+ in this case. +==== + .Hadoop 2.6.x [TIP] ==== From c0f0283ba2a6c8500a2dab4e716292a9a39b6344 Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Wed, 29 Mar 2017 08:38:07 -0500 Subject: [PATCH 07/27] HBASE-17847 update docs to cover guidance on recent Hadoop releases. Signed-off-by: Michael Stack (cherry picked from commit b290d14e1f7fffa3c06bb19770b53094c3d2459a) Conflicts: src/main/asciidoc/_chapters/configuration.adoc --- src/main/asciidoc/_chapters/configuration.adoc | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 977afa5ee810..4581eb2fbe28 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -202,6 +202,8 @@ See link:http://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Suppor Hadoop 2.x is faster and includes features, such as short-circuit reads, which will help improve your HBase random read profile. Hadoop 2.x also includes important bug fixes that will improve your overall HBase experience. HBase 0.98 drops support for Hadoop 1.0, deprecates use of Hadoop 1.1+, and HBase 1.0 will not support Hadoop 1.x. + +Hadoop 3.x is still in early access releases and has not yet been sufficiently tested by the HBase community for production use cases. ==== Use the following legend to interpret this table: @@ -249,7 +251,13 @@ data loss. This patch is present in Apache Hadoop releases 2.6.1+. .Hadoop 2.7.x [TIP] ==== -Hadoop version 2.7.0 is not tested or supported as the Hadoop PMC has explicitly labeled that release as not being stable. +Hadoop version 2.7.0 is not tested or supported as the Hadoop PMC has explicitly labeled that release as not being stable. (reference the link:https://s.apache.org/hadoop-2.7.0-announcement[announcement of Apache Hadoop 2.7.0].) +==== + +.Hadoop 2.8.x +[TIP] +==== +Hadoop version 2.8.0 is not tested or supported as the Hadoop PMC has explicitly labeled that release as not being stable. (reference the link:https://s.apache.org/hadoop-2.8.0-announcement[announcement of Apache Hadoop 2.8.0].) ==== .Replace the Hadoop Bundled With HBase! From d31062b63110a10fb05ba268e8766b8b9717c19e Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Wed, 12 Apr 2017 09:04:44 -0500 Subject: [PATCH 08/27] HBASE-16215 clean up of ref guide and site for EOM versions. Signed-off-by: Enis Soztutar Signed-off-by: Michael Stack (cherry picked from commit a8e6f33791d787eaf9c1bfe63aa7f3266f25268b) Conflicts: src/main/asciidoc/_chapters/community.adoc src/main/asciidoc/_chapters/configuration.adoc src/main/asciidoc/_chapters/cp.adoc src/main/asciidoc/_chapters/developer.adoc src/main/site/site.xml --- src/main/asciidoc/_chapters/community.adoc | 11 +- .../asciidoc/_chapters/configuration.adoc | 148 +++--------------- src/main/asciidoc/_chapters/developer.adoc | 7 +- src/main/asciidoc/_chapters/upgrading.adoc | 6 +- src/main/site/site.xml | 17 +- 5 files changed, 43 insertions(+), 146 deletions(-) diff --git a/src/main/asciidoc/_chapters/community.adoc b/src/main/asciidoc/_chapters/community.adoc index 4b91b0d6f3ce..8f0e3479387c 100644 --- a/src/main/asciidoc/_chapters/community.adoc +++ b/src/main/asciidoc/_chapters/community.adoc @@ -62,13 +62,12 @@ Any -1 on a patch by anyone vetos a patch; it cannot be committed until the just .How to set fix version in JIRA on issue resolve Here is how link:http://search-hadoop.com/m/azemIi5RCJ1[we agreed] to set versions in JIRA when we resolve an issue. -If trunk is going to be 0.98.0 then: +If master is going to be 2.0.0, and branch-1 1.4.0 then: -* Commit only to trunk: Mark with 0.98 -* Commit to 0.95 and trunk : Mark with 0.98, and 0.95.x -* Commit to 0.94.x and 0.95, and trunk: Mark with 0.98, 0.95.x, and 0.94.x -* Commit to 89-fb: Mark with 89-fb. -* Commit site fixes: no version +* Commit only to master: Mark with 2.0.0 +* Commit to branch-1 and master: Mark with 2.0.0, and 1.4.0 +* Commit to branch-1.3, branch-1, and master: Mark with 2.0.0, 1.4.0, and 1.3.x +* Commit site fixes: no version [[hbase.when.to.close.jira]] .Policy on when to set a RESOLVED JIRA as CLOSED diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 4581eb2fbe28..c0b1089dc6de 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -93,43 +93,33 @@ This section lists required services and some required system configuration. [[java]] .Java -[cols="1,1,1,4", options="header"] +[cols="1,1,4", options="header"] |=== |HBase Version -|JDK 6 |JDK 7 |JDK 8 -|1.2 -|link:http://search-hadoop.com/m/DHED4Zlz0R1[Not Supported] -|yes +|2.0 +|link:http://search-hadoop.com/m/YGbbsPxZ723m3as[Not Supported] |yes -|1.1 -|link:http://search-hadoop.com/m/DHED4Zlz0R1[Not Supported] +|1.3 |yes -|Running with JDK 8 will work but is not well tested. - -|1.0 -|link:http://search-hadoop.com/m/DHED4Zlz0R1[Not Supported] |yes -|Running with JDK 8 will work but is not well tested. -|0.98 +|1.2 |yes |yes -|Running with JDK 8 works but is not well tested. Building with JDK 8 would require removal of the -deprecated `remove()` method of the `PoolMap` class and is under consideration. See -link:https://issues.apache.org/jira/browse/HBASE-7608[HBASE-7608] for more information about JDK 8 -support. -|0.94 -|yes +|1.1 |yes -|N/A +|Running with JDK 8 will work but is not well tested. + |=== -NOTE: In HBase 0.98.5 and newer, you must set `JAVA_HOME` on each node of your cluster. _hbase-env.sh_ provides a handy mechanism to do this. +NOTE: HBase will neither build nor compile with Java 6. + +NOTE: You must set `JAVA_HOME` on each node of your cluster. _hbase-env.sh_ provides a handy mechanism to do this. .Operating System Utilities ssh:: @@ -200,8 +190,8 @@ See link:http://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Suppor [TIP] ==== Hadoop 2.x is faster and includes features, such as short-circuit reads, which will help improve your HBase random read profile. -Hadoop 2.x also includes important bug fixes that will improve your overall HBase experience. -HBase 0.98 drops support for Hadoop 1.0, deprecates use of Hadoop 1.1+, and HBase 1.0 will not support Hadoop 1.x. +Hadoop 2.x also includes important bug fixes that will improve your overall HBase experience. HBase does not support running with +earlier versions of Hadoop. See the table below for requirements specific to different HBase versions. Hadoop 3.x is still in early access releases and has not yet been sufficiently tested by the HBase community for production use cases. ==== @@ -271,107 +261,6 @@ Make sure you replace the jar in HBase everywhere on your cluster. Hadoop version mismatch issues have various manifestations but often all looks like its hung up. ==== -[[hadoop2.hbase_0.94]] -==== Apache HBase 0.94 with Hadoop 2 - -To get 0.94.x to run on Hadoop 2.2.0, you need to change the hadoop 2 and protobuf versions in the _pom.xml_: Here is a diff with pom.xml changes: - -[source] ----- -$ svn diff pom.xml -Index: pom.xml -=================================================================== ---- pom.xml (revision 1545157) -+++ pom.xml (working copy) -@@ -1034,7 +1034,7 @@ - 1.4.3 - 1.2.16 - 1.8.5 -- 2.4.0a -+ 2.5.0 - 1.0.1 - 0.8.0 - 3.4.5 -@@ -2241,7 +2241,7 @@ - - - -- 2.0.0-alpha -+ 2.2.0 - 1.6.1 - - ----- - -The next step is to regenerate Protobuf files and assuming that the Protobuf has been installed: - -* Go to the HBase root folder, using the command line; -* Type the following commands: -+ - -[source,bourne] ----- -$ protoc -Isrc/main/protobuf --java_out=src/main/java src/main/protobuf/hbase.proto ----- -+ - -[source,bourne] ----- -$ protoc -Isrc/main/protobuf --java_out=src/main/java src/main/protobuf/ErrorHandling.proto ----- - - -Building against the hadoop 2 profile by running something like the following command: - ----- -$ mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests ----- - -[[hadoop.hbase_0.94]] -==== Apache HBase 0.92 and 0.94 - -HBase 0.92 and 0.94 versions can work with Hadoop versions, 0.20.205, 0.22.x, 1.0.x, and 1.1.x. -HBase-0.94 can additionally work with Hadoop-0.23.x and 2.x, but you may have to recompile the code using the specific maven profile (see top level pom.xml) - -[[hadoop.hbase_0.96]] -==== Apache HBase 0.96 - -As of Apache HBase 0.96.x, Apache Hadoop 1.0.x at least is required. -Hadoop 2 is strongly encouraged (faster but also has fixes that help MTTR). We will no longer run properly on older Hadoops such as 0.20.205 or branch-0.20-append. -Do not move to Apache HBase 0.96.x if you cannot upgrade your Hadoop. See link:http://search-hadoop.com/m/7vFVx4EsUb2[HBase, mail # dev - DISCUSS: - Have hbase require at least hadoop 1.0.0 in hbase 0.96.0?] - -[[hadoop.older.versions]] -==== Hadoop versions 0.20.x - 1.x - -HBase will lose data unless it is running on an HDFS that has a durable `sync` implementation. -DO NOT use Hadoop 0.20.2, Hadoop 0.20.203.0, and Hadoop 0.20.204.0 which DO NOT have this attribute. -Currently only Hadoop versions 0.20.205.x or any release in excess of this version -- this includes hadoop-1.0.0 -- have a working, durable sync. -The Cloudera blog post link:http://www.cloudera.com/blog/2012/01/an-update-on-apache-hadoop-1-0/[An - update on Apache Hadoop 1.0] by Charles Zedlweski has a nice exposition on how all the Hadoop versions relate. -It's worth checking out if you are having trouble making sense of the Hadoop version morass. - -Sync has to be explicitly enabled by setting `dfs.support.append` equal to true on both the client side -- in _hbase-site.xml_ -- and on the serverside in _hdfs-site.xml_ (The sync facility HBase needs is a subset of the append code path). - -[source,xml] ----- - - - dfs.support.append - true - ----- - -You will have to restart your cluster after making this edit. -Ignore the chicken-little comment you'll find in the _hdfs-default.xml_ in the description for the `dfs.support.append` configuration. - -[[hadoop.security]] -==== Apache HBase on Secure Hadoop - -Apache HBase will run on any Hadoop 0.20.x that incorporates Hadoop security features as long as you do as suggested above and replace the Hadoop jar that ships with HBase with the secure version. -If you want to read more about how to setup Secure HBase, see <>. - - [[dfs.datanode.max.transfer.threads]] ==== `dfs.datanode.max.transfer.threads` (((dfs.datanode.max.transfer.threads))) @@ -404,9 +293,10 @@ See also <> a [[zookeeper.requirements]] === ZooKeeper Requirements -ZooKeeper 3.4.x is required as of HBase 1.0.0. -HBase makes use of the `multi` functionality that is only available since 3.4.0 (The `useMulti` configuration option defaults to `true` in HBase 1.0.0). -See link:https://issues.apache.org/jira/browse/HBASE-12241[HBASE-12241 (The crash of regionServer when taking deadserver's replication queue breaks replication)] and link:https://issues.apache.org/jira/browse/HBASE-6775[HBASE-6775 (Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix)] for background. +ZooKeeper 3.4.x is required. +HBase makes use of the `multi` functionality that is only available since Zookeeper 3.4.0. The `hbase.zookeeper.useMulti` configuration property defaults to `true`. +Refer to link:https://issues.apache.org/jira/browse/HBASE-12241[HBASE-12241 (The crash of regionServer when taking deadserver's replication queue breaks replication)] and link:https://issues.apache.org/jira/browse/HBASE-6775[HBASE-6775 (Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix)] for background. +The property is deprecated and useMulti is always enabled in HBase 2.0. [[standalone_dist]] == HBase run modes: Standalone and Distributed @@ -560,8 +450,6 @@ Check them out especially if HBase had trouble starting. HBase also puts up a UI listing vital attributes. By default it's deployed on the Master host at port 16010 (HBase RegionServers listen on port 16020 by default and put up an informational HTTP server at port 16030). If the Master is running on a host named `master.example.org` on the default port, point your browser at _http://master.example.org:16010_ to see the web interface. -Prior to HBase 0.98 the master UI was deployed on port 60010, and the HBase RegionServers UI on port 60030. - Once HBase has started, see the <> section for how to create tables, add data, scan your insertions, and finally disable and drop your tables. To stop HBase after exiting the HBase shell enter @@ -767,7 +655,7 @@ example9 [[hbase_env]] ==== _hbase-env.sh_ -The following lines in the _hbase-env.sh_ file show how to set the `JAVA_HOME` environment variable (required for HBase 0.98.5 and newer) and set the heap to 4 GB (rather than the default value of 1 GB). If you copy and paste this example, be sure to adjust the `JAVA_HOME` to suit your environment. +The following lines in the _hbase-env.sh_ file show how to set the `JAVA_HOME` environment variable (required for HBase) and set the heap to 4 GB (rather than the default value of 1 GB). If you copy and paste this example, be sure to adjust the `JAVA_HOME` to suit your environment. ---- # The java implementation to use. diff --git a/src/main/asciidoc/_chapters/developer.adoc b/src/main/asciidoc/_chapters/developer.adoc index 4d8e99b159fb..98611ad6236b 100644 --- a/src/main/asciidoc/_chapters/developer.adoc +++ b/src/main/asciidoc/_chapters/developer.adoc @@ -538,7 +538,12 @@ For the build to sign them for you, you a properly configured _settings.xml_ in [[maven.release]] === Making a Release Candidate -Only committers may make releases of hbase artifacts. + +NOTE: These instructions are for building HBase 1.y.z + +.Point Releases +If you are making a point release (for example to quickly address a critical incompatibility or security problem) off of a release branch instead of a development branch, the tagging instructions are slightly different. +I'll prefix those special steps with _Point Release Only_. .Before You Begin Make sure your environment is properly set up. Maven and Git are the main tooling diff --git a/src/main/asciidoc/_chapters/upgrading.adoc b/src/main/asciidoc/_chapters/upgrading.adoc index 47c80a12018d..3e4f5e44e810 100644 --- a/src/main/asciidoc/_chapters/upgrading.adoc +++ b/src/main/asciidoc/_chapters/upgrading.adoc @@ -150,6 +150,9 @@ HBase Private API:: [[hbase.versioning.pre10]] === Pre 1.0 versions +.HBase Pre-1.0 versions are all EOM +NOTE: For new installations, do not deploy 0.94.y, 0.96.y, or 0.98.y. Deploy our stable version. See link:https://issues.apache.org/jira/browse/HBASE-11642[EOL 0.96], link:https://issues.apache.org/jira/browse/HBASE-16215[clean up of EOM releases], and link:http://www.apache.org/dist/hbase/[the header of our downloads]. + Before the semantic versioning scheme pre-1.0, HBase tracked either Hadoop's versions (0.2x) or 0.9x versions. If you are into the arcane, checkout our old wiki page on link:http://wiki.apache.org/hadoop/Hbase/HBaseVersions[HBase Versioning] which tries to connect the HBase version dots. Below sections cover ONLY the releases before 1.0. [[hbase.development.series]] @@ -263,9 +266,6 @@ A rolling upgrade from 0.94.x directly to 0.98.x does not work. The upgrade path ==== The "Singularity" -.HBase 0.96.x was EOL'd, September 1st, 2014 -NOTE: Do not deploy 0.96.x Deploy at least 0.98.x. See link:https://issues.apache.org/jira/browse/HBASE-11642[EOL 0.96]. - You will have to stop your old 0.94.x cluster completely to upgrade. If you are replicating between clusters, both clusters will have to go down to upgrade. Make sure it is a clean shutdown. The less WAL files around, the faster the upgrade will run (the upgrade will split any log files it finds in the filesystem as part of the upgrade process). All clients must be upgraded to 0.96 too. The API has changed. You will need to recompile your code against 0.96 and you may need to adjust applications to go against new APIs (TODO: List of changes). diff --git a/src/main/site/site.xml b/src/main/site/site.xml index b7debd35c50b..736ab7a3d9d9 100644 --- a/src/main/site/site.xml +++ b/src/main/site/site.xml @@ -77,12 +77,17 @@ - - - - - - + + + + + + + + + + + From 408246139871699c21356402d1e49a68d008b7f1 Mon Sep 17 00:00:00 2001 From: Chia-Ping Tsai Date: Mon, 11 Sep 2017 01:26:38 +0800 Subject: [PATCH 09/27] HBASE-18421 update hadoop prerequisites docs to call out 2.8.1 (cherry picked from commit b12d071b24cfe3cdcc3ad11e66ede466335667e6) Conflicts: src/main/asciidoc/_chapters/configuration.adoc --- src/main/asciidoc/_chapters/configuration.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index c0b1089dc6de..39d0d3cc2488 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -247,7 +247,7 @@ Hadoop version 2.7.0 is not tested or supported as the Hadoop PMC has explicitly .Hadoop 2.8.x [TIP] ==== -Hadoop version 2.8.0 is not tested or supported as the Hadoop PMC has explicitly labeled that release as not being stable. (reference the link:https://s.apache.org/hadoop-2.8.0-announcement[announcement of Apache Hadoop 2.8.0].) +Hadoop version 2.8.0 and 2.8.1 are not tested or supported as the Hadoop PMC has explicitly labeled that releases as not being stable. (reference the link:https://s.apache.org/hadoop-2.8.0-announcement[announcement of Apache Hadoop 2.8.0] and link:https://s.apache.org/hadoop-2.8.1-announcement[announcement of Apache Hadoop 2.8.1].) ==== .Replace the Hadoop Bundled With HBase! From fa6be26e1fde192cdb573ebf831db6ff2a6c60c8 Mon Sep 17 00:00:00 2001 From: Apekshit Sharma Date: Sun, 17 Dec 2017 19:10:04 -0800 Subject: [PATCH 10/27] HBASE-19526 Update hadoop version to 3.0 GA (cherry picked from commit 2a5e5e9e2c0e1a2f8cdb67783c78ba29695604b9) Conflicts: dev-support/hbase-personality.sh hbase-resource-bundle/src/main/resources/supplemental-models.xml pom.xml src/main/asciidoc/_chapters/configuration.adoc --- .../src/main/resources/supplemental-models.xml | 18 +++++++++--------- src/main/asciidoc/_chapters/configuration.adoc | 15 +++++++++++++++ src/main/asciidoc/_chapters/developer.adoc | 2 +- 3 files changed, 25 insertions(+), 10 deletions(-) diff --git a/hbase-resource-bundle/src/main/resources/supplemental-models.xml b/hbase-resource-bundle/src/main/resources/supplemental-models.xml index 165e7589e118..f3c8ec4c71a5 100644 --- a/hbase-resource-bundle/src/main/resources/supplemental-models.xml +++ b/hbase-resource-bundle/src/main/resources/supplemental-models.xml @@ -1390,10 +1390,10 @@ Copyright 2006 Envoi Solutions LLC - - com.nimbusds - nimbus-jose-jwt - 3.9 + + com.codahale.metrics + metrics-core + 3.0.1 Apache License, Version 2.0 @@ -1404,7 +1404,7 @@ Copyright 2006 Envoi Solutions LLC - + com.squareup.okhttp okhttp 2.4.0 @@ -1418,7 +1418,7 @@ Copyright 2006 Envoi Solutions LLC - + com.squareup.okio okio 1.4.0 @@ -1461,7 +1461,7 @@ Copyright 2006 Envoi Solutions LLC - + de.ruedigermoeller fst 2.24 @@ -1480,7 +1480,7 @@ Copyright 2006 Envoi Solutions LLC - + org.objenesis objenesis 2.1 @@ -1631,7 +1631,7 @@ EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - + org.codehaus.woodstox stax2-api 3.1.4 diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 39d0d3cc2488..6ed32e3815e2 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -316,6 +316,21 @@ Standalone mode is what is described in the <> section. In standalone mode, HBase does not use HDFS -- it uses the local filesystem instead -- and it runs all HBase daemons and a local ZooKeeper all up in the same JVM. Zookeeper binds to a well known port so clients may talk to HBase. +[source,xml] +---- + + + hbase.rootdir + hdfs://namenode.example.org:8020/hbase + + + hbase.cluster.distributed + false + + +---- + +[[distributed]] === Distributed Distributed mode can be subdivided into distributed but all daemons run on a single node -- a.k.a _pseudo-distributed_ -- and _fully-distributed_ where the daemons are spread across all nodes in the cluster. diff --git a/src/main/asciidoc/_chapters/developer.adoc b/src/main/asciidoc/_chapters/developer.adoc index 98611ad6236b..f88e31f8325f 100644 --- a/src/main/asciidoc/_chapters/developer.adoc +++ b/src/main/asciidoc/_chapters/developer.adoc @@ -401,7 +401,7 @@ mvn -Dhadoop.profile=3.0 ... The above will build against whatever explicit hadoop 3.y version we have in our _pom.xml_ as our '3.0' version. Tests may not all pass so you may need to pass `-DskipTests` unless you are inclined to fix the failing tests. -To pick a particular Hadoop 3.y release, you'd set e.g. `-Dhadoop-three.version=3.0.0-alpha1`. +To pick a particular Hadoop 3.y release, you'd set hadoop-three.version property e.g. `-Dhadoop-three.version=3.0.0`. [[build.protobuf]] ==== Build Protobuf From 6b656862aa63e9d5c0d6e386829cb9aad3b4c0c1 Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Sat, 24 Feb 2018 13:13:40 -0600 Subject: [PATCH 11/27] HBASE-20072 remove 1.1 release line from the prerequisite tables. Signed-off-by: Ted Yu (cherry picked from commit 7889df37118a0593dfb0156c6edf1bb96d4c94b9) Conflicts: src/main/asciidoc/_chapters/configuration.adoc --- src/main/asciidoc/_chapters/configuration.adoc | 4 ---- 1 file changed, 4 deletions(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 6ed32e3815e2..cf9212f2eba1 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -111,10 +111,6 @@ This section lists required services and some required system configuration. |yes |yes -|1.1 -|yes -|Running with JDK 8 will work but is not well tested. - |=== NOTE: HBase will neither build nor compile with Java 6. From 003568e43565926b2250faf96cf00c837f438c53 Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Fri, 23 Mar 2018 11:53:04 -0500 Subject: [PATCH 12/27] HBASE-19983 update Hadoop support matrix for Hadoop releases 2.8.2+ and 2.9.0. Signed-off-by: Mike Drob (cherry picked from commit 65bd0881e25981a41373ba70a5929dc15cf1c037) Conflicts: src/main/asciidoc/_chapters/configuration.adoc --- src/main/asciidoc/_chapters/configuration.adoc | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index cf9212f2eba1..fe4785d8c4b5 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -234,16 +234,10 @@ HBase on top of an HDFS Encryption Zone. Failure to do so will result in cluster data loss. This patch is present in Apache Hadoop releases 2.6.1+. ==== -.Hadoop 2.7.x +.Hadoop 2.y.0 Releases [TIP] ==== -Hadoop version 2.7.0 is not tested or supported as the Hadoop PMC has explicitly labeled that release as not being stable. (reference the link:https://s.apache.org/hadoop-2.7.0-announcement[announcement of Apache Hadoop 2.7.0].) -==== - -.Hadoop 2.8.x -[TIP] -==== -Hadoop version 2.8.0 and 2.8.1 are not tested or supported as the Hadoop PMC has explicitly labeled that releases as not being stable. (reference the link:https://s.apache.org/hadoop-2.8.0-announcement[announcement of Apache Hadoop 2.8.0] and link:https://s.apache.org/hadoop-2.8.1-announcement[announcement of Apache Hadoop 2.8.1].) +Starting around the time of Hadoop version 2.7.0, the Hadoop PMC got into the habit of calling out new minor releases on their major version 2 release line as not stable / production ready. As such, HBase expressly advises downstream users to avoid running on top of these releases. Note that additionally the 2.8.1 was release was given the same caveat by the Hadoop PMC. For reference, see the release announcements for link:https://s.apache.org/hadoop-2.7.0-announcement[Apache Hadoop 2.7.0], link:https://s.apache.org/hadoop-2.8.0-announcement[Apache Hadoop 2.8.0], link:https://s.apache.org/hadoop-2.8.1-announcement[Apache Hadoop 2.8.1], and link:https://s.apache.org/hadoop-2.9.0-announcement[Apache Hadoop 2.9.0]. ==== .Replace the Hadoop Bundled With HBase! From 9ac71e0c11479850e48ed7f63cb12c638a8fb97f Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Fri, 23 Mar 2018 08:48:28 -0500 Subject: [PATCH 13/27] HBASE-20264 add Javas 9 and 10 to the prerequisites table and add a note about using LTS releases. * Make the #java anchor point at a section instead of directly at a table * Add a note to the intro of that section about LTS JDKs * Add columns for JDK9 and JDK10 that say unsupported and point to HBASE-20264 Signed-off-by: Zach York (cherry picked from commit 9ea1a7d422080fab2e7621ad5629322dc01de1f7) --- .../asciidoc/_chapters/configuration.adoc | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index fe4785d8c4b5..4c024e108c17 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -93,23 +93,41 @@ This section lists required services and some required system configuration. [[java]] .Java -[cols="1,1,4", options="header"] + +The following table summarizes the recommendation of the HBase community wrt deploying on various Java versions. An entry of "yes" is meant to indicate a base level of testing and willingness to help diagnose and address issues you might run into. Similarly, an entry of "no" or "Not Supported" generally means that should you run into an issue the community is likely to ask you to change the Java environment before proceeding to help. In some cases, specific guidance on limitations (e.g. wether compiling / unit tests work, specific operational issues, etc) will also be noted. + +.Long Term Support JDKs are recommended +[TIP] +==== +HBase recommends downstream users rely on JDK releases that are marked as Long Term Supported (LTS) either from the OpenJDK project or vendors. As of March 2018 that means Java 8 is the only applicable version and that the next likely version to see testing will be Java 11 near Q3 2018. +==== + +.Java support by release line +[cols="1,1,1,1,1", options="header"] |=== |HBase Version |JDK 7 |JDK 8 +|JDK 9 +|JDK 10 |2.0 |link:http://search-hadoop.com/m/YGbbsPxZ723m3as[Not Supported] |yes +|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] +|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] |1.3 |yes |yes +|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] +|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] |1.2 |yes |yes +|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] +|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] |=== From 0014bb8edce7dcfa4a9f6efc8db04df273884ce6 Mon Sep 17 00:00:00 2001 From: Mike Drob Date: Thu, 22 Mar 2018 15:05:30 -0500 Subject: [PATCH 14/27] HBASE-20260 Remove old content from book (cherry picked from commit 6a5c14b2278ba972e4f25ff5a4b7e028121f79dd) Conflicts: src/main/asciidoc/_chapters/architecture.adoc src/main/asciidoc/_chapters/backup_restore.adoc src/main/asciidoc/_chapters/configuration.adoc src/main/asciidoc/_chapters/getting_started.adoc src/main/asciidoc/_chapters/troubleshooting.adoc src/main/asciidoc/_chapters/upgrading.adoc --- src/main/asciidoc/_chapters/architecture.adoc | 17 +- src/main/asciidoc/_chapters/compression.adoc | 2 +- .../asciidoc/_chapters/configuration.adoc | 29 +- .../asciidoc/_chapters/getting_started.adoc | 29 -- src/main/asciidoc/_chapters/ops_mgt.adoc | 39 +-- .../asciidoc/_chapters/troubleshooting.adoc | 109 +------ src/main/asciidoc/_chapters/upgrading.adoc | 268 +----------------- 7 files changed, 59 insertions(+), 434 deletions(-) diff --git a/src/main/asciidoc/_chapters/architecture.adoc b/src/main/asciidoc/_chapters/architecture.adoc index 7309410dedd0..5287801ec641 100644 --- a/src/main/asciidoc/_chapters/architecture.adoc +++ b/src/main/asciidoc/_chapters/architecture.adoc @@ -1449,10 +1449,19 @@ Alphanumeric Rowkeys:: Using a Custom Algorithm:: The RegionSplitter tool is provided with HBase, and uses a _SplitAlgorithm_ to determine split points for you. As parameters, you give it the algorithm, desired number of regions, and column families. - It includes two split algorithms. - The first is the `link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/util/RegionSplitter.HexStringSplit.html[HexStringSplit]` algorithm, which assumes the row keys are hexadecimal strings. - The second, `link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/util/RegionSplitter.UniformSplit.html[UniformSplit]`, assumes the row keys are random byte arrays. - You will probably need to develop your own `link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/util/RegionSplitter.SplitAlgorithm.html[SplitAlgorithm]`, using the provided ones as models. + It includes three split algorithms. + The first is the + `link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/RegionSplitter.HexStringSplit.html[HexStringSplit]` + algorithm, which assumes the row keys are hexadecimal strings. + The second is the + `link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/RegionSplitter.DecimalStringSplit.html[DecimalStringSplit]` + algorithm, which assumes the row keys are decimal strings in the range 00000000 to 99999999. + The third, + `link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/RegionSplitter.UniformSplit.html[UniformSplit]`, + assumes the row keys are random byte arrays. + You will probably need to develop your own + `link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/RegionSplitter.SplitAlgorithm.html[SplitAlgorithm]`, + using the provided ones as models. === Online Region Merges diff --git a/src/main/asciidoc/_chapters/compression.adoc b/src/main/asciidoc/_chapters/compression.adoc index 78fc6a2f1efb..8853d9b12d8a 100644 --- a/src/main/asciidoc/_chapters/compression.adoc +++ b/src/main/asciidoc/_chapters/compression.adoc @@ -414,7 +414,7 @@ $ hbase org.apache.hadoop.hbase.util.LoadTestTool -write 1:10:100 -num_keys 1000 ==== [[data.block.encoding.enable]] -== Enable Data Block Encoding +=== Enable Data Block Encoding Codecs are built into HBase so no extra configuration is needed. Codecs are enabled on a table by setting the `DATA_BLOCK_ENCODING` property. diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 4c024e108c17..3e8f0c916bac 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -131,7 +131,7 @@ HBase recommends downstream users rely on JDK releases that are marked as Long T |=== -NOTE: HBase will neither build nor compile with Java 6. +NOTE: HBase will neither build nor run with Java 6. NOTE: You must set `JAVA_HOME` on each node of your cluster. _hbase-env.sh_ provides a handy mechanism to do this. @@ -140,11 +140,7 @@ ssh:: HBase uses the Secure Shell (ssh) command and utilities extensively to communicate between cluster nodes. Each server in the cluster must be running `ssh` so that the Hadoop and HBase daemons can be managed. You must be able to connect to all nodes via SSH, including the local node, from the Master as well as any backup Master, using a shared key rather than a password. You can see the basic methodology for such a set-up in Linux or Unix systems at "<>". If your cluster nodes use OS X, see the section, link:http://wiki.apache.org/hadoop/Running_Hadoop_On_OS_X_10.5_64-bit_%28Single-Node_Cluster%29[SSH: Setting up Remote Desktop and Enabling Self-Login] on the Hadoop wiki. DNS:: - HBase uses the local hostname to self-report its IP address. Both forward and reverse DNS resolving must work in versions of HBase previous to 0.92.0. The link:https://github.com/sujee/hadoop-dns-checker[hadoop-dns-checker] tool can be used to verify DNS is working correctly on the cluster. The project `README` file provides detailed instructions on usage. - -Loopback IP:: - Prior to hbase-0.96.0, HBase only used the IP address `127.0.0.1` to refer to `localhost`, and this could not be configured. - See <> for more details. + HBase uses the local hostname to self-report its IP address. NTP:: The clocks on cluster nodes should be synchronized. A small amount of variation is acceptable, but larger amounts of skew can cause erratic and unexpected behavior. Time synchronization is one of the first things to check if you see unexplained problems in your cluster. It is recommended that you run a Network Time Protocol (NTP) service, or another time-synchronization mechanism, on your cluster, and that all nodes look to the same service for time synchronization. See the link:http://www.tldp.org/LDP/sag/html/basic-ntp-config.html[Basic NTP Configuration] at [citetitle]_The Linux Documentation Project (TLDP)_ to set up NTP. @@ -187,14 +183,14 @@ Linux Shell:: All of the shell scripts that come with HBase rely on the link:http://www.gnu.org/software/bash[GNU Bash] shell. Windows:: - Prior to HBase 0.96, testing for running HBase on Microsoft Windows was limited. - Running a on Windows nodes is not recommended for production systems. + Running production systems on Windows machines is not recommended. [[hadoop]] === link:http://hadoop.apache.org[Hadoop](((Hadoop))) -The following table summarizes the versions of Hadoop supported with each version of HBase. +The following table summarizes the versions of Hadoop supported with each version of HBase. Older versions not appearing in this table are considered unsupported and likely missing necessary features, while newer versions are untested but may be suitable. + Based on the version of HBase, you should select the most appropriate version of Hadoop. You can use Apache Hadoop, or a vendor's distribution of Hadoop. No distinction is made here. @@ -302,9 +298,6 @@ See also <> a === ZooKeeper Requirements ZooKeeper 3.4.x is required. -HBase makes use of the `multi` functionality that is only available since Zookeeper 3.4.0. The `hbase.zookeeper.useMulti` configuration property defaults to `true`. -Refer to link:https://issues.apache.org/jira/browse/HBASE-12241[HBASE-12241 (The crash of regionServer when taking deadserver's replication queue breaks replication)] and link:https://issues.apache.org/jira/browse/HBASE-6775[HBASE-6775 (Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix)] for background. -The property is deprecated and useMulti is always enabled in HBase 2.0. [[standalone_dist]] == HBase run modes: Standalone and Distributed @@ -801,7 +794,6 @@ For most use patterns, most of the time, you should use automatic splitting. See <> for more information about manual region splitting. Instead of allowing HBase to split your regions automatically, you can choose to manage the splitting yourself. -This feature was added in HBase 0.90.0. Manually managing splits works if you know your keyspace well, otherwise let HBase figure where to split for you. Manual splitting can mitigate region creation and movement under load. It also makes it so region boundaries are known and invariant (if you disable region splitting). If you use manual splits, it is easier doing staggered, time-based major compactions to spread out your network IO load. @@ -827,13 +819,12 @@ Otherwise, the cluster can be prone to compaction storms where a large number of It is important to understand that the data growth causes compaction storms, and not the manual split decision. If the regions are split into too many large regions, you can increase the major compaction interval by configuring `HConstants.MAJOR_COMPACTION_PERIOD`. -HBase 0.90 introduced `org.apache.hadoop.hbase.util.RegionSplitter`, which provides a network-IO-safe rolling split of all regions. +The `org.apache.hadoop.hbase.util.RegionSplitter` utility also provides a network-IO-safe rolling split of all regions. [[managed.compactions]] ==== Managed Compactions By default, major compactions are scheduled to run once in a 7-day period. -Prior to HBase 0.96.x, major compactions were scheduled to happen once per day by default. If you need to control exactly when and how often major compaction runs, you can disable managed major compactions. See the entry for `hbase.hregion.majorcompaction` in the <> table for details. @@ -953,8 +944,8 @@ To enable monitoring and management from remote systems, you need to set system See the link:http://docs.oracle.com/javase/6/docs/technotes/guides/management/agent.html[official documentation] for more information. Historically, besides above port mentioned, JMX opens two additional random TCP listening ports, which could lead to port conflict problem. (See link:https://issues.apache.org/jira/browse/HBASE-10289[HBASE-10289] for details) -As an alternative, You can use the coprocessor-based JMX implementation provided by HBase. -To enable it in 0.99 or above, add below property in _hbase-site.xml_: +As an alternative, you can use the coprocessor-based JMX implementation provided by HBase. +To enable it, add below property in _hbase-site.xml_: [source,xml] ---- @@ -1049,8 +1040,8 @@ The corresponding properties for port configuration are `master.rmi.registry.por [[dyn_config]] == Dynamic Configuration -Since HBase 1.0.0, it is possible to change a subset of the configuration without requiring a server restart. -In the HBase shell, there are new operators, `update_config` and `update_all_config` that will prompt a server or all servers to reload configuration. +It is possible to change a subset of the configuration without requiring a server restart. +In the HBase shell, the operations `update_config` and `update_all_config` will prompt a server or all servers to reload configuration. Only a subset of all configurations can currently be changed in the running server. Here is an incomplete list: `hbase.regionserver.thread.compaction.large`, `hbase.regionserver.thread.compaction.small`, `hbase.regionserver.thread.split`, `hbase.regionserver.thread.merge`, as well as compaction policy and configurations and adjustment to offpeak hours. diff --git a/src/main/asciidoc/_chapters/getting_started.adoc b/src/main/asciidoc/_chapters/getting_started.adoc index 324ece36defb..986c0ffce013 100644 --- a/src/main/asciidoc/_chapters/getting_started.adoc +++ b/src/main/asciidoc/_chapters/getting_started.adoc @@ -39,35 +39,6 @@ This is not an appropriate configuration for a production instance of HBase, but This section shows you how to create a table in HBase using the `hbase shell` CLI, insert rows into the table, perform put and scan operations against the table, enable or disable the table, and start and stop HBase. Apart from downloading HBase, this procedure should take less than 10 minutes. -.Local Filesystem and Durability -WARNING: _The following is fixed in HBase 0.98.3 and beyond. See link:https://issues.apache.org/jira/browse/HBASE-11272[HBASE-11272] and link:https://issues.apache.org/jira/browse/HBASE-11218[HBASE-11218]._ - -Using HBase with a local filesystem does not guarantee durability. -The HDFS local filesystem implementation will lose edits if files are not properly closed. -This is very likely to happen when you are experimenting with new software, starting and stopping the daemons often and not always cleanly. -You need to run HBase on HDFS to ensure all writes are preserved. -Running against the local filesystem is intended as a shortcut to get you familiar with how the general system works, as the very first phase of evaluation. -See link:https://issues.apache.org/jira/browse/HBASE-3696[HBASE-3696] and its associated issues for more details about the issues of running on the local filesystem. - -[[loopback.ip]] -.Loopback IP - HBase 0.94.x and earlier -NOTE: _The below advice is for hbase-0.94.x and older versions only. This is fixed in hbase-0.96.0 and beyond._ - -Prior to HBase 0.94.x, HBase expected the loopback IP address to be 127.0.0.1. Ubuntu and some other distributions default to 127.0.1.1 and this will cause problems for you. See link:http://devving.com/?p=414[Why does HBase care about /etc/hosts?] for detail - - -.Example /etc/hosts File for Ubuntu -==== -The following _/etc/hosts_ file works correctly for HBase 0.94.x and earlier, on Ubuntu. Use this as a template if you run into trouble. -[listing] ----- -127.0.0.1 localhost -127.0.0.1 ubuntu.ubuntu-domain ubuntu ----- - -==== - - === JDK Version Requirements HBase requires that a JDK be installed. diff --git a/src/main/asciidoc/_chapters/ops_mgt.adoc b/src/main/asciidoc/_chapters/ops_mgt.adoc index 722357e3c445..f9edebbe45e1 100644 --- a/src/main/asciidoc/_chapters/ops_mgt.adoc +++ b/src/main/asciidoc/_chapters/ops_mgt.adoc @@ -44,17 +44,18 @@ Some commands, such as `version`, `pe`, `ltt`, `clean`, are not available in pre $ bin/hbase Usage: hbase [] [] Options: - --config DIR Configuration direction to use. Default: ./conf - --hosts HOSTS Override the list in 'regionservers' file + --config DIR Configuration direction to use. Default: ./conf + --hosts HOSTS Override the list in 'regionservers' file + --auth-as-server Authenticate to ZooKeeper using servers configuration Commands: Some commands take arguments. Pass no args or -h for usage. shell Run the HBase shell hbck Run the hbase 'fsck' tool + snapshot Tool for managing snapshots wal Write-ahead-log analyzer hfile Store file analyzer zkcli Run the ZooKeeper shell - upgrade Upgrade hbase master Run an HBase HMaster node regionserver Run an HBase HRegionServer node zookeeper Run a Zookeeper server @@ -66,6 +67,7 @@ Some commands take arguments. Pass no args or -h for usage. mapredcp Dump CLASSPATH entries required by mapreduce pe Run PerformanceEvaluation ltt Run LoadTestTool + canary Run the Canary tool version Print the version CLASSNAME Run the class named CLASSNAME ---- @@ -81,20 +83,27 @@ To see the usage, use the `--help` parameter. ---- $ ${HBASE_HOME}/bin/hbase canary -help -Usage: bin/hbase org.apache.hadoop.hbase.tool.Canary [opts] [table1 [table2]...] | [regionserver1 [regionserver2]..] +Usage: hbase org.apache.hadoop.hbase.tool.Canary [opts] [table1 [table2]...] | [regionserver1 [regionserver2]..] where [opts] are: -help Show this help and exit. -regionserver replace the table argument to regionserver, which means to enable regionserver mode + -allRegions Tries all regions on a regionserver, + only works in regionserver mode. + -zookeeper Tries to grab zookeeper.znode.parent + on each zookeeper instance -daemon Continuous check at defined intervals. -interval Interval between checks (sec) - -e Use region/regionserver as regular expression - which means the region/regionserver is regular expression pattern + -e Use table/regionserver as regular expression + which means the table/regionserver is regular expression pattern -f stop whole program if first error occurs, default is true - -t timeout for a check, default is 600000 (milliseconds) + -t timeout for a check, default is 600000 (millisecs) + -writeTableTimeout write timeout for the writeTable, default is 600000 (millisecs) + -readTableTimeouts =,=, ... comma-separated list of read timeouts per table (no spaces), default is 600000 (millisecs) -writeSniffing enable the write sniffing in canary -treatFailureAsError treats read / write failure as error -writeTable The table used for write sniffing. Default is hbase:canary + -Dhbase.canary.read.raw.enabled= Use this flag to enable or disable raw scan during read canary test Default is false and raw is not enabled during scan -D= assigning or override the configuration params ---- @@ -107,6 +116,7 @@ private static final int USAGE_EXIT_CODE = 1; private static final int INIT_ERROR_EXIT_CODE = 2; private static final int TIMEOUT_ERROR_EXIT_CODE = 3; private static final int ERROR_EXIT_CODE = 4; +private static final int FAILURE_EXIT_CODE = 5; ---- Here are some examples based on the following given case. @@ -673,27 +683,20 @@ Options: === `hbase pe` -The `hbase pe` command is a shortcut provided to run the `org.apache.hadoop.hbase.PerformanceEvaluation` tool, which is used for testing. -The `hbase pe` command was introduced in HBase 0.98.4. +The `hbase pe` command runs the PerformanceEvaluation tool, which is used for testing. The PerformanceEvaluation tool accepts many different options and commands. For usage instructions, run the command with no options. -To run PerformanceEvaluation prior to HBase 0.98.4, issue the command `hbase org.apache.hadoop.hbase.PerformanceEvaluation`. - The PerformanceEvaluation tool has received many updates in recent HBase releases, including support for namespaces, support for tags, cell-level ACLs and visibility labels, multiget support for RPC calls, increased sampling sizes, an option to randomly sleep during testing, and ability to "warm up" the cluster before testing starts. === `hbase ltt` -The `hbase ltt` command is a shortcut provided to run the `org.apache.hadoop.hbase.util.LoadTestTool` utility, which is used for testing. -The `hbase ltt` command was introduced in HBase 0.98.4. +The `hbase ltt` command runs the LoadTestTool utility, which is used for testing. -You must specify either `-write` or `-update-read` as the first option. +You must specify one of `-write`, `-update`, or `-read` as the first option. For general usage instructions, pass the `-h` option. -To run LoadTestTool prior to HBase 0.98.4, issue the command +hbase - org.apache.hadoop.hbase.util.LoadTestTool+. - The LoadTestTool has received many updates in recent HBase releases, including support for namespaces, support for tags, cell-level ACLS and visibility labels, testing security-related features, ability to specify the number of regions per server, tests for multi-get RPC calls, and tests relating to replication. [[ops.regionmgt]] @@ -756,7 +759,7 @@ See <> below. [NOTE] ==== In hbase-2.0, in the bin directory, we added a script named _considerAsDead.sh_ that can be used to kill a regionserver. -Hardware issues could be detected by specialized monitoring tools before the zookeeper timeout has expired. _considerAsDead.sh_ is a simple function to mark a RegionServer as dead. +Hardware issues could be detected by specialized monitoring tools before the zookeeper timeout has expired. _considerAsDead.sh_ is a simple function to mark a RegionServer as dead. It deletes all the znodes of the server, starting the recovery process. Plug in the script into your monitoring/fault detection tools to initiate faster failover. Be careful how you use this disruptive tool. diff --git a/src/main/asciidoc/_chapters/troubleshooting.adoc b/src/main/asciidoc/_chapters/troubleshooting.adoc index 906b2f8d2154..cb3342421c51 100644 --- a/src/main/asciidoc/_chapters/troubleshooting.adoc +++ b/src/main/asciidoc/_chapters/troubleshooting.adoc @@ -101,6 +101,11 @@ To disable, set the logging level back to `INFO` level. [[trouble.log.gc]] === JVM Garbage Collection Logs +[NOTE] +---- +All example Garbage Collection logs in this section are based on Java 8 output. The introduction of Unified Logging in Java 9 and newer will result in very different looking logs. +---- + HBase is memory intensive, and using the default GC you can see long pauses in all threads including the _Juliet Pause_ aka "GC of Death". To help debug this or confirm this is happening GC logging can be turned on in the Java virtual machine. To enable, in _hbase-env.sh_, uncomment one of the below lines : @@ -253,7 +258,6 @@ link:https://issues.apache.org/jira/browse/HBASE[JIRA] is also really helpful wh ==== Master Web Interface The Master starts a web-interface on port 16010 by default. -(Up to and including 0.98 this was port 60010) The Master web UI lists created tables and their definition (e.g., ColumnFamilies, blocksize, etc.). Additionally, the available RegionServers in the cluster are listed along with selected high-level metrics (requests, number of regions, usedHeap, maxHeap). The Master web UI allows navigation to each RegionServer's web UI. @@ -261,7 +265,6 @@ The Master web UI lists created tables and their definition (e.g., ColumnFamilie ==== RegionServer Web Interface RegionServers starts a web-interface on port 16030 by default. -(Up to an including 0.98 this was port 60030) The RegionServer web UI lists online regions and their start/end keys, as well as point-in-time RegionServer metrics (requests, regions, storeFileIndexSize, compactionQueueSize, etc.). @@ -557,7 +560,7 @@ You can also tail all the logs at the same time, edit files, etc. [[trouble.client]] == Client -For more information on the HBase client, see <>. +For more information on the HBase client, see <>. [[trouble.client.scantimeout]] === ScannerTimeoutException or UnknownScannerException @@ -670,12 +673,6 @@ A workaround is passing your client-side JVM a reasonable value for `-XX:MaxDire By default, the `MaxDirectMemorySize` is equal to your `-Xmx` max heapsize setting (if `-Xmx` is set). Try setting it to something smaller (for example, one user had success setting it to `1g` when they had a client-side heap of `12g`). If you set it too small, it will bring on `FullGCs` so keep it a bit hefty. You want to make this setting client-side only especially if you are running the new experimental server-side off-heap cache since this feature depends on being able to use big direct buffers (You may have to keep separate client-side and server-side config dirs). -[[trouble.client.slowdown.admin]] -=== Client Slowdown When Calling Admin Methods (flush, compact, etc.) - -This is a client issue fixed by link:https://issues.apache.org/jira/browse/HBASE-5073[HBASE-5073] in 0.90.6. -There was a ZooKeeper leak in the client and the client was getting pummeled by ZooKeeper events with each additional invocation of the admin API. - [[trouble.client.security.rpc]] === Secure Client Cannot Connect ([Caused by GSSException: No valid credentials provided(Mechanism level: Failed to find any Kerberos tgt)]) @@ -848,7 +845,6 @@ See <> for more information on managing compactions. === Loopback IP HBase expects the loopback IP Address to be 127.0.0.1. -See the Getting Started section on <>. [[trouble.network.ints]] === Network Interfaces @@ -1071,13 +1067,6 @@ This exception is returned back to the client and then the client goes back to ` However, if the NotServingRegionException is logged ERROR, then the client ran out of retries and something probably wrong. -[[trouble.rs.runtime.double_listed_regions]] -==== Regions listed by domain name, then IP - -Fix your DNS. -In versions of Apache HBase before 0.92.x, reverse DNS needs to give same answer as forward lookup. -See link:https://issues.apache.org/jira/browse/HBASE-3431[HBASE 3431 RegionServer is not using the name given it by the master; double entry in master listing of servers] for gorey details. - [[brand.new.compressor]] ==== Logs flooded with '2011-01-10 12:40:48,407 INFO org.apache.hadoop.io.compress.CodecPool: Gotbrand-new compressor' messages @@ -1234,31 +1223,6 @@ See Andrew's answer here, up on the user list: link:http://search-hadoop.com/m/s [[trouble.versions]] == HBase and Hadoop version issues -[[trouble.versions.205]] -=== `NoClassDefFoundError` when trying to run 0.90.x on hadoop-0.20.205.x (or hadoop-1.0.x) - -Apache HBase 0.90.x does not ship with hadoop-0.20.205.x, etc. -To make it run, you need to replace the hadoop jars that Apache HBase shipped with in its _lib_ directory with those of the Hadoop you want to run HBase on. -If even after replacing Hadoop jars you get the below exception: - -[source] ----- - -sv4r6s38: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration -sv4r6s38: at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.(DefaultMetricsSystem.java:37) -sv4r6s38: at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.(DefaultMetricsSystem.java:34) -sv4r6s38: at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51) -sv4r6s38: at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:209) -sv4r6s38: at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:177) -sv4r6s38: at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:229) -sv4r6s38: at org.apache.hadoop.security.KerberosName.(KerberosName.java:83) -sv4r6s38: at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:202) -sv4r6s38: at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:177) ----- - -you need to copy under _hbase/lib_, the _commons-configuration-X.jar_ you find in your Hadoop's _lib_ directory. -That should fix the above complaint. - [[trouble.wrong.version]] === ...cannot communicate with client version... @@ -1267,67 +1231,6 @@ If you see something like the following in your logs [computeroutput]+... 2012-0 shutdown. org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4 ...+ ...are you trying to talk to an Hadoop 2.0.x from an HBase that has an Hadoop 1.0.x client? Use the HBase built against Hadoop 2.0 or rebuild your HBase passing the +-Dhadoop.profile=2.0+ attribute to Maven (See <> for more). -== IPC Configuration Conflicts with Hadoop - -If the Hadoop configuration is loaded after the HBase configuration, and you have configured custom IPC settings in both HBase and Hadoop, the Hadoop values may overwrite the HBase values. -There is normally no need to change these settings for HBase, so this problem is an edge case. -However, link:https://issues.apache.org/jira/browse/HBASE-11492[HBASE-11492] renames these settings for HBase to remove the chance of a conflict. -Each of the setting names have been prefixed with `hbase.`, as shown in the following table. -No action is required related to these changes unless you are already experiencing a conflict. - -These changes were backported to HBase 0.98.x and apply to all newer versions. - -[cols="1,1", options="header"] -|=== -| Pre-0.98.x -| 0.98-x And Newer - -| ipc.server.listen.queue.size -| hbase.ipc.server.listen.queue.size - -| ipc.server.max.callqueue.size -| hbase.ipc.server.max.callqueue.size - -| ipc.server.callqueue.handler.factor -| hbase.ipc.server.callqueue.handler.factor - -| ipc.server.callqueue.read.share -| hbase.ipc.server.callqueue.read.share - -| ipc.server.callqueue.type -| hbase.ipc.server.callqueue.type - -| ipc.server.queue.max.call.delay -| hbase.ipc.server.queue.max.call.delay - -| ipc.server.max.callqueue.length -| hbase.ipc.server.max.callqueue.length - -| ipc.server.read.threadpool.size -| hbase.ipc.server.read.threadpool.size - -| ipc.server.tcpkeepalive -| hbase.ipc.server.tcpkeepalive - -| ipc.server.tcpnodelay -| hbase.ipc.server.tcpnodelay - -| ipc.client.call.purge.timeout -| hbase.ipc.client.call.purge.timeout - -| ipc.client.connection.maxidletime -| hbase.ipc.client.connection.maxidletime - -| ipc.client.idlethreshold -| hbase.ipc.client.idlethreshold - -| ipc.client.kill.max -| hbase.ipc.client.kill.max - -| ipc.server.scan.vtime.weight -| hbase.ipc.server.scan.vtime.weight -|=== - == HBase and HDFS General configuration guidance for Apache HDFS is out of the scope of this guide. diff --git a/src/main/asciidoc/_chapters/upgrading.adoc b/src/main/asciidoc/_chapters/upgrading.adoc index 3e4f5e44e810..91af2b63b082 100644 --- a/src/main/asciidoc/_chapters/upgrading.adoc +++ b/src/main/asciidoc/_chapters/upgrading.adoc @@ -27,19 +27,15 @@ :icons: font :experimental: -You cannot skip major versions when upgrading. If you are upgrading from version 0.90.x to 0.94.x, you must first go from 0.90.x to 0.92.x and then go from 0.92.x to 0.94.x. - -NOTE: It may be possible to skip across versions -- for example go from 0.92.2 straight to 0.98.0 just following the 0.96.x upgrade instructions -- but these scenarios are untested. +You cannot skip major versions when upgrading. If you are upgrading from version 0.98.x to 2.x, you must first go from 0.98.x to 1.2.x and then go from 1.2.x to 2.x. Review <>, in particular <>. Familiarize yourself with <>. [[hbase.versioning]] == HBase version number and compatibility -HBase has two versioning schemes, pre-1.0 and post-1.0. Both are detailed below. - [[hbase.versioning.post10]] -=== Post 1.0 versions +=== Aspirational Semantic Versioning Starting with the 1.0.0 release, HBase is working towards link:http://semver.org/[Semantic Versioning] for its release versioning. In summary: @@ -147,23 +143,9 @@ HBase LimitedPrivate API:: HBase Private API:: All classes annotated with InterfaceAudience.Private or all classes that do not have the annotation are for HBase internal use only. The interfaces and method signatures can change at any point in time. If you are relying on a particular interface that is marked Private, you should open a jira to propose changing the interface to be Public or LimitedPrivate, or an interface exposed for this purpose. -[[hbase.versioning.pre10]] -=== Pre 1.0 versions - -.HBase Pre-1.0 versions are all EOM -NOTE: For new installations, do not deploy 0.94.y, 0.96.y, or 0.98.y. Deploy our stable version. See link:https://issues.apache.org/jira/browse/HBASE-11642[EOL 0.96], link:https://issues.apache.org/jira/browse/HBASE-16215[clean up of EOM releases], and link:http://www.apache.org/dist/hbase/[the header of our downloads]. - -Before the semantic versioning scheme pre-1.0, HBase tracked either Hadoop's versions (0.2x) or 0.9x versions. If you are into the arcane, checkout our old wiki page on link:http://wiki.apache.org/hadoop/Hbase/HBaseVersions[HBase Versioning] which tries to connect the HBase version dots. Below sections cover ONLY the releases before 1.0. - -[[hbase.development.series]] -.Odd/Even Versioning or "Development" Series Releases -Ahead of big releases, we have been putting up preview versions to start the feedback cycle turning-over earlier. These "Development" Series releases, always odd-numbered, come with no guarantees, not even regards being able to upgrade between two sequential releases (we reserve the right to break compatibility across "Development" Series releases). Needless to say, these releases are not for production deploys. They are a preview of what is coming in the hope that interested parties will take the release for a test drive and flag us early if we there are issues we've missed ahead of our rolling a production-worthy release. - -Our first "Development" Series was the 0.89 set that came out ahead of HBase 0.90.0. HBase 0.95 is another "Development" Series that portends HBase 0.96.0. 0.99.x is the last series in "developer preview" mode before 1.0. Afterwards, we will be using semantic versioning naming scheme (see above). - [[hbase.binary.compatibility]] .Binary Compatibility -When we say two HBase versions are compatible, we mean that the versions are wire and binary compatible. Compatible HBase versions means that clients can talk to compatible but differently versioned servers. It means too that you can just swap out the jars of one version and replace them with the jars of another, compatible version and all will just work. Unless otherwise specified, HBase point versions are (mostly) binary compatible. You can safely do rolling upgrades between binary compatible versions; i.e. across point versions: e.g. from 0.94.5 to 0.94.6. See link:[Does compatibility between versions also mean binary compatibility?] discussion on the HBase dev mailing list. +When we say two HBase versions are compatible, we mean that the versions are wire and binary compatible. Compatible HBase versions means that clients can talk to compatible but differently versioned servers. It means too that you can just swap out the jars of one version and replace them with the jars of another, compatible version and all will just work. Unless otherwise specified, HBase point versions are (mostly) binary compatible. You can safely do rolling upgrades between binary compatible versions; i.e. across maintenance releases: e.g. from 1.2.4 to 1.2.6. See link:[Does compatibility between versions also mean binary compatibility?] discussion on the HBase dev mailing list. [[hbase.rolling.upgrade]] === Rolling Upgrades @@ -181,9 +163,9 @@ The rolling-restart script will first gracefully stop and restart the master, an [[hbase.rolling.restart]] .Rolling Upgrade Between Versions that are Binary/Wire Compatible -Unless otherwise specified, HBase point versions are binary compatible. You can do a <> between HBase point versions. For example, you can go to 0.94.6 from 0.94.5 by doing a rolling upgrade across the cluster replacing the 0.94.5 binary with a 0.94.6 binary. +Unless otherwise specified, HBase minor versions are binary compatible. You can do a <> between HBase point versions. For example, you can go to 1.2.4 from 1.2.6 by doing a rolling upgrade across the cluster replacing the 1.2.4 binary with a 1.2.6 binary. -In the minor version-particular sections below, we call out where the versions are wire/protocol compatible and in this case, it is also possible to do a <>. For example, in <>, we state that it is possible to do a rolling upgrade between hbase-0.98.x and hbase-1.0.0. +In the minor version-particular sections below, we call out where the versions are wire/protocol compatible and in this case, it is also possible to do a <>. == Upgrade Paths @@ -241,244 +223,10 @@ The Date Tiered Compaction feature available as of 0.98.19 is available in the 1 [[upgrade1.0.rolling.upgrade]] ==== Rolling upgrade from 0.98.x to HBase 1.0.0 -.From 0.96.x to 1.0.0 -NOTE: You cannot do a <> from 0.96.x to 1.0.0 without first doing a rolling upgrade to 0.98.x. See comment in link:https://issues.apache.org/jira/browse/HBASE-11164?focusedCommentId=14182330&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14182330[HBASE-11164 Document and test rolling updates from 0.98 -> 1.0] for the why. Also because HBase 1.0.0 enables HFile v3 by default, link:https://issues.apache.org/jira/browse/HBASE-9801[HBASE-9801 Change the default HFile version to V3], and support for HFile v3 only arrives in 0.98, this is another reason you cannot rolling upgrade from HBase 0.96.x; if the rolling upgrade stalls, the 0.96.x servers cannot open files written by the servers running the newer HBase 1.0.0 with HFile's of version 3. There are no known issues running a <> from HBase 0.98.x to HBase 1.0.0. -[[upgrade1.0.from.0.94]] -==== Upgrading to 1.0 from 0.94 -You cannot rolling upgrade from 0.94.x to 1.x.x. You must stop your cluster, install the 1.x.x software, run the migration described at <> (substituting 1.x.x. wherever we make mention of 0.96.x in the section below), and then restart. Be sure to upgrade your ZooKeeper if it is a version less than the required 3.4.x. - -[[upgrade0.98]] -=== Upgrading from 0.96.x to 0.98.x -A rolling upgrade from 0.96.x to 0.98.x works. The two versions are not binary compatible. - -Additional steps are required to take advantage of some of the new features of 0.98.x, including cell visibility labels, cell ACLs, and transparent server side encryption. See <> for more information. Significant performance improvements include a change to the write ahead log threading model that provides higher transaction throughput under high load, reverse scanners, MapReduce over snapshot files, and striped compaction. - -Clients and servers can run with 0.98.x and 0.96.x versions. However, applications may need to be recompiled due to changes in the Java API. - -=== Upgrading from 0.94.x to 0.98.x -A rolling upgrade from 0.94.x directly to 0.98.x does not work. The upgrade path follows the same procedures as <>. Additional steps are required to use some of the new features of 0.98.x. See <> for an abbreviated list of these features. - -[[upgrade0.96]] -=== Upgrading from 0.94.x to 0.96.x - -==== The "Singularity" - -You will have to stop your old 0.94.x cluster completely to upgrade. If you are replicating between clusters, both clusters will have to go down to upgrade. Make sure it is a clean shutdown. The less WAL files around, the faster the upgrade will run (the upgrade will split any log files it finds in the filesystem as part of the upgrade process). All clients must be upgraded to 0.96 too. - -The API has changed. You will need to recompile your code against 0.96 and you may need to adjust applications to go against new APIs (TODO: List of changes). - -[[executing.the.0.96.upgrade]] -==== Executing the 0.96 Upgrade - -.HDFS and ZooKeeper must be up! -NOTE: HDFS and ZooKeeper should be up and running during the upgrade process. - -HBase 0.96.0 comes with an upgrade script. Run - -[source,bash] ----- -$ bin/hbase upgrade ----- -to see its usage. The script has two main modes: `-check`, and `-execute`. - -.check -The check step is run against a running 0.94 cluster. Run it from a downloaded 0.96.x binary. The check step is looking for the presence of HFile v1 files. These are unsupported in HBase 0.96.0. To have them rewritten as HFile v2 you must run a compaction. - -The check step prints stats at the end of its run (grep for `“Result:”` in the log) printing absolute path of the tables it scanned, any HFile v1 files found, the regions containing said files (these regions will need a major compaction), and any corrupted files if found. A corrupt file is unreadable, and so is undefined (neither HFile v1 nor HFile v2). - -To run the check step, run - -[source,bash] ----- -$ bin/hbase upgrade -check ----- - -Here is sample output: ----- -Tables Processed: -hdfs://localhost:41020/myHBase/.META. -hdfs://localhost:41020/myHBase/usertable -hdfs://localhost:41020/myHBase/TestTable -hdfs://localhost:41020/myHBase/t - -Count of HFileV1: 2 -HFileV1: -hdfs://localhost:41020/myHBase/usertable /fa02dac1f38d03577bd0f7e666f12812/family/249450144068442524 -hdfs://localhost:41020/myHBase/usertable /ecdd3eaee2d2fcf8184ac025555bb2af/family/249450144068442512 - -Count of corrupted files: 1 -Corrupted Files: -hdfs://localhost:41020/myHBase/usertable/fa02dac1f38d03577bd0f7e666f12812/family/1 -Count of Regions with HFileV1: 2 -Regions to Major Compact: -hdfs://localhost:41020/myHBase/usertable/fa02dac1f38d03577bd0f7e666f12812 -hdfs://localhost:41020/myHBase/usertable/ecdd3eaee2d2fcf8184ac025555bb2af - -There are some HFileV1, or corrupt files (files with incorrect major version) ----- - -In the above sample output, there are two HFile v1 files in two regions, and one corrupt file. Corrupt files should probably be removed. The regions that have HFile v1s need to be major compacted. To major compact, start up the hbase shell and review how to compact an individual region. After the major compaction is done, rerun the check step and the HFile v1 files should be gone, replaced by HFile v2 instances. - -By default, the check step scans the HBase root directory (defined as `hbase.rootdir` in the configuration). To scan a specific directory only, pass the `-dir` option. -[source,bash] ----- -$ bin/hbase upgrade -check -dir /myHBase/testTable ----- -The above command would detect HFile v1 files in the _/myHBase/testTable_ directory. - -Once the check step reports all the HFile v1 files have been rewritten, it is safe to proceed with the upgrade. - -.execute -After the _check_ step shows the cluster is free of HFile v1, it is safe to proceed with the upgrade. Next is the _execute_ step. You must *SHUTDOWN YOUR 0.94.x CLUSTER* before you can run the execute step. The execute step will not run if it detects running HBase masters or RegionServers. - -[NOTE] -==== -HDFS and ZooKeeper should be up and running during the upgrade process. If zookeeper is managed by HBase, then you can start zookeeper so it is available to the upgrade by running -[source,bash] ----- -$ ./hbase/bin/hbase-daemon.sh start zookeeper ----- -==== - -The execute upgrade step is made of three substeps. - -* Namespaces: HBase 0.96.0 has support for namespaces. The upgrade needs to reorder directories in the filesystem for namespaces to work. +[[upgrade2.0]] +=== Upgrading to 2.x -* ZNodes: All znodes are purged so that new ones can be written in their place using a new protobuf'ed format and a few are migrated in place: e.g. replication and table state znodes - -* WAL Log Splitting: If the 0.94.x cluster shutdown was not clean, we'll split WAL logs as part of migration before we startup on 0.96.0. This WAL splitting runs slower than the native distributed WAL splitting because it is all inside the single upgrade process (so try and get a clean shutdown of the 0.94.0 cluster if you can). - -To run the _execute_ step, make sure that first you have copied HBase 0.96.0 binaries everywhere under servers and under clients. Make sure the 0.94.0 cluster is down. Then do as follows: -[source,bash] ----- -$ bin/hbase upgrade -execute ----- -Here is some sample output. - ----- -Starting Namespace upgrade -Created version file at hdfs://localhost:41020/myHBase with version=7 -Migrating table testTable to hdfs://localhost:41020/myHBase/.data/default/testTable -..... -Created version file at hdfs://localhost:41020/myHBase with version=8 -Successfully completed NameSpace upgrade. -Starting Znode upgrade -..... -Successfully completed Znode upgrade - -Starting Log splitting -... -Successfully completed Log splitting ----- - -If the output from the execute step looks good, stop the zookeeper instance you started to do the upgrade: -[source,bash] ----- -$ ./hbase/bin/hbase-daemon.sh stop zookeeper ----- -Now start up hbase-0.96.0. - -[[s096.migration.troubleshooting]] -=== Troubleshooting - -[[s096.migration.troubleshooting.old.client]] -.Old Client connecting to 0.96 cluster -It will fail with an exception like the below. Upgrade. ----- -17:22:15 Exception in thread "main" java.lang.IllegalArgumentException: Not a host:port pair: PBUF -17:22:15 * -17:22:15 api-compat-8.ent.cloudera.com �� ���( -17:22:15 at org.apache.hadoop.hbase.util.Addressing.parseHostname(Addressing.java:60) -17:22:15 at org.apache.hadoop.hbase.ServerName.&init>(ServerName.java:101) -17:22:15 at org.apache.hadoop.hbase.ServerName.parseVersionedServerName(ServerName.java:283) -17:22:15 at org.apache.hadoop.hbase.MasterAddressTracker.bytesToServerName(MasterAddressTracker.java:77) -17:22:15 at org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:61) -17:22:15 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:703) -17:22:15 at org.apache.hadoop.hbase.client.HBaseAdmin.&init>(HBaseAdmin.java:126) -17:22:15 at Client_4_3_0.setup(Client_4_3_0.java:716) -17:22:15 at Client_4_3_0.main(Client_4_3_0.java:63) ----- - -==== Upgrading `META` to use Protocol Buffers (Protobuf) - -When you upgrade from versions prior to 0.96, `META` needs to be converted to use protocol buffers. This is controlled by the configuration option `hbase.MetaMigrationConvertingToPB`, which is set to `true` by default. Therefore, by default, no action is required on your part. - -The migration is a one-time event. However, every time your cluster starts, `META` is scanned to ensure that it does not need to be converted. If you have a very large number of regions, this scan can take a long time. Starting in 0.98.5, you can set `hbase.MetaMigrationConvertingToPB` to `false` in _hbase-site.xml_, to disable this start-up scan. This should be considered an expert-level setting. - -[[upgrade0.94]] -=== Upgrading from 0.92.x to 0.94.x -We used to think that 0.92 and 0.94 were interface compatible and that you can do a rolling upgrade between these versions but then we figured that link:https://issues.apache.org/jira/browse/HBASE-5357[HBASE-5357 Use builder pattern in HColumnDescriptor] changed method signatures so rather than return `void` they instead return `HColumnDescriptor`. This will throw`java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V` so 0.92 and 0.94 are NOT compatible. You cannot do a rolling upgrade between them. - -[[upgrade0.92]] -=== Upgrading from 0.90.x to 0.92.x -==== Upgrade Guide -You will find that 0.92.0 runs a little differently to 0.90.x releases. Here are a few things to watch out for upgrading from 0.90.x to 0.92.0. - -.tl:dr -[NOTE] -==== -These are the important things to know before upgrading. -. Once you upgrade, you can’t go back. - -. MSLAB is on by default. Watch that heap usage if you have a lot of regions. - -. Distributed Log Splitting is on by default. It should make RegionServer failover faster. - -. There’s a separate tarball for security. - -. If `-XX:MaxDirectMemorySize` is set in your _hbase-env.sh_, it’s going to enable the experimental off-heap cache (You may not want this). -==== - -.You can’t go back! -To move to 0.92.0, all you need to do is shutdown your cluster, replace your HBase 0.90.x with HBase 0.92.0 binaries (be sure you clear out all 0.90.x instances) and restart (You cannot do a rolling restart from 0.90.x to 0.92.x -- you must restart). On startup, the `.META.` table content is rewritten removing the table schema from the `info:regioninfo` column. Also, any flushes done post first startup will write out data in the new 0.92.0 file format, <>. This means you cannot go back to 0.90.x once you’ve started HBase 0.92.0 over your HBase data directory. - -.MSLAB is ON by default -In 0.92.0, the `<>` flag is set to `true` (See <>). In 0.90.x it was false. When it is enabled, memstores will step allocate memory in MSLAB 2MB chunks even if the memstore has zero or just a few small elements. This is fine usually but if you had lots of regions per RegionServer in a 0.90.x cluster (and MSLAB was off), you may find yourself OOME'ing on upgrade because the `thousands of regions * number of column families * 2MB MSLAB` (at a minimum) puts your heap over the top. Set `hbase.hregion.memstore.mslab.enabled` to `false` or set the MSLAB size down from 2MB by setting `hbase.hregion.memstore.mslab.chunksize` to something less. - -[[dls]] -.Distributed Log Splitting is on by default -Previous, WAL logs on crash were split by the Master alone. In 0.92.0, log splitting is done by the cluster (See link:https://issues.apache.org/jira/browse/hbase-1364[HBASE-1364 [performance\] Distributed splitting of regionserver commit logs] or see the blog post link:http://blog.cloudera.com/blog/2012/07/hbase-log-splitting/[Apache HBase Log Splitting]). This should cut down significantly on the amount of time it takes splitting logs and getting regions back online again. - -.Memory accounting is different now -In 0.92.0, <> indices and bloom filters take up residence in the same LRU used caching blocks that come from the filesystem. In 0.90.x, the HFile v1 indices lived outside of the LRU so they took up space even if the index was on a ‘cold’ file, one that wasn’t being actively used. With the indices now in the LRU, you may find you have less space for block caching. Adjust your block cache accordingly. See the <> for more detail. The block size default size has been changed in 0.92.0 from 0.2 (20 percent of heap) to 0.25. - -.On the Hadoop version to use -Run 0.92.0 on Hadoop 1.0.x (or CDH3u3). The performance benefits are worth making the move. Otherwise, our Hadoop prescription is as it has been; you need an Hadoop that supports a working sync. See <>. - -If running on Hadoop 1.0.x (or CDH3u3), enable local read. See link:http://files.meetup.com/1350427/hug_ebay_jdcryans.pdf[Practical Caching] presentation for ruminations on the performance benefits ‘going local’ (and for how to enable local reads). - -.HBase 0.92.0 ships with ZooKeeper 3.4.2 -If you can, upgrade your ZooKeeper. If you can’t, 3.4.2 clients should work against 3.3.X ensembles (HBase makes use of 3.4.2 API). - -.Online alter is off by default -In 0.92.0, we’ve added an experimental online schema alter facility (See <>). It's off by default. Enable it at your own risk. Online alter and splitting tables do not play well together so be sure your cluster quiescent using this feature (for now). - -.WebUI -The web UI has had a few additions made in 0.92.0. It now shows a list of the regions currently transitioning, recent compactions/flushes, and a process list of running processes (usually empty if all is well and requests are being handled promptly). Other additions including requests by region, a debugging servlet dump, etc. - -.Security tarball -We now ship with two tarballs; secure and insecure HBase. Documentation on how to setup a secure HBase is on the way. - -.Changes in HBase replication -0.92.0 adds two new features: multi-slave and multi-master replication. The way to enable this is the same as adding a new peer, so in order to have multi-master you would just run add_peer for each cluster that acts as a master to the other slave clusters. Collisions are handled at the timestamp level which may or may not be what you want, this needs to be evaluated on a per use case basis. Replication is still experimental in 0.92 and is disabled by default, run it at your own risk. - -.RegionServer now aborts if OOME -If an OOME, we now have the JVM kill -9 the RegionServer process so it goes down fast. Previous, a RegionServer might stick around after incurring an OOME limping along in some wounded state. To disable this facility, and recommend you leave it in place, you’d need to edit the bin/hbase file. Look for the addition of the -XX:OnOutOfMemoryError="kill -9 %p" arguments (See link:https://issues.apache.org/jira/browse/HBASE-4769[HBASE-4769 - ‘Abort RegionServer Immediately on OOME’]). - -.HFile v2 and the “Bigger, Fewer” Tendency -0.92.0 stores data in a new format, <>. As HBase runs, it will move all your data from HFile v1 to HFile v2 format. This auto-migration will run in the background as flushes and compactions run. HFile v2 allows HBase run with larger regions/files. In fact, we encourage that all HBasers going forward tend toward Facebook axiom #1, run with larger, fewer regions. If you have lots of regions now -- more than 100s per host -- you should look into setting your region size up after you move to 0.92.0 (In 0.92.0, default size is now 1G, up from 256M), and then running online merge tool (See link:https://issues.apache.org/jira/browse/HBASE-1621[HBASE-1621 merge tool should work on online cluster, but disabled table]). - -[[upgrade0.90]] -=== Upgrading to HBase 0.90.x from 0.20.x or 0.89.x -This version of 0.90.x HBase can be started on data written by HBase 0.20.x or HBase 0.89.x. There is no need of a migration step. HBase 0.89.x and 0.90.x does write out the name of region directories differently -- it names them with a md5 hash of the region name rather than a jenkins hash -- so this means that once started, there is no going back to HBase 0.20.x. - -Be sure to remove the _hbase-default.xml_ from your _conf_ directory on upgrade. A 0.20.x version of this file will have sub-optimal configurations for 0.90.x HBase. The _hbase-default.xml_ file is now bundled into the HBase jar and read from there. If you would like to review the content of this file, see it in the src tree at _src/main/resources/hbase-default.xml_ or see <>. - -Finally, if upgrading from 0.20.x, check your .META. schema in the shell. In the past we would recommend that users run with a 16kb MEMSTORE_FLUSHSIZE. Run ----- -hbase> scan '-ROOT-' ----- -in the shell. This will output the current `.META.` schema. Check `MEMSTORE_FLUSHSIZE` size. Is it 16kb (16384)? If so, you will need to change this (The 'normal'/default value is 64MB (67108864)). Run the script `bin/set_meta_memstore_size.rb`. This will make the necessary edit to your `.META.` schema. Failure to run this change will make for a slow cluster. See link:https://issues.apache.org/jira/browse/HBASE-3499[HBASE-3499 Users upgrading to 0.90.0 need to have their .META. table updated with the right MEMSTORE_SIZE]. +Please see the reference guide for the latest 2.x release for guidance on upgrading. From 6d955d19669f0dbffda3804ddd1b6eb8292fa7f0 Mon Sep 17 00:00:00 2001 From: Michael Stack Date: Tue, 10 Apr 2018 16:58:09 -0700 Subject: [PATCH 15/27] HBASE-17554 Figure 2.0.0 Hadoop Version Support; update refguide Minor edit on hadoop section. Mark 2.8.3 as supported and 2.8.2 as NT. (cherry picked from commit 1b2e80389d6cfee95d17c07d4634374f49b78c08) Conflicts: src/main/asciidoc/_chapters/configuration.adoc --- .../asciidoc/_chapters/configuration.adoc | 23 +++++++++++-------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 3e8f0c916bac..0d7a371a420d 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -29,7 +29,7 @@ This chapter expands upon the <> chapter to further explain configuration of Apache HBase. Please read this chapter carefully, especially the <> -to ensure that your HBase testing and deployment goes smoothly, and prevent data loss. +to ensure that your HBase testing and deployment goes smoothly. Familiarize yourself with <> as well. == Configuration Files @@ -162,9 +162,9 @@ It is recommended to raise the ulimit to at least 10,000, but more likely 10,240 + For example, assuming that a schema had 3 ColumnFamilies per region with an average of 3 StoreFiles per ColumnFamily, and there are 100 regions per RegionServer, the JVM will open `3 * 3 * 100 = 900` file descriptors, not counting open JAR files, configuration files, and others. Opening a file does not take many resources, and the risk of allowing a user to open too many files is minimal. + -Another related setting is the number of processes a user is allowed to run at once. In Linux and Unix, the number of processes is set using the `ulimit -u` command. This should not be confused with the `nproc` command, which controls the number of CPUs available to a given user. Under load, a `ulimit -u` that is too low can cause OutOfMemoryError exceptions. See Jack Levin's major HDFS issues thread on the hbase-users mailing list, from 2011. +Another related setting is the number of processes a user is allowed to run at once. In Linux and Unix, the number of processes is set using the `ulimit -u` command. This should not be confused with the `nproc` command, which controls the number of CPUs available to a given user. Under load, a `ulimit -u` that is too low can cause OutOfMemoryError exceptions. + -Configuring the maximum number of file descriptors and processes for the user who is running the HBase process is an operating system configuration, rather than an HBase configuration. It is also important to be sure that the settings are changed for the user that actually runs HBase. To see which user started HBase, and that user's ulimit configuration, look at the first line of the HBase log for that instance. A useful read setting config on you hadoop cluster is Aaron Kimballs' Configuration Parameters: What can you just ignore? +Configuring the maximum number of file descriptors and processes for the user who is running the HBase process is an operating system configuration, rather than an HBase configuration. It is also important to be sure that the settings are changed for the user that actually runs HBase. To see which user started HBase, and that user's ulimit configuration, look at the first line of the HBase log for that instance. + .`ulimit` Settings on Ubuntu ==== @@ -199,7 +199,8 @@ See link:http://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Suppor .Hadoop 2.x is recommended. [TIP] ==== -Hadoop 2.x is faster and includes features, such as short-circuit reads, which will help improve your HBase random read profile. +Hadoop 2.x is faster and includes features, such as short-circuit reads (see <>), +which will help improve your HBase random read profile. Hadoop 2.x also includes important bug fixes that will improve your overall HBase experience. HBase does not support running with earlier versions of Hadoop. See the table below for requirements specific to different HBase versions. @@ -251,18 +252,20 @@ data loss. This patch is present in Apache Hadoop releases 2.6.1+. .Hadoop 2.y.0 Releases [TIP] ==== -Starting around the time of Hadoop version 2.7.0, the Hadoop PMC got into the habit of calling out new minor releases on their major version 2 release line as not stable / production ready. As such, HBase expressly advises downstream users to avoid running on top of these releases. Note that additionally the 2.8.1 was release was given the same caveat by the Hadoop PMC. For reference, see the release announcements for link:https://s.apache.org/hadoop-2.7.0-announcement[Apache Hadoop 2.7.0], link:https://s.apache.org/hadoop-2.8.0-announcement[Apache Hadoop 2.8.0], link:https://s.apache.org/hadoop-2.8.1-announcement[Apache Hadoop 2.8.1], and link:https://s.apache.org/hadoop-2.9.0-announcement[Apache Hadoop 2.9.0]. +Starting around the time of Hadoop version 2.7.0, the Hadoop PMC got into the habit of calling out new minor releases on their major version 2 release line as not stable / production ready. As such, HBase expressly advises downstream users to avoid running on top of these releases. Note that additionally the 2.8.1 release was given the same caveat by the Hadoop PMC. For reference, see the release announcements for link:https://s.apache.org/hadoop-2.7.0-announcement[Apache Hadoop 2.7.0], link:https://s.apache.org/hadoop-2.8.0-announcement[Apache Hadoop 2.8.0], link:https://s.apache.org/hadoop-2.8.1-announcement[Apache Hadoop 2.8.1], and link:https://s.apache.org/hadoop-2.9.0-announcement[Apache Hadoop 2.9.0]. ==== .Replace the Hadoop Bundled With HBase! [NOTE] ==== -Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its _lib_ directory. -The bundled jar is ONLY for use in standalone mode. +Because HBase depends on Hadoop, it bundles Hadoop jars under its _lib_ directory. +The bundled jars are ONLY for use in standalone mode. In distributed mode, it is _critical_ that the version of Hadoop that is out on your cluster match what is under HBase. -Replace the hadoop jar found in the HBase lib directory with the hadoop jar you are running on your cluster to avoid version mismatch issues. -Make sure you replace the jar in HBase everywhere on your cluster. -Hadoop version mismatch issues have various manifestations but often all looks like its hung up. +Replace the hadoop jars found in the HBase lib directory with the equivalent hadoop jars from the version you are running +on your cluster to avoid version mismatch issues. +Make sure you replace the jars under HBase across your whole cluster. +Hadoop version mismatch issues have various manifestations. Check for mismatch if +HBase appears hung. ==== [[dfs.datanode.max.transfer.threads]] From 017a5746898dd4a5d4a729eb7712545bf19bea16 Mon Sep 17 00:00:00 2001 From: Mike Drob Date: Fri, 27 Apr 2018 16:23:51 -0500 Subject: [PATCH 16/27] HBASE-20502 Document necessity of YARN-7190 and non-production for Hadoop 3.1.0 Amending-Author: Sean Busbey Signed-off-by: zhangduo (cherry picked from commit 79e960f4ee7c2f10f514016a4f51a017d193d2c1) --- src/main/asciidoc/_chapters/configuration.adoc | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 0d7a371a420d..339b5418a193 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -228,7 +228,9 @@ Use the following legend to interpret this table: |Hadoop-2.8.2 | NT | NT | NT | NT | NT |Hadoop-2.8.3+ | NT | NT | NT | S | S |Hadoop-2.9.0 | X | X | X | X | X -|Hadoop-3.0.0 | NT | NT | NT | NT | NT +|Hadoop-2.9.1+ | NT | NT | NT | NT | NT +|Hadoop-3.0.x | X | X | X | X | X +|Hadoop-3.1.0 | X | X | X | X | X |=== .Hadoop Pre-2.6.1 and JDK 1.8 Kerberos @@ -255,6 +257,18 @@ data loss. This patch is present in Apache Hadoop releases 2.6.1+. Starting around the time of Hadoop version 2.7.0, the Hadoop PMC got into the habit of calling out new minor releases on their major version 2 release line as not stable / production ready. As such, HBase expressly advises downstream users to avoid running on top of these releases. Note that additionally the 2.8.1 release was given the same caveat by the Hadoop PMC. For reference, see the release announcements for link:https://s.apache.org/hadoop-2.7.0-announcement[Apache Hadoop 2.7.0], link:https://s.apache.org/hadoop-2.8.0-announcement[Apache Hadoop 2.8.0], link:https://s.apache.org/hadoop-2.8.1-announcement[Apache Hadoop 2.8.1], and link:https://s.apache.org/hadoop-2.9.0-announcement[Apache Hadoop 2.9.0]. ==== +.Hadoop 3.0.x Releases +[TIP] +==== +Hadoop distributions that include the Application Timeline Service feature may cause unexpected versions of HBase classes to be present in the application classpath. Users planning on running MapReduce applications with HBase should make sure that link:https://issues.apache.org/jira/browse/YARN-7190[YARN-7190] is present in their YARN service (currently fixed in 2.9.1+ and 3.1.0+). +==== + +.Hadoop 3.1.0 Release +[TIP] +==== +The Hadoop PMC called out the 3.1.0 release as not stable / production ready. As such, HBase expressly advises downstream users to avoid running on top of this release. For reference, see the link:https://s.apache.org/hadoop-3.1.0-announcement[release announcement for Hadoop 3.1.0]. +==== + .Replace the Hadoop Bundled With HBase! [NOTE] ==== From df602f6a1b7e1acffa4339401113e8f1455c65e1 Mon Sep 17 00:00:00 2001 From: Josh Elser Date: Mon, 3 Sep 2018 20:05:07 -0700 Subject: [PATCH 17/27] HBASE-21091 Update Hadoop and Java "supported" versions tables Amending-Author: Peter Somogyi (cherry picked from commit 7d6ce3569df88e1f974e2b5e91a79cb508863663) (cherry picked from commit 9b957d8c78c2a8d82e434eed9dbe5aeae76bc2dc) --- .../asciidoc/_chapters/configuration.adoc | 58 ++++++++++--------- 1 file changed, 30 insertions(+), 28 deletions(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 339b5418a193..8fe277e2bef8 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -112,22 +112,22 @@ HBase recommends downstream users rely on JDK releases that are marked as Long T |JDK 10 |2.0 -|link:http://search-hadoop.com/m/YGbbsPxZ723m3as[Not Supported] -|yes -|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] -|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] +|icon:times-circle[role="red"] +|icon:check-circle[role="green"] +|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] +|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] |1.3 -|yes -|yes -|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] -|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] +|icon:check-circle[role="green"] +|icon:check-circle[role="green"] +|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] +|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] |1.2 -|yes -|yes -|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] -|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported] +|icon:check-circle[role="green"] +|icon:check-circle[role="green"] +|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] +|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] |=== @@ -211,26 +211,28 @@ Use the following legend to interpret this table: .Hadoop version support matrix -* "S" = supported -* "X" = not supported -* "NT" = Not tested +* icon:check-circle[role="green"] = Tested to be fully-functional +* icon:times-circle[role="red"] = Known to not be fully-functional +* icon:exclamation-circle[role="yellow"] = Not tested, may/may-not function [cols="1,1,1,1,1,1", options="header"] |=== | | HBase-1.2.x | HBase-1.3.x | HBase-1.5.x | HBase-2.0.x | HBase-2.1.x -|Hadoop-2.4.x | S | S | X | X | X -|Hadoop-2.5.x | S | S | X | X | X -|Hadoop-2.6.0 | X | X | X | X | X -|Hadoop-2.6.1+ | S | S | X | S | X -|Hadoop-2.7.0 | X | X | X | X | X -|Hadoop-2.7.1+ | S | S | S | S | S -|Hadoop-2.8.[0-1] | X | X | X | X | X -|Hadoop-2.8.2 | NT | NT | NT | NT | NT -|Hadoop-2.8.3+ | NT | NT | NT | S | S -|Hadoop-2.9.0 | X | X | X | X | X -|Hadoop-2.9.1+ | NT | NT | NT | NT | NT -|Hadoop-3.0.x | X | X | X | X | X -|Hadoop-3.1.0 | X | X | X | X | X +|Hadoop-2.4.x | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.5.x | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.6.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.6.1+ | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] +|Hadoop-2.7.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.7.1+ | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] +|Hadoop-2.8.[0-1] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.8.2 | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] +|Hadoop-2.8.3+ | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] +|Hadoop-2.9.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.9.1+ | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] +|Hadoop-3.0.[0-2] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-3.0.3+ | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] +|Hadoop-3.1.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-3.1.1+ | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] |=== .Hadoop Pre-2.6.1 and JDK 1.8 Kerberos From b1ad70d351de4b92c092b2da3f0371ce9c10ed77 Mon Sep 17 00:00:00 2001 From: Peter Somogyi Date: Wed, 9 Jan 2019 13:59:30 +0100 Subject: [PATCH 18/27] HBASE-21295 Update compatibility matrices Signed-off-by: Sean Busbey (cherry picked from commit 2b5b190882daeb0f9f0cf5361df3525db7119bea) (cherry picked from commit aa31dbefd89a11430f38b65e8d1641f67ba9b96d) Conflicts: src/main/asciidoc/_chapters/configuration.adoc --- .../asciidoc/_chapters/configuration.adoc | 74 ++++++++++--------- 1 file changed, 40 insertions(+), 34 deletions(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 8fe277e2bef8..b04340ec5939 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -94,7 +94,10 @@ This section lists required services and some required system configuration. [[java]] .Java -The following table summarizes the recommendation of the HBase community wrt deploying on various Java versions. An entry of "yes" is meant to indicate a base level of testing and willingness to help diagnose and address issues you might run into. Similarly, an entry of "no" or "Not Supported" generally means that should you run into an issue the community is likely to ask you to change the Java environment before proceeding to help. In some cases, specific guidance on limitations (e.g. wether compiling / unit tests work, specific operational issues, etc) will also be noted. +The following table summarizes the recommendation of the HBase community wrt deploying on various Java versions. +A icon:check-circle[role="green"] symbol is meant to indicate a base level of testing and willingness to help diagnose and address issues you might run into. +Similarly, an entry of icon:exclamation-circle[role="yellow"] or icon:times-circle[role="red"] generally means that should you run into an issue the community is likely to ask you to change the Java environment before proceeding to help. +In some cases, specific guidance on limitations (e.g. whether compiling / unit tests work, specific operational issues, etc) will also be noted. .Long Term Support JDKs are recommended [TIP] @@ -103,31 +106,34 @@ HBase recommends downstream users rely on JDK releases that are marked as Long T ==== .Java support by release line -[cols="1,1,1,1,1", options="header"] +[cols="6*^.^", options="header"] |=== |HBase Version |JDK 7 |JDK 8 -|JDK 9 -|JDK 10 +|JDK 9 (Non-LTS) +|JDK 10 (Non-LTS) +|JDK 11 -|2.0 +|2.0+ |icon:times-circle[role="red"] |icon:check-circle[role="green"] -|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] -|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] - -|1.3 -|icon:check-circle[role="green"] -|icon:check-circle[role="green"] -|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] -|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] - -|1.2 +v|icon:exclamation-circle[role="yellow"] +link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] +v|icon:exclamation-circle[role="yellow"] +link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] +v|icon:exclamation-circle[role="yellow"] +link:https://issues.apache.org/jira/browse/HBASE-21110[HBASE-21110] + +|1.2+ |icon:check-circle[role="green"] |icon:check-circle[role="green"] -|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] -|icon:exclamation-circle[role="yellow"] Non-LTS, see link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] +v|icon:exclamation-circle[role="yellow"] +link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] +v|icon:exclamation-circle[role="yellow"] +link:https://issues.apache.org/jira/browse/HBASE-20264[HBASE-20264] +v|icon:exclamation-circle[role="yellow"] +link:https://issues.apache.org/jira/browse/HBASE-21110[HBASE-21110] |=== @@ -215,24 +221,24 @@ Use the following legend to interpret this table: * icon:times-circle[role="red"] = Known to not be fully-functional * icon:exclamation-circle[role="yellow"] = Not tested, may/may-not function -[cols="1,1,1,1,1,1", options="header"] +[cols="1,4*^.^", options="header"] |=== -| | HBase-1.2.x | HBase-1.3.x | HBase-1.5.x | HBase-2.0.x | HBase-2.1.x -|Hadoop-2.4.x | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.5.x | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.6.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.6.1+ | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] -|Hadoop-2.7.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.7.1+ | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] -|Hadoop-2.8.[0-1] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.8.2 | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] -|Hadoop-2.8.3+ | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] -|Hadoop-2.9.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.9.1+ | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] -|Hadoop-3.0.[0-2] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-3.0.3+ | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] -|Hadoop-3.1.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-3.1.1+ | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] +| | HBase-1.2.x, HBase-1.3.x | HBase-1.4.x | HBase-2.0.x | HBase-2.1.x +|Hadoop-2.4.x | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.5.x | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.6.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.6.1+ | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] +|Hadoop-2.7.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.7.1+ | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] +|Hadoop-2.8.[0-1] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.8.2 | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] +|Hadoop-2.8.3+ | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] +|Hadoop-2.9.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.9.1+ | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] +|Hadoop-3.0.[0-2] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-3.0.3+ | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] +|Hadoop-3.1.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-3.1.1+ | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] |=== .Hadoop Pre-2.6.1 and JDK 1.8 Kerberos From 0ec9f326cd59363e5e39771c1d755817edc1e1a6 Mon Sep 17 00:00:00 2001 From: Andrew Purtell Date: Mon, 29 Apr 2019 13:43:42 -0700 Subject: [PATCH 19/27] HBASE-22321 Add 1.5 release line to the Hadoop supported versions table Signed-off-by: Peter Somogyi (cherry picked from commit 9743b3c70d4ea4539ab5dcf89dc874f916704e71) (cherry picked from commit fb7a020492fd5871d7a47434c639e68e61cdeb21) --- src/main/asciidoc/_chapters/configuration.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index b04340ec5939..2aeefc931d82 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -223,7 +223,7 @@ Use the following legend to interpret this table: [cols="1,4*^.^", options="header"] |=== -| | HBase-1.2.x, HBase-1.3.x | HBase-1.4.x | HBase-2.0.x | HBase-2.1.x +| | HBase-1.2.x, HBase-1.3.x | HBase-1.4.x, HBase-1.5.x | HBase-2.0.x | HBase-2.1.x |Hadoop-2.4.x | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] |Hadoop-2.5.x | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] |Hadoop-2.6.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] From a88563def92c8d03a8864fc547c830bae68980e3 Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Mon, 13 May 2019 10:05:59 -0500 Subject: [PATCH 20/27] HBASE-22405 Update Ref Guide for EOL of Hadoop 2.7 Signed-off-by: Andrew Purtell (cherry picked from commit feee97fffb7a25e7a9c0343e74c44607f423b32a) (cherry picked from commit 673302504f917f9ce1a466ce1f8d511a67b6ada5) --- .../asciidoc/_chapters/configuration.adoc | 34 +++++++++---------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 2aeefc931d82..75e3e5abf0da 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -221,24 +221,24 @@ Use the following legend to interpret this table: * icon:times-circle[role="red"] = Known to not be fully-functional * icon:exclamation-circle[role="yellow"] = Not tested, may/may-not function -[cols="1,4*^.^", options="header"] +[cols="1,6*^.^", options="header"] |=== -| | HBase-1.2.x, HBase-1.3.x | HBase-1.4.x, HBase-1.5.x | HBase-2.0.x | HBase-2.1.x -|Hadoop-2.4.x | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.5.x | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.6.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.6.1+ | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] -|Hadoop-2.7.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.7.1+ | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] -|Hadoop-2.8.[0-1] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.8.2 | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] -|Hadoop-2.8.3+ | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] -|Hadoop-2.9.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-2.9.1+ | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] -|Hadoop-3.0.[0-2] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-3.0.3+ | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] -|Hadoop-3.1.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] -|Hadoop-3.1.1+ | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] +| | HBase-1.2.x, HBase-1.3.x | HBase-1.4.x | HBase-1.5.x | HBase-2.0.x | HBase-2.1.x | HBase-2.2.x +|Hadoop-2.4.x | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.5.x | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.6.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.6.1+ | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.7.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.7.1+ | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:times-circle[role="red"] +|Hadoop-2.8.[0-1] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.8.2 | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:check-circle[role="green"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:check-circle[role="green"] +|Hadoop-2.8.3+ | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] +|Hadoop-2.9.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-2.9.1+ | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] | icon:exclamation-circle[role="yellow"] +|Hadoop-3.0.[0-2] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-3.0.3+ | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] +|Hadoop-3.1.0 | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] +|Hadoop-3.1.1+ | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:times-circle[role="red"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] | icon:check-circle[role="green"] |=== .Hadoop Pre-2.6.1 and JDK 1.8 Kerberos From c8178e0ff4dd928c98ec85b74756e152c7eb4b9c Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Mon, 13 May 2019 20:08:52 -0500 Subject: [PATCH 21/27] WIP fixup to remove hadoop3 stuff --- .../src/main/resources/supplemental-models.xml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/hbase-resource-bundle/src/main/resources/supplemental-models.xml b/hbase-resource-bundle/src/main/resources/supplemental-models.xml index f3c8ec4c71a5..165e7589e118 100644 --- a/hbase-resource-bundle/src/main/resources/supplemental-models.xml +++ b/hbase-resource-bundle/src/main/resources/supplemental-models.xml @@ -1390,10 +1390,10 @@ Copyright 2006 Envoi Solutions LLC - - com.codahale.metrics - metrics-core - 3.0.1 + + com.nimbusds + nimbus-jose-jwt + 3.9 Apache License, Version 2.0 @@ -1404,7 +1404,7 @@ Copyright 2006 Envoi Solutions LLC - + com.squareup.okhttp okhttp 2.4.0 @@ -1418,7 +1418,7 @@ Copyright 2006 Envoi Solutions LLC - + com.squareup.okio okio 1.4.0 @@ -1461,7 +1461,7 @@ Copyright 2006 Envoi Solutions LLC - + de.ruedigermoeller fst 2.24 @@ -1480,7 +1480,7 @@ Copyright 2006 Envoi Solutions LLC - + org.objenesis objenesis 2.1 @@ -1631,7 +1631,7 @@ EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - + org.codehaus.woodstox stax2-api 3.1.4 From d94051f59b8bacff921b2fd4fc6b95f24799de4d Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Mon, 13 May 2019 20:11:15 -0500 Subject: [PATCH 22/27] WIP fixup removing repo branch status update --- src/main/asciidoc/_chapters/community.adoc | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/main/asciidoc/_chapters/community.adoc b/src/main/asciidoc/_chapters/community.adoc index 8f0e3479387c..4b91b0d6f3ce 100644 --- a/src/main/asciidoc/_chapters/community.adoc +++ b/src/main/asciidoc/_chapters/community.adoc @@ -62,12 +62,13 @@ Any -1 on a patch by anyone vetos a patch; it cannot be committed until the just .How to set fix version in JIRA on issue resolve Here is how link:http://search-hadoop.com/m/azemIi5RCJ1[we agreed] to set versions in JIRA when we resolve an issue. -If master is going to be 2.0.0, and branch-1 1.4.0 then: +If trunk is going to be 0.98.0 then: -* Commit only to master: Mark with 2.0.0 -* Commit to branch-1 and master: Mark with 2.0.0, and 1.4.0 -* Commit to branch-1.3, branch-1, and master: Mark with 2.0.0, 1.4.0, and 1.3.x -* Commit site fixes: no version +* Commit only to trunk: Mark with 0.98 +* Commit to 0.95 and trunk : Mark with 0.98, and 0.95.x +* Commit to 0.94.x and 0.95, and trunk: Mark with 0.98, 0.95.x, and 0.94.x +* Commit to 89-fb: Mark with 89-fb. +* Commit site fixes: no version [[hbase.when.to.close.jira]] .Policy on when to set a RESOLVED JIRA as CLOSED From 3a3f4685b697c8b2765513c4fad7c8e42f6d1f8a Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Mon, 13 May 2019 20:12:23 -0500 Subject: [PATCH 23/27] WIP fixup removing formatting fix --- src/main/asciidoc/_chapters/compression.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/asciidoc/_chapters/compression.adoc b/src/main/asciidoc/_chapters/compression.adoc index 8853d9b12d8a..78fc6a2f1efb 100644 --- a/src/main/asciidoc/_chapters/compression.adoc +++ b/src/main/asciidoc/_chapters/compression.adoc @@ -414,7 +414,7 @@ $ hbase org.apache.hadoop.hbase.util.LoadTestTool -write 1:10:100 -num_keys 1000 ==== [[data.block.encoding.enable]] -=== Enable Data Block Encoding +== Enable Data Block Encoding Codecs are built into HBase so no extra configuration is needed. Codecs are enabled on a table by setting the `DATA_BLOCK_ENCODING` property. From 67426efd8437a567d9309ad1bfcd566b7ed641c0 Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Mon, 13 May 2019 23:17:07 -0500 Subject: [PATCH 24/27] WIP mentioning java 7 for gc logs. --- src/main/asciidoc/_chapters/troubleshooting.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/asciidoc/_chapters/troubleshooting.adoc b/src/main/asciidoc/_chapters/troubleshooting.adoc index cb3342421c51..e60eb41396b4 100644 --- a/src/main/asciidoc/_chapters/troubleshooting.adoc +++ b/src/main/asciidoc/_chapters/troubleshooting.adoc @@ -103,7 +103,7 @@ To disable, set the logging level back to `INFO` level. [NOTE] ---- -All example Garbage Collection logs in this section are based on Java 8 output. The introduction of Unified Logging in Java 9 and newer will result in very different looking logs. +All example Garbage Collection logs in this section are based on Java 7 and Java 8 output. The introduction of Unified Logging in Java 9 and newer will result in very different looking logs. ---- HBase is memory intensive, and using the default GC you can see long pauses in all threads including the _Juliet Pause_ aka "GC of Death". To help debug this or confirm this is happening GC logging can be turned on in the Java virtual machine. From 65494aeecccafc4fe2f60f391277812fead18bcf Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Mon, 13 May 2019 23:40:18 -0500 Subject: [PATCH 25/27] WIP fixup removing config blob for standalone-over-hdfs since that section is missing. --- src/main/asciidoc/_chapters/configuration.adoc | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc index 75e3e5abf0da..e2013bd43760 100644 --- a/src/main/asciidoc/_chapters/configuration.adoc +++ b/src/main/asciidoc/_chapters/configuration.adoc @@ -342,20 +342,6 @@ Standalone mode is what is described in the <> section. In standalone mode, HBase does not use HDFS -- it uses the local filesystem instead -- and it runs all HBase daemons and a local ZooKeeper all up in the same JVM. Zookeeper binds to a well known port so clients may talk to HBase. -[source,xml] ----- - - - hbase.rootdir - hdfs://namenode.example.org:8020/hbase - - - hbase.cluster.distributed - false - - ----- - [[distributed]] === Distributed From 429d4ed4fee7e7110cb522de0ea470eb241bab01 Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Wed, 12 Jun 2019 21:56:43 -0500 Subject: [PATCH 26/27] WIP fix canary help based on review feedback from Peter. --- src/main/asciidoc/_chapters/ops_mgt.adoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/main/asciidoc/_chapters/ops_mgt.adoc b/src/main/asciidoc/_chapters/ops_mgt.adoc index f9edebbe45e1..fed7157a327c 100644 --- a/src/main/asciidoc/_chapters/ops_mgt.adoc +++ b/src/main/asciidoc/_chapters/ops_mgt.adoc @@ -93,6 +93,8 @@ Usage: hbase org.apache.hadoop.hbase.tool.Canary [opts] [table1 [table2]...] | [ -zookeeper Tries to grab zookeeper.znode.parent on each zookeeper instance -daemon Continuous check at defined intervals. + -permittedZookeeperFailures Ignore first N failures when attempting to + connect to individual zookeeper nodes in the ensemble -interval Interval between checks (sec) -e Use table/regionserver as regular expression which means the table/regionserver is regular expression pattern From edf99bb5dcc0d3495de61792de373ef110cf83c4 Mon Sep 17 00:00:00 2001 From: Sean Busbey Date: Wed, 12 Jun 2019 21:59:32 -0500 Subject: [PATCH 27/27] WIP update hbase command help based on review feedback from Andrew. --- src/main/asciidoc/_chapters/ops_mgt.adoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/main/asciidoc/_chapters/ops_mgt.adoc b/src/main/asciidoc/_chapters/ops_mgt.adoc index fed7157a327c..61a5e81f12bf 100644 --- a/src/main/asciidoc/_chapters/ops_mgt.adoc +++ b/src/main/asciidoc/_chapters/ops_mgt.adoc @@ -53,9 +53,11 @@ Some commands take arguments. Pass no args or -h for usage. shell Run the HBase shell hbck Run the hbase 'fsck' tool snapshot Tool for managing snapshots + snapshotinfo Tool for dumping snapshot information wal Write-ahead-log analyzer hfile Store file analyzer zkcli Run the ZooKeeper shell + upgrade Upgrade hbase master Run an HBase HMaster node regionserver Run an HBase HRegionServer node zookeeper Run a Zookeeper server @@ -65,6 +67,7 @@ Some commands take arguments. Pass no args or -h for usage. clean Run the HBase clean up script classpath Dump hbase CLASSPATH mapredcp Dump CLASSPATH entries required by mapreduce + completebulkload Run LoadIncrementalHFiles tool pe Run PerformanceEvaluation ltt Run LoadTestTool canary Run the Canary tool