-
Couldn't load subscription status.
- Fork 3.4k
HBASE-27144 Improvement bulkload rpc #4558
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
💔 -1 overall
This message was automatically generated. |
2bdb8b3 to
6972b58
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
6972b58 to
7a19540
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
7a19540 to
46635e6
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Bulkload will consume a lot of resources in the cluster. We try to reduce the impact of bulkload on online services and do simple resource isolation for bulkload. This feature has been running online for 2 years and works well @Apache9 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall LGTM.
Just some concerns about the implementation details.
| * @param highPriorityLevel high Priority Level | ||
| * @param priority Function to extract request priority. | ||
| */ | ||
| public SimpleRpcScheduler(Configuration conf, int handlerCount, int priorityHandlerCount, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the below constructor still valid? In general I suggest we move the code in the below method to this method and make the below method call this method, with bulkLoadHandlerCount = 0, and better also make the below method as deprecated, and remove it in 3.0.0, for example, as it is IA.LimitedPrivate.
| highPriorityLevel); | ||
| } | ||
|
|
||
| public SimpleRpcScheduler(Configuration conf, int handlerCount, int priorityHandlerCount, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this method used for?
| conf.getInt(HConstants.MASTER_META_TRANSITION_HANDLER_COUNT, | ||
| HConstants.MASTER__META_TRANSITION_HANDLER_COUNT_DEFAULT), | ||
| priority, server, HConstants.QOS_THRESHOLD); | ||
| boolean bulkLoadIndependenceQueueEnabled = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need a separated config to enable this feature? I suppose we just set the handler count to 0 is enough?
46635e6 to
ffe4eb9
Compare
|
@Apache9 Edit as you suggested |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Can this PR be merged into the master branch? @Apache9 |
ffe4eb9 to
c0f5d95
Compare
|
Independent bulkload rpc is not enabled by default bulkLoadHandlerCount = 0 |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Co-authored-by: SiCheng-Zheng <[email protected]> Signed-off-by: Duo Zhang <[email protected]> (cherry picked from commit ff8eb59) Conflicts: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcScheduler.java hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AnnotationReadingPriorityFunction.java
No description provided.