HBASE-29427 Merge all commits related to custom tiering into the feature branch #7124

wchevreuil · 2025-06-26T15:36:56Z

No description provided.

Copilot

Pull Request Overview

This PR merges all custom cell‐based tiering functionality into the feature branch, introducing new compaction policies, writers, store engines, data‐tiering abstractions, and accompanying tests.

Introduces a pluggable TieringValueProvider and CustomTieredCompactor to drive compactions by per‐cell cut‐off values
Refactors existing date‐tiered compaction into reusable components and adds hooks for custom tiering
Updates HFile, block cache, and DataTieringManager to track and honor custom time ranges

Reviewed Changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCustomCellTieredCompactor.java	Adds integration test for custom cell tiered compaction.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java	Updates tests for exception expectations and env initialization.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCustomCellTieredCompactionPolicy.java	Adds unit tests for custom compaction policy boundaries.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCustomCellDataTieringManager.java	Adds extensive tests for custom cell data‐tiering manager.
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileInlineToRootChunkConversion.java	Fixes type usage in HFile inline‐to‐root chunk conversion test.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactor.java	Exposes `needEmptyFile` and adds `createMultiWriter` hook.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.java	Refactors `shouldPerformMajorCompaction` into helper methods.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CustomTieredCompactor.java	Implements custom tiered compactor and cell decoration hook.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CustomTieredCompactionPolicy.java	Implements custom-tier compaction policy based on metadata.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CustomCellTieringValueProvider.java	Provides cell‐qualifier based tiering‐value provider.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CustomCellTieredUtils.java	Validates required qualifier config when altering tables.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java	Adds `decorateCells` override for pre‐append processing.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java	Adds `appendCustomCellTimestampsToMetadata` support.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DateTieredStoreEngine.java	Introduces `createCompactionPolicy` via reflection.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DateTieredMultiFileWriter.java	Injects a tiering‐function for boundary decisions.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringType.java	Extends enum to hold `DataTiering` instances.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java	Refactors `isHotData` logic to use generic timestamp providers.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTiering.java	Introduces `DataTiering` interface.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CustomTieringMultiFileWriter.java	Tracks per‐tier time ranges and writes custom metadata.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CustomTiering.java	Implements `DataTiering` for custom cell timestamps.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CustomTieredStoreEngine.java	Wires up custom compaction policy, compactor, and flusher.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CellTSTiering.java	Provides cell-timestamp tiering for date‐only tiering.
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java	Enforces custom-cell config on table modification.
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java	Enforces custom-cell config on table creation.
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java	Updates eviction logic to honor custom cold file lists.
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java	Integrates custom timestamp tracking and cache decisions.
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java	Adjusts read caching to use `cacheOnRead` flag.
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java	Adds new metadata append API for custom timestamps.
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java	Harmonizes block-cache signatures.
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java	Skips double‐caching for data blocks when bucket cache present.
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java	Updates default `shouldCacheBlock` signature.
hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java	Adds `CELL_VALUE_TIERING_TAG_TYPE` tag constant.

Comments suppressed due to low confidence (3)

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java:318

[nitpick] Using checkForModifyTable in a create‐table context can be confusing. Consider renaming to a more generic validateCustomCellTieringConfig to reflect both create and modify scenarios.

    CustomCellTieredUtils.checkForModifyTable(tableDescriptor);

hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileInlineToRootChunkConversion.java:89

The builder’s setType expects a byte code, not an enum. Use Cell.Type.Put.getCode() or the appropriate type code to avoid a compilation or runtime mismatch.

        .setValue(v).setType(Cell.Type.Put).build());

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:257

Comparing path strings may miss scheme or authority differences. Prefer using file.getPath().equals(hFilePath) or Path#equals for accurate Path comparison.

      if (file.getPath().toUri().getPath().toString().equals(hFilePath.toString())) {

Copilot · 2025-07-08T00:13:31Z

...in/java/org/apache/hadoop/hbase/regionserver/compactions/CustomCellTieringValueProvider.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */


[nitpick] There’s a duplicated license header; remove the second block to keep the file clean and avoid confusion.

Suggested change

/*

* Licensed to the Apache Software Foundation (ASF) under one

* or more contributor license agreements. See the NOTICE file

* distributed with this work for additional information

* regarding copyright ownership. The ASF licenses this file

* to you under the Apache License, Version 2.0 (the

* "License"); you may not use this file except in compliance

* with the License. You may obtain a copy of the License at

*

* http://www.apache.org/licenses/LICENSE-2.0

*

* Unless required by applicable law or agreed to in writing, software

* distributed under the License is distributed on an "AS IS" BASIS,

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

* See the License for the specific language governing permissions and

* limitations under the License.

*/

looks like we need to fix the redundant

taklwu

I need another round of review....but here is the first set of comments

taklwu · 2025-07-08T03:39:56Z

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java

      || (prefetchOnOpen && (category != BlockCategory.META && category != BlockCategory.UNKNOWN));
  }

  public boolean shouldCacheBlockOnRead(BlockCategory category, HFileInfo hFileInfo,


is this branch based on master? I didn't find this function on the master branch.

Yeah, this is based out of the HBASE-28463 branch, which was last rebased with master around 23/06/2025. The reason this method is not in master branch is because it was added on this HBASE-28463 branch by HBASE-28467.

taklwu · 2025-07-08T03:41:05Z

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java

+    /**
+     * Add Custom cell timestamp to Metadata
+     */
+    public void appendCustomCellTimestampsToMetadata(TimeRangeTracker timeRangeTracker)


nit: why do we need TimeRangeTracker instead of max timestamp here?

We need min and max to calculate compaction boundaries properly, as we can have files where the max TS is higher than the threshold, but the min is lower. In such cases, the tiered compaction should know it has to split the file.

taklwu · 2025-07-08T03:43:06Z

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java

        // Cache the block if necessary
        cacheConf.getBlockCache().ifPresent(cache -> {
-          if (cacheBlock && cacheConf.shouldCacheBlockOnRead(category)) {
+          if (cacheBlock && cacheOnRead) {


nit: we change from cacheConf.shouldCacheBlockOnRead(category) to cacheOnRead = cacheConf.shouldCacheBlockOnRead(category, getHFileInfo(), conf); , is it right about only caching DATA block?

is it right about only caching DATA block?

We are not caching only DATA blocks here. We are actually applying the DataTieringManager for DATA blocks only. If the block type isn't DATA, we just apply the shouldCacheBlockOnRead(BlockCategory category) logic (as we call it at line #292).

taklwu · 2025-07-08T03:43:58Z

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java

    this.path = path;
    this.name = path != null ? path.getName() : outputStream.toString();
    this.hFileContext = fileContext;
+    // TODO: Move this back to upper layer


nit : should we delete this TODO ? or should we have new JIRA for it?

Yeah, added this back then because I wasn't happy with adding time range tracking logic into HFileWriterImpl (originally, it was in StoreFileWriter). This was introduced here by HBASE-28469, IIRC, as a mean to track dates also when flushing, rather than at compaction only.

I think we should keep the TODO and revisit this code later on a separate JIRA. We still have to work on how to track custom tiering value on flushes, maybe that would be a good time to solve this TODO as well.

taklwu · 2025-07-08T03:45:27Z

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java

+    // TODO: The StoreFileReader always converts the byte[] to TimeRange
+    // via TimeRangeTracker, so we should write the serialization data of TimeRange directly.


nit: new JIRA need?

This whole code (TODO comments included) came together from StoreFileWriter with the changes of HBASE-28469.
I think this TODO refers to a nice to have that would easy debugging. I guess a separate, minor priority JIRA.

taklwu · 2025-07-08T03:54:30Z

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java

    HStore hStore = getHStore(hFilePath);
    for (HStoreFile file : hStore.getStorefiles()) {
-      if (file.getPath().equals(hFilePath)) {
+      if (file.getPath().toUri().getPath().toString().equals(hFilePath.toString())) {


nit: doesn't getPath.toString would first check this.uri.getPath() != null and append URI before compare?

@jhungund would you remember on the context of this change here? I guess this was added at some point to address UT failures, let me see if I can recover any discussion around this change.

taklwu · 2025-07-08T03:56:10Z

...in/java/org/apache/hadoop/hbase/regionserver/compactions/CustomCellTieringValueProvider.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */


looks like we need to fix the redundant

...main/java/org/apache/hadoop/hbase/regionserver/compactions/CustomTieredCompactionPolicy.java

taklwu

few more minor comments, otherwise LGTM

...in/java/org/apache/hadoop/hbase/regionserver/compactions/CustomCellTieringValueProvider.java

taklwu · 2025-07-09T03:15:50Z

...in/java/org/apache/hadoop/hbase/regionserver/compactions/CustomCellTieringValueProvider.java

+  @Override
+  public List<ExtendedCell> decorateCells(List<ExtendedCell> cells) {
+    // if no tiering qualifier properly set, skips the whole flow
+    if (tieringQualifier != null) {


nit: doesn't it tieringQualifier is always non-null this is class such that we don't need the first if-condition check ?

It would be null in case of miss configuration, when the TIERING_CELL_QUALIFIER property is missed in the configuration.

taklwu · 2025-07-09T03:18:25Z

...er/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CustomTieredCompactor.java

+      needEmptyFile(request), CustomTieredCompactor.this.tieringValueProvider::getTieringValue);
+  }
+
+  public interface TieringValueProvider {


nit: why do we want to have interface within this class?

I thought put it there since it's a simple three method interface with a tight coupling to the custom tiered compactor. Can move to it's own file, if you feel it's needed.

taklwu · 2025-07-09T03:20:58Z

...c/main/java/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.java

+          (lowerWindowIndex < 0) ? Math.abs(lowerWindowIndex + 2) : lowerWindowIndex;
+        upperWindowIndex =
+          (upperWindowIndex < 0) ? Math.abs(upperWindowIndex + 2) : upperWindowIndex;
+        if (lowerWindowIndex != upperWindowIndex) {


nit: do we also need check if lowerWindowIndex < upperWindowIndex ?

This specific logic was not introduced within this commit, it was already like that since HBASE-15400, it just appears in this PR because we split shouldPerformMajorCompaction into some extra granular methods for code reuse (i.e isMajorCompactionTime, checkForTtl, isMajorOrBulkloadResult and checkBlockLocality), as we may leverage those in overrides for shouldPerformMajorCompaction (as we do in CustomDateTieredCompactionPolicy).

taklwu

LGTM

nit: please check javadoc and checkstyle , and maybe tell if those unit tests should fail.

…ues other than cell timestamp Change-Id: I9574d01c5cb88d1ba35db3b0970d2d710207fdf4

Change-Id: I6eff2199afbc8c02426b07b696c18af258bcb36c

Co-authored-by: Wellington Ramos Chevreuil <[email protected]>

…pactionPolicy

…pactions

Change-Id: I85b7529da3ba5d0d734f98d2db224740f4ced26f

…d compaction related classes

…st master (and fix conflicts) Change-Id: I0704c8f060d06c9dc7b43d8d7c81a3274d8fd333

Change-Id: I434e61aafd31a0c47f656e23eee5239aef7910f1

Change-Id: I992f3eebe00943c4f073d1c7ea96010db21d07e6

Change-Id: I49d287cddf81170514474b27a642c249a7bc79d7

Apache-HBase · 2025-07-15T00:08:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ HBASE-28463 Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 31s		HBASE-28463 passed
+1 💚	compile	5m 25s		HBASE-28463 passed
+1 💚	checkstyle	1m 11s		HBASE-28463 passed
+1 💚	spotbugs	2m 40s		HBASE-28463 passed
-1 ❌	spotless	0m 54s		branch has 50 errors when running spotless:check, run spotless:apply to fix.
-0 ⚠️	patch	1m 8s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	4m 21s		the patch passed
+1 💚	compile	5m 19s		the patch passed
+1 💚	javac	5m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 49s	/results-checkstyle-hbase-server.txt	hbase-server: The patch generated 7 new + 25 unchanged - 1 fixed = 32 total (was 26)
+1 💚	spotbugs	2m 54s		the patch passed
+1 💚	hadoopcheck	17m 50s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	1m 14s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		60m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7124/9/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7124
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 7aa6d4c54cad 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-28463 / `7496a26`
Default Java	Eclipse Adoptium-17.0.11+9
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7124/9/artifact/yetus-general-check/output/branch-spotless.txt
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7124/9/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-07-15T03:51:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ HBASE-28463 Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 38s		HBASE-28463 passed
+1 💚	compile	1m 44s		HBASE-28463 passed
+1 💚	javadoc	0m 57s		HBASE-28463 passed
+1 💚	shadedjars	7m 57s		branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	patch	8m 17s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s		Maven dependency ordering for patch
+1 💚	mvninstall	4m 31s		the patch passed
+1 💚	compile	1m 38s		the patch passed
+1 💚	javac	1m 38s		the patch passed
+1 💚	javadoc	0m 52s		the patch passed
+1 💚	shadedjars	7m 35s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	2m 52s		hbase-common in the patch passed.
+1 💚	unit	241m 51s		hbase-server in the patch passed.
		280m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7124/9/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7124
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 7d73fc149223 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-28463 / `7496a26`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7124/9/testReport/
Max. process+thread count	5165 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7124/9/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

…ure branch (#7124) This is the whole custom tiering implementation and involves the following individual works: * HBASE-29412 Extend date tiered compaction to allow for tiering by values other than cell timestamp * HBASE-29413 Implement a custom qualifier tiered compaction * HBASE-29414 Refactor DataTieringManager to make priority logic pluggable * HBASE-29422 Implement selectMinorCompation in CustomCellDateTieredCompactionPolicy * HBASE-29424 Implement configuration validation for custom tiering compactions * HBASE-29425 Refine and polish code * HBASE-29426 Define a tiering value provider and refactor custom tiered compaction related classes * HBASE-28463 Rebase time based priority branch (HBASE-28463) with latest master (and fix conflicts) Co-authored-by: Janardhan Hungund <[email protected]> Signed-off-by: Tak Lon (Stephen) Wu <[email protected]>

…ure branch (apache#7124) This is the whole custom tiering implementation and involves the following individual works: * HBASE-29412 Extend date tiered compaction to allow for tiering by values other than cell timestamp * HBASE-29413 Implement a custom qualifier tiered compaction * HBASE-29414 Refactor DataTieringManager to make priority logic pluggable * HBASE-29422 Implement selectMinorCompation in CustomCellDateTieredCompactionPolicy * HBASE-29424 Implement configuration validation for custom tiering compactions * HBASE-29425 Refine and polish code * HBASE-29426 Define a tiering value provider and refactor custom tiered compaction related classes * HBASE-28463 Rebase time based priority branch (HBASE-28463) with latest master (and fix conflicts) Co-authored-by: Janardhan Hungund <[email protected]> Signed-off-by: Tak Lon (Stephen) Wu <[email protected]> Change-Id: Ib10409b23a8cb735af5210e5ae4fc843b04b2d10

…ure branch (#7124) This is the whole custom tiering implementation and involves the following individual works: * HBASE-29412 Extend date tiered compaction to allow for tiering by values other than cell timestamp * HBASE-29413 Implement a custom qualifier tiered compaction * HBASE-29414 Refactor DataTieringManager to make priority logic pluggable * HBASE-29422 Implement selectMinorCompation in CustomCellDateTieredCompactionPolicy * HBASE-29424 Implement configuration validation for custom tiering compactions * HBASE-29425 Refine and polish code * HBASE-29426 Define a tiering value provider and refactor custom tiered compaction related classes * HBASE-28463 Rebase time based priority branch (HBASE-28463) with latest master (and fix conflicts) Co-authored-by: Janardhan Hungund <[email protected]> Signed-off-by: Tak Lon (Stephen) Wu <[email protected]>

…ure branch (apache#7124) This is the whole custom tiering implementation and involves the following individual works: * HBASE-29412 Extend date tiered compaction to allow for tiering by values other than cell timestamp * HBASE-29413 Implement a custom qualifier tiered compaction * HBASE-29414 Refactor DataTieringManager to make priority logic pluggable * HBASE-29422 Implement selectMinorCompation in CustomCellDateTieredCompactionPolicy * HBASE-29424 Implement configuration validation for custom tiering compactions * HBASE-29425 Refine and polish code * HBASE-29426 Define a tiering value provider and refactor custom tiered compaction related classes * HBASE-28463 Rebase time based priority branch (HBASE-28463) with latest master (and fix conflicts) Co-authored-by: Janardhan Hungund <[email protected]> Signed-off-by: Tak Lon (Stephen) Wu <[email protected]> Change-Id: Ib10409b23a8cb735af5210e5ae4fc843b04b2d10

…ure branch (apache#7124) This is the whole custom tiering implementation and involves the following individual works: * HBASE-29412 Extend date tiered compaction to allow for tiering by values other than cell timestamp * HBASE-29413 Implement a custom qualifier tiered compaction * HBASE-29414 Refactor DataTieringManager to make priority logic pluggable * HBASE-29422 Implement selectMinorCompation in CustomCellDateTieredCompactionPolicy * HBASE-29424 Implement configuration validation for custom tiering compactions * HBASE-29425 Refine and polish code * HBASE-29426 Define a tiering value provider and refactor custom tiered compaction related classes * HBASE-28463 Rebase time based priority branch (HBASE-28463) with latest master (and fix conflicts) Co-authored-by: Janardhan Hungund <[email protected]> Signed-off-by: Tak Lon (Stephen) Wu <[email protected]>

…ure branch (#7124) This is the whole custom tiering implementation and involves the following individual works: * HBASE-29412 Extend date tiered compaction to allow for tiering by values other than cell timestamp * HBASE-29413 Implement a custom qualifier tiered compaction * HBASE-29414 Refactor DataTieringManager to make priority logic pluggable * HBASE-29422 Implement selectMinorCompation in CustomCellDateTieredCompactionPolicy * HBASE-29424 Implement configuration validation for custom tiering compactions * HBASE-29425 Refine and polish code * HBASE-29426 Define a tiering value provider and refactor custom tiered compaction related classes * HBASE-28463 Rebase time based priority branch (HBASE-28463) with latest master (and fix conflicts) Co-authored-by: Janardhan Hungund <[email protected]> Signed-off-by: Tak Lon (Stephen) Wu <[email protected]>

…ure branch (apache#7124) This is the whole custom tiering implementation and involves the following individual works: * HBASE-29412 Extend date tiered compaction to allow for tiering by values other than cell timestamp * HBASE-29413 Implement a custom qualifier tiered compaction * HBASE-29414 Refactor DataTieringManager to make priority logic pluggable * HBASE-29422 Implement selectMinorCompation in CustomCellDateTieredCompactionPolicy * HBASE-29424 Implement configuration validation for custom tiering compactions * HBASE-29425 Refine and polish code * HBASE-29426 Define a tiering value provider and refactor custom tiered compaction related classes * HBASE-28463 Rebase time based priority branch (HBASE-28463) with latest master (and fix conflicts) Co-authored-by: Janardhan Hungund <[email protected]> Signed-off-by: Tak Lon (Stephen) Wu <[email protected]> Change-Id: I5dc170e31e47b297e1d81f07d625f680c5e0d848

…ure branch (#7124) This is the whole custom tiering implementation and involves the following individual works: * HBASE-29412 Extend date tiered compaction to allow for tiering by values other than cell timestamp * HBASE-29413 Implement a custom qualifier tiered compaction * HBASE-29414 Refactor DataTieringManager to make priority logic pluggable * HBASE-29422 Implement selectMinorCompation in CustomCellDateTieredCompactionPolicy * HBASE-29424 Implement configuration validation for custom tiering compactions * HBASE-29425 Refine and polish code * HBASE-29426 Define a tiering value provider and refactor custom tiered compaction related classes * HBASE-28463 Rebase time based priority branch (HBASE-28463) with latest master (and fix conflicts) Co-authored-by: Janardhan Hungund <[email protected]> Signed-off-by: Tak Lon (Stephen) Wu <[email protected]> Change-Id: I5dc170e31e47b297e1d81f07d625f680c5e0d848

This comment has been minimized.

Sign in to view

wchevreuil force-pushed the HBASE-29427 branch from 25f56b3 to df05f01 Compare June 26, 2025 21:34

This comment has been minimized.

Sign in to view

wchevreuil force-pushed the HBASE-29427 branch from df05f01 to 2783d1f Compare June 27, 2025 10:07

This comment has been minimized.

Sign in to view

taklwu requested a review from Copilot July 8, 2025 00:10

Copilot AI reviewed Jul 8, 2025

View reviewed changes

taklwu reviewed Jul 8, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

taklwu reviewed Jul 9, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

taklwu approved these changes Jul 9, 2025

View reviewed changes

wchevreuil force-pushed the HBASE-29427 branch from ee456d4 to 316da78 Compare July 10, 2025 13:16

wchevreuil force-pushed the HBASE-28463 branch from cdd316b to 09c8e85 Compare July 11, 2025 22:10

wchevreuil and others added 9 commits July 11, 2025 23:11

HBASE-29412 Extend date tiered compaction to allow for tiering by val…

5f20c0f

…ues other than cell timestamp Change-Id: I9574d01c5cb88d1ba35db3b0970d2d710207fdf4

HBASE-29413 Implement a custom qualifier tiered compaction

edddf3e

Change-Id: I6eff2199afbc8c02426b07b696c18af258bcb36c

HBASE-29414 Refactor DataTieringManager to make priority logic pluggable

2793766

Co-authored-by: Wellington Ramos Chevreuil <[email protected]>

HBASE-29422 Implement selectMinorCompation in CustomCellDateTieredCom…

2398e3f

…pactionPolicy

HBASE-29424 Implement configuration validation for custom tiering com…

9946e36

…pactions

HBASE-29425 Refine and polish code

c967603

Change-Id: I85b7529da3ba5d0d734f98d2db224740f4ced26f

HBASE-29426 Define a tiering value provider and refactor custom tiere…

5a0a3a6

…d compaction related classes

HBASE-28463 Rebase time based priority branch (HBASE-28463) with late…

adb40a4

…st master (and fix conflicts) Change-Id: I0704c8f060d06c9dc7b43d8d7c81a3274d8fd333

addressing review comments

2833205

Change-Id: I434e61aafd31a0c47f656e23eee5239aef7910f1

wchevreuil added 2 commits July 11, 2025 23:11

addressing another round of reviews and spotless failures

63a290a

Change-Id: I992f3eebe00943c4f073d1c7ea96010db21d07e6

addressing javadoc, checkstyle and spotless issues

7496a26

Change-Id: I49d287cddf81170514474b27a642c249a7bc79d7

wchevreuil force-pushed the HBASE-29427 branch from 316da78 to 7496a26 Compare July 11, 2025 22:12

This comment has been minimized.

Sign in to view

wchevreuil merged commit efa1ba3 into apache:HBASE-28463 Jul 15, 2025
1 check failed

taklwu mentioned this pull request Aug 13, 2025

HBASE-28463 Time Based Priority for BucketCache #7192

Merged

		// TODO: The StoreFileReader always converts the byte[] to TimeRange
		// via TimeRangeTracker, so we should write the serialization data of TimeRange directly.

HBASE-29427 Merge all commits related to custom tiering into the feature branch #7124

HBASE-29427 Merge all commits related to custom tiering into the feature branch #7124

Uh oh!

Conversation

wchevreuil commented Jun 26, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taklwu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wchevreuil Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

taklwu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

wchevreuil Jul 8, 2025 •

edited

Loading