HBASE-27686: Recovery of BucketCache and Prefetched data after RS Crash #5080
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
wchevreuil
reviewed
Mar 6, 2023
There was a problem hiding this comment.
I think this is a great, yet simple solution for the persistent cache map. We could also solve the problem of evicted blocks by setting cacheDirty to true after we remove cache keys from the map here.
Comment on lines
475
to
477
| if (ioEngine.isPersistent()){ | ||
| setCacheDirty(true); | ||
| } |
There was a problem hiding this comment.
At this point we are not guaranteed to have added the block, as it's just sent to the writer threads. We should be doing this here instead.
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCachePersister.java
Show resolved
Hide resolved
| private final BucketCacheStats cacheStats = new BucketCacheStats(); | ||
|
|
||
| private final String persistencePath; | ||
| boolean isCacheDirty; |
| this.backingMap = new ConcurrentHashMap<>((int) blockNumCapacity); | ||
|
|
||
| if (ioEngine.isPersistent() && persistencePath != null) { | ||
| BucketCachePersister cachePersister = new BucketCachePersister(this, 1000); |
There was a problem hiding this comment.
Make this interval configurable. Maybe worth decreasing this default.
| Thread.sleep(1000); | ||
| } | ||
| // Kill the RS | ||
| cluster.killRegionServer(cluster.getRegionServer(0).getServerName()); |
There was a problem hiding this comment.
I had noticed that even when using this killRegionServer, we still go through the normal shutdown path for block cache, so we can't really rely on this. But since the main logic we want to test is that updates happen while the RS is running, we could check for the file existence without killing the RS. Also, can we validate the contents of the persisted file? At this point, I think we should certify that all the cache entries are there.
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
wchevreuil
reviewed
Mar 14, 2023
| prefetchPersistencePath = testDir + "/prefetch.persistence"; | ||
| conf.set(CacheConfig.PREFETCH_PERSISTENCE_PATH_KEY, prefetchPersistencePath); | ||
| FileSystem fs = HFileSystem.get(conf); | ||
| long BUCKETCACHE_PERSIST_INTERVAL = 3000; |
There was a problem hiding this comment.
Can we decrease this interval to a sub-second value and make this test more expedite?
There was a problem hiding this comment.
So writeStoreFile() and readStoreFile() methods take at least 2400 millis. So I used a value higher than that.
| prefetchPersistencePath = testDir + "/prefetch.persistence"; | ||
| conf.set(CacheConfig.PREFETCH_PERSISTENCE_PATH_KEY, prefetchPersistencePath); | ||
| FileSystem fs = HFileSystem.get(conf); | ||
| long BUCKETCACHE_PERSIST_INTERVAL = 3000; |
There was a problem hiding this comment.
Let's use camel case for local variable naming.
| Path storeFile2 = writeStoreFile("TestPrefetch1", conf, cacheConf, fs); | ||
| readStoreFile(storeFile, 0, fs, cacheConf, conf, bucketCache); | ||
| readStoreFile(storeFile2, 0, fs, cacheConf, conf, bucketCache); | ||
| Thread.sleep(BUCKETCACHE_PERSIST_INTERVAL - startTime + System.currentTimeMillis() + 100); |
There was a problem hiding this comment.
So readStoreFile makes sure to only return once prefetch is complete. Do we still this extra (- startTime + System.currentTimeMillis()), plus the 100 milis to pass this test?
| final int writerQLen = BucketCache.DEFAULT_WRITER_QUEUE_ITEMS; | ||
|
|
||
| @Test | ||
| public void testPrefetchPersistenceCrash() throws Exception { |
There was a problem hiding this comment.
The two test methods are almost identical. Can we do some code reuse with some lambdas for the different pre-conditions/predicates?
|
Lets also update the flag for evictions. That way, we would sort the need for update on evictions as well. |
Kota-SH
force-pushed
the
hbase-27686
branch
2 times, most recently
from
e252244 to
52b230b
Compare
|
💔 -1 overall
This message was automatically generated. |
wchevreuil
reviewed
Mar 15, 2023
| cache.setCacheInconsistent(false); | ||
| } | ||
| } catch (IOException | InterruptedException e) { | ||
| LOG.info("Exception in BucketCachePersister" + e.getMessage()); |
| private static final Logger LOG = LoggerFactory.getLogger(BucketCachePersister.class); | ||
|
|
||
| public BucketCachePersister(BucketCache cache, long intervalMillis) { | ||
| this.cache = cache; |
There was a problem hiding this comment.
Nit: Let's set a meaningful thread name for this one.
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
wchevreuil
approved these changes
Mar 16, 2023
wchevreuil
pushed a commit
that referenced
this pull request
Mar 16, 2023
…sh (#5080) Signed-off-by: Wellington Ramos Chevreuil <[email protected]> (cherry picked from commit 58cb1f4)
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
vinayakphegde
pushed a commit
to vinayakphegde/hbase
that referenced
this pull request
Apr 4, 2024
…tched data after RS Crash (apache#5080) Signed-off-by: Wellington Ramos Chevreuil <[email protected]> (cherry picked from commit 58cb1f4) Change-Id: I0d446b51f16f11d91ab8a6619ce5b01f8de7da91
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HBASE-27686