HBASE-28988 Enhance WALPlayer for restore of BulkLoad #6523
+180
−18
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Need to update for newly suggested backup directory structure |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@ankitsol You need to run |
There was a problem hiding this comment.
In general, I found the following issues:
- All the new log lines are at the INFO level, which may not be necessary. Consider reducing them to DEBUG or TRACE in some cases.
- Javadoc/comments have not been updated to reflect the latest changes.
- There are code style issues that need to be fixed so we can run the unit tests and update them if necessary.
- New unit tests need to be added.
| */ | ||
| protected static class WALMapper | ||
| extends Mapper<WALKey, WALEdit, ImmutableBytesWritable, Mutation> { | ||
| extends Mapper<WALKey, WALEdit, ImmutableBytesWritable, Pair<Mutation, List<String>>> { |
There was a problem hiding this comment.
Instead of Pair here, can we use a Custom Class? So that The exclusivity between Mutation and BulkLoadFiles is enforced programmatically.
|
|
||
| // Retrieve configuration and set up file systems for backup and staging locations | ||
| Configuration conf = context.getConfiguration(); | ||
| Path backupLocation = new Path(conf.get(BULKLOAD_BACKUP_LOCATION)); |
There was a problem hiding this comment.
check for if backupLocation is not specified.
|
|
||
| try { | ||
| for (String file : bulkloadFilesWithFullPath) { | ||
| // Full file path from S3 |
There was a problem hiding this comment.
remove the hardcoded S3 here
| List<String> stagingPaths = new ArrayList<>(); | ||
|
|
||
| try { | ||
| for (String file : bulkloadFilesWithFullPath) { |
There was a problem hiding this comment.
these are not full paths, but the relative paths from namespace
| for (ExtendedCell cell : WALEditInternalHelper.getExtendedCells(value)) { | ||
| context.getCounter(Counter.CELLS_READ).increment(1); | ||
|
|
||
| if (CellUtil.matchingQualifier(cell, WALEdit.BULK_LOAD)) { |
There was a problem hiding this comment.
I believe the processing of bulkloaded files can be simplified, and we could reduce the log level from INFO to DEBUG or TRACE in some cases.
| setupTime(conf, WALInputFormat.START_TIME_KEY); | ||
| setupTime(conf, WALInputFormat.END_TIME_KEY); | ||
| String inputDirs = args[0]; | ||
| String walDir = new Path(inputDirs, "WALs").toString(); |
There was a problem hiding this comment.
I don't think this is correct. We are hard-coding the directories here.
We could introduce a new optional parameter that the user can specify if they have bulkloaded files for us to process.
For example:
hbase org.apache.hadoop.hbase.mapreduce.WALPlayer /backuplogdir oldTable1,oldTable2 newTable1,newTable2 -Dwal.bulk.backup.location=/bulkload-files-dir
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
As discussed internally we need to add this enhancement outside of the One thing I don't understand reading this patch: how the original code skipped BulkLoad edits and how the new code enabled them? |
anmolnar
force-pushed
the
HBASE-28957
branch
from
7eb6a76 to
71efe29
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enhance WALPlayer for restore of BulkLoad WAL entries
https://issues.apache.org/jira/browse/HBASE-28988