-
Couldn't load subscription status.
- Fork 3.4k
HBASE-29604 BackupHFileCleaner uses flawed time based check #7360
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles. Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion.
93e93bc to
5d2e22d
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Testing failures seem unrelated to the PR. |
| // timestamp of 2nd most recent read from backup system table | ||
| private long secondPrevReadFromBackupTbl = 0; | ||
| // timestamp of most recent completed cleaning run | ||
| private volatile long previousCleaningCompletionTimestamp = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So the first run of the chore during the process lifecycle will always do nothing? Is there something else we can base it's initialization value on?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct. That was also the (intended, but not actual) behavior of this cleaner before this PR I think.
I guess we can just initialize this to X time before the current time. Could pick X rudimentary, I think a minute is sufficient, or we could set X to the interval in which the cleaner runs, if I trace the code correctly, that would be the value of hbase.master.cleaner.interval.
|
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not clear to me why the cleaner bases the HFile "grace period" threshold on the last run of the cleaner; to me it seems like just using a threshold of 60 seconds before preClean() is called (an therefore 60 seconds before the files are traversed on disk and before the backup table is scanned) would be simpler. But agree the logic here was flawed and that this seems to fix it
| private volatile long previousCleaningCompletionTimestamp = 0; | ||
|
|
||
| @Override | ||
| public void postClean() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree that this definitely looks like an oversight from the original implementation, and that this change corrects it to work as it seems it was originally intended
|
🎊 +1 overall
This message was automatically generated. |
) Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles. Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion. Signed-off-by: Nick Dimiduk <[email protected]>
|
Merged to master and backport to branch-3 is clean. Backport to branch-2 has minor conflict in the unit test. I think this means we've missed a backport along the way. It's a minor thing re: the |
Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles. Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion. Signed-off-by: Nick Dimiduk <[email protected]>
) Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles. Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion. Signed-off-by: Nick Dimiduk <[email protected]>
) Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles. Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion. Signed-off-by: Nick Dimiduk <[email protected]>
) Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles. Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion. Signed-off-by: Nick Dimiduk <[email protected]>
Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles. Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion. Signed-off-by: Nick Dimiduk <[email protected]>
Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles. Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion. Signed-off-by: Nick Dimiduk <[email protected]>
) Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles. Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion. Signed-off-by: Nick Dimiduk <[email protected]>
…ed check (apache#7360) (will be in 2.6.4) (#209) Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles. Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion. Signed-off-by: Nick Dimiduk <[email protected]> Co-authored-by: DieterDP <[email protected]>
Sorry for the late response, but I guess it's already figured out? Didn't quite get what the conflict was though. |
|
It was a simple difference in collection utilities used on those lines, new in JDK11 APIs ... I've been away from HBase too long. |
…ed check (apache#7360) (will be in 2.6.4) (#209) Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles. Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion. Signed-off-by: Nick Dimiduk <[email protected]> Co-authored-by: DieterDP <[email protected]>
Adds javadoc mentioning the concurrent usage and thread-safety need of FileCleanerDelegate#getDeletableFiles.
Fixes a potential thread-safety issue in BackupHFileCleaner: this class tracks timestamps to block the deletion of recently loaded HFiles that might be needed for backup purposes. The timestamps were being registered from inside the concurrent method, which could result in recently added files getting deleted. Moved the timestamp registration to the postClean method, which is called only a single time per cleaner run, so recently loaded HFiles are in fact protected from deletion.