[SPARK-17235][SQL] Support purging of old logs in MetadataLog #14802
Conversation
|
|
||
| def testManager(basePath: Path, fm: FileManager): Unit = { | ||
| /** Basic test case for [[FileManager]] implementation. */ | ||
| private def testFileManager(basePath: Path, fm: FileManager): Unit = { |
There was a problem hiding this comment.
I renamed this because initially I thought it's a noun meaning "manager for testing", rather than "to test the file manager".
|
Looks like this is a little similar to this one #13513 . |
|
Test build #64403 has finished for PR 14802 at commit
|
|
It would be great if we can reuse codes in |
|
@zsxwing yup I plan to consolidate them. |
|
@petermaxlee mind to submit the consolidation PR instead when you finish? |
|
I can but I'm doing a lot of work in this area and it is a lot more difficult since they have dependencies. It would be better to merge the logically atomic pull requests. |
|
LGTM. I have written nearly the exact same thing as part of [https://github.com//pull/14553], but can use this version of the method instead. |
|
Alright I'm going to merge this in master/2.0. @petermaxlee and @frreiss can you guys work together? |
## What changes were proposed in this pull request? This patch adds a purge interface to MetadataLog, and an implementation in HDFSMetadataLog. The purge function is currently unused, but I will use it to purge old execution and file source logs in follow-up patches. These changes are required in a production structured streaming job that runs for a long period of time. ## How was this patch tested? Added a unit test case in HDFSMetadataLogSuite. Author: petermaxlee <[email protected]> Closes #14802 from petermaxlee/SPARK-17235. (cherry picked from commit f64a1dd) Signed-off-by: Reynold Xin <[email protected]>
6 participants
What changes were proposed in this pull request?
This patch adds a purge interface to MetadataLog, and an implementation in HDFSMetadataLog. The purge function is currently unused, but I will use it to purge old execution and file source logs in follow-up patches. These changes are required in a production structured streaming job that runs for a long period of time.
How was this patch tested?
Added a unit test case in HDFSMetadataLogSuite.