[SPARK-18776][SS] Make Offset for FileStreamSource corrected formatted in json #16205
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
tdas
commented
Dec 8, 2016
| * Offset for the [[FileStreamSource]]. | ||
| * @param logOffset Position in the [[FileStreamSourceLog]] | ||
| */ | ||
| case class FileStreamSourceOffset(logOffset: Long) extends Offset { |
Contributor
Author
There was a problem hiding this comment.
I named it logoffset to make a more clear that this is a offset in the log of FileStreamSource
|
Test build #69841 has finished for PR 16205 at commit
|
Member
|
|
Member
|
Oh, please also support to read the old format in case RC2 is passed. |
Member
|
LGTM pending tests |
|
Test build #69888 has finished for PR 16205 at commit
|
Contributor
Author
|
Merging to master and 2.1 |
asfgit
pushed a commit
that referenced
this pull request
Dec 9, 2016
…d in json ## What changes were proposed in this pull request? - Changed FileStreamSource to use new FileStreamSourceOffset rather than LongOffset. The field is named as `logOffset` to make it more clear that this is a offset in the file stream log. - Fixed bug in FileStreamSourceLog, the field endId in the FileStreamSourceLog.get(startId, endId) was not being used at all. No test caught it earlier. Only my updated tests caught it. Other minor changes - Dont use batchId in the FileStreamSource, as calling it batch id is extremely miss leading. With multiple sources, it may happen that a new batch has no new data from a file source. So offset of FileStreamSource != batchId after that batch. ## How was this patch tested? Updated unit test. Author: Tathagata Das <[email protected]> Closes #16205 from tdas/SPARK-18776. (cherry picked from commit 458fa33) Signed-off-by: Tathagata Das <[email protected]>
robert3005
pushed a commit
to palantir/spark
that referenced
this pull request
Dec 15, 2016
…d in json ## What changes were proposed in this pull request? - Changed FileStreamSource to use new FileStreamSourceOffset rather than LongOffset. The field is named as `logOffset` to make it more clear that this is a offset in the file stream log. - Fixed bug in FileStreamSourceLog, the field endId in the FileStreamSourceLog.get(startId, endId) was not being used at all. No test caught it earlier. Only my updated tests caught it. Other minor changes - Dont use batchId in the FileStreamSource, as calling it batch id is extremely miss leading. With multiple sources, it may happen that a new batch has no new data from a file source. So offset of FileStreamSource != batchId after that batch. ## How was this patch tested? Updated unit test. Author: Tathagata Das <[email protected]> Closes apache#16205 from tdas/SPARK-18776.
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
…d in json ## What changes were proposed in this pull request? - Changed FileStreamSource to use new FileStreamSourceOffset rather than LongOffset. The field is named as `logOffset` to make it more clear that this is a offset in the file stream log. - Fixed bug in FileStreamSourceLog, the field endId in the FileStreamSourceLog.get(startId, endId) was not being used at all. No test caught it earlier. Only my updated tests caught it. Other minor changes - Dont use batchId in the FileStreamSource, as calling it batch id is extremely miss leading. With multiple sources, it may happen that a new batch has no new data from a file source. So offset of FileStreamSource != batchId after that batch. ## How was this patch tested? Updated unit test. Author: Tathagata Das <[email protected]> Closes apache#16205 from tdas/SPARK-18776.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
logOffsetto make it more clear that this is a offset in the file stream log.Other minor changes
How was this patch tested?
Updated unit test.