[SPARK-43281][SQL] Fix concurrent writer does not update file metrics #40952
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
cc @cloud-fan |
cloud-fan
approved these changes
Apr 26, 2023
Contributor
cloud-fan
left a comment
cloud-fan
left a comment
There was a problem hiding this comment.
LGTM. Is this a long-standing bug?
Contributor
Author
|
@cloud-fan , it happened since #32198 and with concurrent writer on. |
ulysses-you
commented
Apr 26, 2023
| val missing = new Path(tempDirPath, "missing") | ||
| val tracker = new BasicWriteTaskStatsTracker(conf) | ||
| tracker.newFile(missing.toString) | ||
| tracker.closeFile(missing.toString) |
Contributor
Author
There was a problem hiding this comment.
after refactor of #32198, one newFile should have one closeFile
Contributor
Author
|
@cloud-fan any comments ? |
Member
|
cc @mridulm , too. |
Contributor
|
thanks, merging to master/3.4! |
cloud-fan
pushed a commit
that referenced
this pull request
May 16, 2023
### What changes were proposed in this pull request? `DynamicPartitionDataConcurrentWriter` it uses temp file path to get file status after commit task. However, the temp file has already moved to new path during commit task. This pr calls `closeFile` before commit task. ### Why are the changes needed? fix bug ### Does this PR introduce _any_ user-facing change? yes, after this pr the metrics is correct ### How was this patch tested? add test Closes #40952 from ulysses-you/SPARK-43281. Authored-by: ulysses-you <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 592e922) Signed-off-by: Wenchen Fan <[email protected]>
snmvaughan
pushed a commit
to snmvaughan/spark
that referenced
this pull request
Jun 20, 2023
### What changes were proposed in this pull request? `DynamicPartitionDataConcurrentWriter` it uses temp file path to get file status after commit task. However, the temp file has already moved to new path during commit task. This pr calls `closeFile` before commit task. ### Why are the changes needed? fix bug ### Does this PR introduce _any_ user-facing change? yes, after this pr the metrics is correct ### How was this patch tested? add test Closes apache#40952 from ulysses-you/SPARK-43281. Authored-by: ulysses-you <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 592e922) Signed-off-by: Wenchen Fan <[email protected]>
GladwinLee
pushed a commit
to lyft/spark
that referenced
this pull request
Oct 10, 2023
### What changes were proposed in this pull request? `DynamicPartitionDataConcurrentWriter` it uses temp file path to get file status after commit task. However, the temp file has already moved to new path during commit task. This pr calls `closeFile` before commit task. ### Why are the changes needed? fix bug ### Does this PR introduce _any_ user-facing change? yes, after this pr the metrics is correct ### How was this patch tested? add test Closes apache#40952 from ulysses-you/SPARK-43281. Authored-by: ulysses-you <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 592e922) Signed-off-by: Wenchen Fan <[email protected]>
catalinii
pushed a commit
to lyft/spark
that referenced
this pull request
Oct 10, 2023
### What changes were proposed in this pull request? `DynamicPartitionDataConcurrentWriter` it uses temp file path to get file status after commit task. However, the temp file has already moved to new path during commit task. This pr calls `closeFile` before commit task. ### Why are the changes needed? fix bug ### Does this PR introduce _any_ user-facing change? yes, after this pr the metrics is correct ### How was this patch tested? add test Closes apache#40952 from ulysses-you/SPARK-43281. Authored-by: ulysses-you <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 592e922) Signed-off-by: Wenchen Fan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
DynamicPartitionDataConcurrentWriterit uses temp file path to get file status after commit task. However, the temp file has already moved to new path during commit task.This pr calls
closeFilebefore commit task.Why are the changes needed?
fix bug
Does this PR introduce any user-facing change?
yes, after this pr the metrics is correct
How was this patch tested?
add test