[SPARK-20731][SQL] Add ability to change or omit .csv file extension in CSV Data Source #17973
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mikkokupsu
changed the title
Member
|
ok to test |
gatorsmile
reviewed
May 13, 2017
| cars.coalesce(1).write | ||
| .option("header", "true") | ||
| .option("fileExtension", ".tsv") | ||
| .option("delimiter", "\t") |
Member
There was a problem hiding this comment.
When delimiter is set to \t, is it still a CSV? : )
Author
There was a problem hiding this comment.
Sorry, I have really hard time understanding what you asking here. Could you be more specific?
Member
There was a problem hiding this comment.
I guess the whole API and implementation is called "CSV" still. Of course, you can already read/write files with different names and different delimiters. Does this matter enough to make a new option? what if I delimit with, eh, null characters?
Author
There was a problem hiding this comment.
@srowen and @gatorsmile, the reason why I pushed this pull request is actually to omit the file extension completely. I guess we can discuss the semantics of different delimiters and file formats but the whole point of this pull request was to give users the option to change a hard coded value.
Member
There was a problem hiding this comment.
I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user.
I added those extentions long ago and one of the motivation was auto detection of datasource like Haddop does (which we ended up with not adding it yet due to the cost of listing files and etc).
Member
There was a problem hiding this comment.
What is the reason why Hive introduced the conf hive.output.file.extension?
Member
There was a problem hiding this comment.
Also, what is your usage scenario? It sounds like you want to omit the extension?
Author
There was a problem hiding this comment.
@gatorsmile, Yes, my use case is to omit the extension, but I decided to make the implementation flexible i.e. .option("fileExtension", "")
Member
There was a problem hiding this comment.
Just curious what is the reason you need to omit the extension?
Author
There was a problem hiding this comment.
@gatorsmile, We provide data files to our clients and specify the file format to TAB separated. I want to avoid all confusion where someone receives just the dataset and confuses the data to be COMMA separated.
|
Test build #76896 has finished for PR 17973 at commit
|
Member
|
@mikkokupsu We do not plan to support this after an offline discussion with the other Committers. Thanks for your contribution. Could you close this PR please? |
|
Can one of the admins verify this patch? |
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Dec 18, 2024
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Dec 20, 2024
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Dec 20, 2024
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Dec 20, 2024
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Jan 9, 2025
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Jan 9, 2025
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Jan 9, 2025
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Jan 9, 2025
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Jan 9, 2025
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Jan 9, 2025
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Jan 9, 2025
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
jabbaugh
pushed a commit
to jabbaugh/spark
that referenced
this pull request
Jan 9, 2025
What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request apache#17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. Does this PR introduce any user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. How was this patch tested? One unit test was added to validate a file is written with the new extension. Was this patch authored or co-authored using generative AI tooling? No
dongjoon-hyun
pushed a commit
that referenced
this pull request
Jan 10, 2025
### What changes were proposed in this pull request? The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file). ### Why are the changes needed? This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...). Notes on Previous Pull Request #17973 A pull request adding this option was discussed 7 years ago. One reason it wasn't added was: "I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user." I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension. ### Does this PR introduce _any_ user-facing change? Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option. ### How was this patch tested? One unit test was added to validate a file is written with the new extension. ### Was this patch authored or co-authored using generative AI tooling? No Closes #49233 from jabbaugh/jbaugh-add-csv-file-ext. Authored-by: Jim Baugh <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add new option to CSV Data Source to make changing hard coded ".csv" file extension to something else.
https://issues.apache.org/jira/browse/SPARK-20731
How was this patch tested?
Unit tests.