[SPARK-17287] [PYSPARK] Add recursive kwarg to Python SparkContext.addFile #14861
Conversation
jpiper
changed the title
recursive kwarg to Java Python SparkContext.addFile
|
ok to test |
|
Test build #64591 has finished for PR 14861 at commit
|
jpiper
force-pushed
the
jpiper/pyspark_addfiles
branch
from
cabcca3 to
03d29b8
Compare
|
I fixed the Python style issue and amended the last commit - tests should be good now |
|
Test build #64606 has finished for PR 14861 at commit
|
|
Is anyone available to review this small change? What's the proper process here, @MLnick? It seems @JoshRosen committed the original |
| @@ -0,0 +1 @@ | |||
| Hello World! | |||
There was a problem hiding this comment.
Sorry didn't notice this is for test. But why not use the existing folder test_folder directly ?
There was a problem hiding this comment.
I wanted to ensure that the recursiveness was working and it seemed a bit heavy handed to distribute the entire /test_support/sql/ folder using addFile - however I'm happy to just use that if you think it's better practice.
| FTP URI. | ||
| A directory can be given if the recursive option is set to true. | ||
| Currently directories are onlysupported for Hadoop-supported filesystems. |
There was a problem hiding this comment.
Minor nit: typo (onlysupported needs a space)
|
@jpiper are you still working on this PR? If so you can merge in the latest version of master so we can continue to review? (If not interested that's ok just let us know :)). |
|
@holdenk sorry I've been on vacation! I'll fix the typo and merge in the latest master for you later today or tomorrow. Cheers! |
jpiper
force-pushed
the
jpiper/pyspark_addfiles
branch
from
03d29b8 to
190c63b
Compare
|
Looks like this was actually added in #15140, so we can close this :) |
|
Test build #66593 has finished for PR 14861 at commit
|
5 participants
What changes were proposed in this pull request?
Add the ability to add entire directories using the PySpark interface
SparkContext.addFile(dir, recursive=True)How was this patch tested?
I've added a test file in a nested folders in
python/test_support. I useaddFileto distribute this folder, and then read the file back using the directory structure.