Skip to content

Commit 3487b02

Browse files
BryanCutlerrxin
authored andcommitted
[SPARK-17805][PYSPARK] Fix in sqlContext.read.text when pass in list of paths
## What changes were proposed in this pull request? If given a list of paths, `pyspark.sql.readwriter.text` will attempt to use an undefined variable `paths`. This change checks if the param `paths` is a basestring and then converts it to a list, so that the same variable `paths` can be used for both cases ## How was this patch tested? Added unit test for reading list of files Author: Bryan Cutler <[email protected]> Closes #15379 from BryanCutler/sql-readtext-paths-SPARK-17805. (cherry picked from commit bcaa799) Signed-off-by: Reynold Xin <[email protected]>
1 parent 380b099 commit 3487b02

File tree

2 files changed

+8
-2
lines changed

2 files changed

+8
-2
lines changed

python/pyspark/sql/readwriter.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -287,8 +287,8 @@ def text(self, paths):
287287
[Row(value=u'hello'), Row(value=u'this')]
288288
"""
289289
if isinstance(paths, basestring):
290-
path = [paths]
291-
return self._df(self._jreader.text(self._spark._sc._jvm.PythonUtils.toSeq(path)))
290+
paths = [paths]
291+
return self._df(self._jreader.text(self._spark._sc._jvm.PythonUtils.toSeq(paths)))
292292

293293
@since(2.0)
294294
def csv(self, path, schema=None, sep=None, encoding=None, quote=None, escape=None,

python/pyspark/sql/tests.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1697,6 +1697,12 @@ def test_cache(self):
16971697
"does_not_exist",
16981698
lambda: spark.catalog.uncacheTable("does_not_exist"))
16991699

1700+
def test_read_text_file_list(self):
1701+
df = self.spark.read.text(['python/test_support/sql/text-test.txt',
1702+
'python/test_support/sql/text-test.txt'])
1703+
count = df.count()
1704+
self.assertEquals(count, 4)
1705+
17001706

17011707
class HiveSparkSubmitTests(SparkSubmitTests):
17021708

0 commit comments

Comments
 (0)