[SPARK-15143][SPARK-15144][SQL] Add CSV tests with HadoopFsRelationTest and support for nullValue for other types #12921

HyukjinKwon · 2016-05-05T05:55:29Z

What changes were proposed in this pull request?

Currently, nullValue option does not work for some types, BooleanType, TimestampType, DateType and StringType. So, currently there is no way to read null for those types. This PR adds the support just like the other types.

Also, CSV data source is not being tested with HadoopFsRelationTest as a HadoopFsRelation. HadoopFsRelationTest includes 50 more tests (eg. partitioned table tests).

This PR adds two variables, extraReadOptions and extraWriteOptions in HadoopFsRelationTest so that the child class gives some options for reading and writing. In order to get the tests in HadoopFsRelationTest passed, CSV data source needs to give options header and inferSchema as true for reading and header as true for writing.

How was this patch tested?

Unittests in CSVHadoopFsRelationTest, CSVTypeCastSuite and edited tests in CSVSuite

… other types

HyukjinKwon · 2016-05-05T06:03:04Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala

    verifyCars(cars, withHeader = true, checkValues = false)
    val results = cars.collect()
-    assert(results(0).toSeq === Array(2012, "Tesla", "S", "null", "null"))
+    assert(results(0).toSeq === Array(2012, "Tesla", "S", null, null))


This is being tested against the data as below:

year,make,model,comment,blank "2012","Tesla","S",null, 1997,Ford,E350,"Go get one now they are going fast", null,Chevy,Volt

Since the header is year,make,model,comment,blank, this should produce the values 2012,Tesla,S,null,null because nullValue is set to "null".

HyukjinKwon · 2016-05-05T06:10:45Z

cc @rxin @falaki

SparkQA · 2016-05-05T06:46:45Z

Test build #57845 has finished for PR 12921 at commit ef71599.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-05T06:52:34Z

Test build #57846 has finished for PR 12921 at commit f80df8c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-05T06:59:40Z

Test build #57847 has finished for PR 12921 at commit 2ea702c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-05-05T08:22:09Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala

-        DateTimeUtils.millisToDays(DateTimeUtils.stringToTime(datum).getTime)
-      case _: StringType => UTF8String.fromString(datum)
-      case _ => throw new RuntimeException(s"Unsupported type: ${castType.typeName}")
+    if (datum == null || (datum == options.nullValue && nullable)) {


Simply the logic below was added just like inferField():

if (datum == null || (datum == options.nullValue && nullable)) { null } else { ...

SparkQA · 2016-05-05T08:57:02Z

Test build #57856 has finished for PR 12921 at commit 33288c8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-05-05T09:12:54Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVTypeCastSuite.scala

        UTF8String.fromString(""))
    assert(
-      CSVTypeCast.castTo("", StringType, nullable = false, CSVOptions()) ==
+      CSVTypeCast.castTo("", StringType, nullable = false, CSVOptions("nullValue", null)) ==


@falaki I just noticed and thought this test implies nullValue does not apply for StringType. Is this intendedly being exclusive? I thought nullValue should be applied for all the types equivalently.

Otherwise, nulls for StringType will be lost in the roundtrip of reading and writing.

SparkQA · 2016-05-05T10:42:43Z

Test build #57871 has finished for PR 12921 at commit f31f69e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-07T03:36:10Z

Test build #58045 has finished for PR 12921 at commit 1233bd7.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class NoSuchPermanentFunctionException(db: String, func: String)
- class NoSuchFunctionException(db: String, func: String)
- case class GetExternalRowField(

SparkQA · 2016-05-11T04:34:48Z

Test build #58319 has finished for PR 12921 at commit 75f1cb8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-05-13T03:40:28Z

Please excuse my ping @rxin @falaki

HyukjinKwon · 2016-05-22T11:28:15Z

Hi @cloud-fan, Could you please take a look?

HyukjinKwon · 2016-07-09T14:57:01Z

Closing this since another PR has (I think) a better change. I will maybe submit another PR for adding some tests in the future.

Add CSV tests with HadoopFsRelationTest and support for nullValue for…

ef71599

… other types

HyukjinKwon reviewed May 5, 2016
View reviewed changes

HyukjinKwon added 2 commits May 5, 2016 15:06

Add some more unittests

f80df8c

Leave the test assert as it is

2ea702c

Ignore flaky string tests for now and optimize the casting logic

33288c8

HyukjinKwon reviewed May 5, 2016
View reviewed changes

HyukjinKwon mentioned this pull request May 5, 2016

[SPARK-15125][SQL] Changing CSV data source mapping of empty quoted strings in the data to empty strings instead of null #12904

Closed

Fix existing tests for null

f31f69e

HyukjinKwon reviewed May 5, 2016
View reviewed changes

Resolve conflicts

1233bd7

This was referenced May 11, 2016

[SPARK-15250][SQL] Remove deprecated json API in DataFrameReader #13040

Closed

[SPARK-15264][SPARK-15274][SQL] CSV Reader Error on Blank Column Names #13041

Closed

Resolve conflicts

75f1cb8

HyukjinKwon mentioned this pull request Jul 9, 2016

[SPARK-16462][SPARK-16460][SPARK-15144][SQL] Make CSV cast null values properly #14118

Closed

HyukjinKwon closed this Jul 9, 2016

HyukjinKwon deleted the SPARK-15143-15144 branch January 2, 2018 03:40

[SPARK-15143][SPARK-15144][SQL] Add CSV tests with HadoopFsRelationTest and support for nullValue for other types #12921

[SPARK-15143][SPARK-15144][SQL] Add CSV tests with HadoopFsRelationTest and support for nullValue for other types #12921

Uh oh!

Conversation

HyukjinKwon commented May 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

HyukjinKwon May 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented May 5, 2016

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

HyukjinKwon May 5, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

HyukjinKwon May 5, 2016

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon May 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

SparkQA commented May 7, 2016

Uh oh!

SparkQA commented May 11, 2016

Uh oh!

HyukjinKwon commented May 13, 2016

Uh oh!

HyukjinKwon commented May 22, 2016

Uh oh!

HyukjinKwon commented Jul 9, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HyukjinKwon commented May 5, 2016 •

edited

Loading

HyukjinKwon May 5, 2016 •

edited

Loading

HyukjinKwon May 13, 2016 •

edited

Loading