[SPARK-13309][SQL] Fix type inference issue with CSV data

[tanwanirahul]

Fix type inference issue for sparse CSV data - https://issues.apache.org/jira/browse/SPARK-13309

[rxin]

cc @HyukjinKwon want to review this one?

[SparkQA]

Test build #2542 has finished for PR 11194 at commit 5edaa2a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

@rxin Sure.

[HyukjinKwon]

(This is not the part of diff but it might be great to change ((a, b)) to (a, b))

[HyukjinKwon]

Overall, it looks good to me. This was already merged in databricks/spark-csv#261 and the logic looks identical.

[HyukjinKwon]

Actually, I have had a thought that we might have to make a class such as TestCSVData for dataset for testing (similarly with TestJsonData for JSON datasource) or a class like CSVTest (similarly with OrcTest fpr ORC datasource) rather than adding test CSV files for everytime.

I think this might better be done in another PR. If you agree on this, I will create an issue and PR for this after this one is merged.

[tanwanirahul]

Yes, could be done. I personally prefer to keep code and data separate. This way:

• Changing the data does not require us to compile the code.
• Reading through the source code is friendly if it does not contain data in between.

[tanwanirahul]

@HyukjinKwon @rxin Is this waiting on me? Just want to confirm it I am expected to add anything more.

[tanwanirahul]

Could we please merge this?

[rxin]

Sorry for the delay. I'm merging this in master. Thanks!

[tanwanirahul]

@rxin @HyukjinKwon thank you.