Skip to content

Conversation

@ghost
Copy link

@ghost ghost commented Mar 4, 2019

This working patch is for [SPARK-25517][SQL] which is about auto inferring date type from the csv dataset while loading the csv dataset into a dataframe, as per the current behavior it reads date column as string type which is wrong it should read as date. however auto-inferring schema for timestamp column is perfectly working fine

What changes were proposed in this pull request?

Modified a single file CSVInferSchema.scala and made changes in the corresponding Suite i.e. CSVInferSchemaSuite and run this suite and it's working perfectly now,

How was this patch tested?

Performed unit testing using the suite CSVInferSchemaSuite.scala

@ghost ghost changed the title [SPARK-25517] fixes for auto inferring date columns as date instead of string [SPARK-25517][SQL] fixes for auto inferSchema for date columns as date instead of string Mar 4, 2019
@ghost
Copy link
Author

ghost commented Mar 4, 2019

Hi @dongjoon-hyun @HyukjinKwon

Regret for being inactive for sometime, this is going to be my first contribution to the spark realm, I am honestly totally new to github and self-experimenting to learn how to operate through github. hence created a separate github account to work on this jira [SPARK-25517] and raised this PR.

over the last weekend, I verified that this issue still exists in the latest spark 2.4.0 hence worked on it and fixed the issue. Updated the test suite CSVInferSchemaSuite.scala and tested through it, working fine now as expected,

so I'd like to reopen this jira and mark this as closed,

please review the changes and suggest, in case of any further inputs or queries you are most welcome, : )

Regards,
Manoranjan

@MaxGekk
Copy link
Member

MaxGekk commented Mar 4, 2019

Similar changes were committed #23202 and reverted by @HyukjinKwon already. @manoranjanmix Welcome here and good luck.

@HyukjinKwon
Copy link
Member

Yup, please make sure of addressing the concerns made there.

@ghost
Copy link
Author

ghost commented Mar 4, 2019

Ohh i see that now Max, thanks for pointing, I had caught this issue way back in Sep-18 but was unfortunate to continue, however I will proceed further from here and try to bring more improvements
Thanks,

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@HyukjinKwon
Copy link
Member

Closing this due to author's inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants