[SPARK-10392] [SQL] Pyspark - Wrong DateType support on JDBC connection

[0x0FFF]

This PR addresses issue SPARK-10392
The problem is that for "start of epoch" date (01 Jan 1970) PySpark class DateType returns 0 instead of the datetime.date due to implementation of its return statement

Issue reproduction on master:

>>> from pyspark.sql.types import *
>>> a = DateType()
>>> a.fromInternal(0)
0
>>> a.fromInternal(1)
datetime.date(1970, 1, 2)

[JoshRosen]

Jenkins, this is ok to test.

[SparkQA]

Test build #41875 has finished for PR 8556 at commit 7985de7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

@davies

[davies]

I think this change is not needed, d will always be true if it's not None

[0x0FFF]

It will be false for None, empty string, empty list, 0. This way it will be returned instead of the date:

>>> [] and 'foo'
[]
>>> 0 and 'foo'
0
>>> 1 and 'foo'
'foo'

[davies]

d could not be list/int/string, it can only be Date or None.

[0x0FFF]

Sorry, I see. Should I revert this change? It is mostly for both methods to follow the same logic, as return x and y is not a very readable code in my opinion. Also the class TimestampType is implemented in a similar way - both methods are starting with input parameter check for None

[davies]

Either is good to me.

[davies]

@0x0FFF Thanks for working on this, could you add a regression test for it?

[0x0FFF]

Added regression test

[SparkQA]

Test build #41886 has finished for PR 8556 at commit f41b125.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public class JavaTrainValidationSplitExample
  • class DCT(JavaTransformer, HasInputCol, HasOutputCol):
  • class SQLTransformer(JavaTransformer):
  • case class LimitNode(limit: Int, child: LocalNode) extends UnaryLocalNode
  • case class UnionNode(children: Seq[LocalNode]) extends LocalNode

[davies]

This test case don't need sqlCtx, it's better to be inside DataTypeTests.

[0x0FFF]

Moved regression test to DataTypeTests class

[davies]

LGTM, will merge once it pass tests.

[SparkQA]

Test build #41889 has finished for PR 8556 at commit a7fd681.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[0x0FFF]

Failed mllib test for python2.6, I didn't change anything that might have affected it. Same test passes locally on my machine

[0x0FFF]

Jenkins, retest this please.

[davies]

@0x0FFF The JVM died during tests.

[SparkQA]

Test build #1708 has finished for PR 8556 at commit a7fd681.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class DCT(JavaTransformer, HasInputCol, HasOutputCol):
  • class SQLTransformer(JavaTransformer):

[davies]

merged into master and 1.5 branch, thanks!

[SparkQA]

Test build #1709 has finished for PR 8556 at commit a7fd681.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class DCT(JavaTransformer, HasInputCol, HasOutputCol):
  • class SQLTransformer(JavaTransformer):