[SPARK-9116] [SQL] [PYSPARK] support Python only UDT in __main__

[davies]

Also we could create a Python UDT without having a Scala one, it's important for Python users.

cc @mengxr @JoshRosen

[SparkQA]

Test build #1094 has finished for PR 7453 at commit 55bb86e.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class PointUDT(UserDefinedType):
  • class PythonUserDefinedType(val sqlType: DataType, pyClass: String) extends UserDefinedType[Any]

[rxin]

are these interfaces designed in UserDefinedType? If yes, can you add override here? it caches a lot of potential problems when we do refactoring down the line.

[SparkQA]

Test build #37588 has finished for PR 7453 at commit 83d65ac.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class PointUDT(UserDefinedType):

[SparkQA]

Test build #37602 has finished for PR 7453 at commit de986d6.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class PointUDT(UserDefinedType):

[JoshRosen]

This conflicts with / overlaps with the changes in #7476, so I guess this patch is blocking until that other one gets reviewed?

[davies]

yes

[SparkQA]

Test build #37651 has finished for PR 7453 at commit 0bcb3ef.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class PointUDT(UserDefinedType):

[JoshRosen]

Why not just use the default __eq__?

[SparkQA]

Test build #37678 has finished for PR 7453 at commit 316a394.

• This patch fails Python style tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[JoshRosen]

Looks like PyLint is complaining about a possibly undefined loop variable v at this line. If this isn't a legitimate error, then we can just add a comment to bypass that warning here.

[davies]

@JoshRosen I think it's annoying to let PyLint report Warning as Error, should we only fail on real errors?

[JoshRosen]

Good point; let's see if we can update the configuration to do that.

[davies]

I tried with putting -E to pylint, it seems even worse, lots of false-negtive errors.

[SparkQA]

Test build #37817 has finished for PR 7453 at commit 655b8a9.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class ShuffledRowRDD(

[SparkQA]

Test build #37859 has finished for PR 7453 at commit 63f52ef.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #38279 has finished for PR 7453 at commit a86e1fc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

@mengxr Could you help to review this?

[JoshRosen]

Ping @mengxr, I think we need to get this in for 1.5.0 since it contains a bugfix for another patch.

[mengxr]

I'll make a pass tomorrow.

[mengxr]

missing doc

[mengxr]

Made one pass. My main comment is to keep the unit test for UDTs that work in both Python and Scala/Java.

[davies]

@mengxr Had kept the unit tests for Python and Scala UDT, please take another round review.

[mengxr]

LGTM

[SparkQA]

Test build #38914 has finished for PR 7453 at commit 4dfd5e1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • The class name of the paired Scala UDT (could be '', if there

[davies]

Merged into master.