[SPARK-6055] [PySpark] fix incorrect DataType.__eq__ (for 1.2)

[davies]

The eq of DataType is not correct, class cache is not use correctly (created class can not be find by dataType), then it will create lots of classes (saved in _cached_cls), never released.

Also, all same DataType have same hash code, there will be many object in a dict with the same hash code, end with hash attach, it's very slow to access this dict (depends on the implementation of CPython).

This PR also improve the performance of inferSchema (avoid the unnecessary converter of object).

[SparkQA]

Test build #28054 has started for PR 4809 at commit 6c2909a.

• This patch merges cleanly.

[SparkQA]

Test build #28054 has finished for PR 4809 at commit 6c2909a.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28054/
Test FAILed.

[SparkQA]

Test build #28074 has started for PR 4809 at commit b576107.

• This patch merges cleanly.

[SparkQA]

Test build #28074 has finished for PR 4809 at commit b576107.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28074/
Test PASSed.

[SparkQA]

Test build #28086 has started for PR 4809 at commit 9b4dadc.

• This patch merges cleanly.

[SparkQA]

Test build #28093 has started for PR 4809 at commit 65c222f.

• This patch merges cleanly.

[JoshRosen]

I noticed that the PR opened against master added typechecking asserts here. Should we also add them in this branch, or is there a reason why we should omit them here?

[davies]

Because we can not cherry-pick the patch from master, I need to re-do all the things on 1.2/1.1, so I'd like to keep the changes minimized.

[JoshRosen]

Fair enough; just wanted to check.

[SparkQA]

Test build #28086 has finished for PR 4809 at commit 9b4dadc.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28086/
Test FAILed.

[JoshRosen]

LGTM, since the changes here are a subset of the changes in the PR opened against master.

[SparkQA]

Test build #28093 has finished for PR 4809 at commit 65c222f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28093/
Test PASSed.

[JoshRosen]

I've merged this into branch-1.2 (1.2.2). Thanks!

[pwendell]

@davies can you close this? (auto close doesn't work for the backport commits).