[SPARK-22667][ML][WIP] Fix model-specific optimization support for ML tuning: Python API

[WeichenXu123]

What changes were proposed in this pull request?

This is python api for #19350

Python CrossValidator/TrainValidationSplit:
With base Estimator implemented in Scala/Java
→ Convert base Estimator to Scala/Java object, and call the JVM fit()
With base Estimator implemented in Python
→ Python needs the same machinery for multi-model fitting and parallelism as Scala. We can call directly into it. New API added:

class Estimator:
  def parallelFit(self, dataset, paramMaps, threadPool, modelCallback):

Doc link

Note This PR also fix the # TODO: persist average/validation metrics as well in CV/TVS. Because the testsuite need to check consistency of avgMetrics/validationMetrics so this need to be fixed.
If this need backport to old spark version, I can split it to a separate PR.

How was this patch tested?

Existing UT already covers each code paths which need test.

[WeichenXu123]

@MrBago @jkbradley I think this PR need to be reviewed and merged first, before reviewing #19627
Because this PR change some critical code path.

[SparkQA]

Test build #84372 has finished for PR 19857 at commit 980c8ec.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #84373 has finished for PR 19857 at commit c6f2250.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[WeichenXu123]

The design of this issue changed. @MrBago will take this over.