[SPARK-22126][ML][WIP] Fix model-specific optimization support for ML tuning

[WeichenXu123]

What changes were proposed in this pull request?

Push down fitting parallelization code from CrossValidator/TrainValidationSplit into Estimators.
See discussions in SPARK-19357.
Design doc here https://docs.google.com/document/d/1xw5M4sp1e0eQie75yIt-r6-GTuD5vpFf_I6v-AFBM3M/edit?usp=sharing

scala api:

def fit(dataset: Dataset[_], paramMaps: Array[ParamMap],
    unpersistDatasetAfterFitting: Boolean, executionContext: ExecutionContext,
    modelCallback: (Model[_], Int) => Unit): Unit

java api:

def fit(dataset: Dataset[_], paramMaps: Array[ParamMap],
    unpersistDatasetAfterFitting: Boolean, executionContext: ExecutionContext,
    modelCallback: VoidFunction2[Model[_], Int]): Unit

Note: Either in scala or in java, developer can use helper method HasParrallelism.getExecutionContext to get the ExecutionContext object which this API requires.

Discussion: Whether we need to provide both scala & java api, or only provide java api ? In this PR, I provide both scala & java api, so in scala side, developer can override scala api (it will be easier to use), and in java side, developer can override java api.

How was this patch tested?

N/A

[WeichenXu123]

I will add doc for this interface later. But it can be reviewed first, currently this interface looks a little ugly.

[SparkQA]

Test build #82189 has finished for PR 19350 at commit d5625a6.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[WeichenXu123]

retest this please.

[SparkQA]

Test build #82191 has finished for PR 19350 at commit d5625a6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #83889 has finished for PR 19350 at commit 5614390.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #84023 has finished for PR 19350 at commit 1b4a384.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

How will this work in a pipeline?

If the Estimator in CV is a Pipeline, then here it will call fit(dataset, paramMap) on the Pipeline which will in turn fit on each stage with that paramMap. This is what the current parallel CV is doing.

But if we have a stage with model-specific optimization (let's say for arguments sake a LinearRegression that can internally optimize maxIter) then its fit will be called with only a single paramMap arg.

So that pushing the parallel fit into Estimator nullifies any benefit from model-specific optimizations?

[WeichenXu123]

@MLnick Oh, the design is still under discussion on JIRA and will be changed I think. I should mark this WIP. thanks!

[WeichenXu123]

@MLnick I dicussed with @jkbradley @MrBago offline and here is the newest proposal
https://docs.google.com/document/d/1xw5M4sp1e0eQie75yIt-r6-GTuD5vpFf_I6v-AFBM3M/edit?usp=sharing
Thanks!

[WeichenXu123]

Design changed. @MrBago will create new PR for this later. New design is here https://docs.google.com/document/d/1xw5M4sp1e0eQie75yIt-r6-GTuD5vpFf_I6v-AFBM3M/edit?usp=sharing