[SPARK-1817] RDD.zip() should verify partition sizes for each partition

[kanzhang]

RDD.zip() will throw an exception if it finds partition sizes are not the same.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[kanzhang]

@mateiz This is to address the remaining issue of #760 (after #776). Since now zip() calls zipPartitions(), I removed ZippedRDD.scala. Pls take a look.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15357/

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15360/

[AmplabJenkins]

Build triggered.

[AmplabJenkins]

Build started.

[AmplabJenkins]

Build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15362/

[kanzhang]

@pwendell I got the following test error, not sure what I can do. Can you help?

Could not find Apache license headers in the following files:
 !????? /root/workspace/SparkPullRequestBuilder/.generated-mima-excludes
Build step 'Execute shell' marked build as failure

[marmbrus]

I think you might need to rebase your change as it probably is not merging cleanly.

[mateiz]

Make this a SparkException instead

[mateiz]

Weird, might be a temporary issue in Jenkins.

[AmplabJenkins]

Build triggered.

[kanzhang]

@mateiz updated
@marmbrus thanks for your note. I'll rebase if it still fails.

[AmplabJenkins]

Build started.

[AmplabJenkins]

Build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15369/

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15371/

[kanzhang]

Are we going to include this patch for 1.0.x? If so, I'll need to update MimaExcludes for it.

[mateiz]

Wait, why did you need to add this to MimaExcludes? You're not changing the API.

[kanzhang]

MiMa complains as follows, since I deleted ZippedRDD.scala. However, both classes are marked private[spark], I suppose it is OK to exclude?

[error]  * class org.apache.spark.rdd.ZippedRDD does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.rdd.ZippedRDD")
[error]  * class org.apache.spark.rdd.ZippedPartition does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.rdd.ZippedPartition")

[mateiz]

Ah, got it. @pwendell what do you think? I think we'd just add them to the excludes.

[mateiz]

(This is probably an oversight in the way we set up MIMA).

[rxin]

Yea we should just add the rules. I had to work around the same problem when I removed SerializableHyperLogLog.

[rxin]

BTW due to two recent PR merges, I think this no longer merges cleanly. Do you mind updating the PR to rebase it to master? Thanks.

[pwendell]

Yeah this is one case where we need to manually add excludes. The reason is MIMA doesn't have proper support for package privacy, and our work-around can't handle the case where we delete a previously package private thinng.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[kanzhang]

@rxin rebased just now. Thanks!

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15431/

[rxin]

Thanks. I've merged this into master branch.