SPARK-554. Add aggregateByKey. #705

sryza · 2014-05-09T01:07:55Z

No description provided.

AmplabJenkins · 2014-05-09T01:07:58Z

Merged build triggered.

AmplabJenkins · 2014-05-09T01:08:04Z

Merged build started.

AmplabJenkins · 2014-05-09T01:09:20Z

Merged build finished.

AmplabJenkins · 2014-05-09T01:09:21Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14833/

mateiz · 2014-05-27T06:05:37Z

Hey Sandy, this needs Java and Python APIs to be merged as well. We're going to stop adding Scala-only APIs at first and then fixing the other languages later.

sryza · 2014-05-28T05:41:42Z

Makes sense. Updated patch adds Java and Python APIs.

AmplabJenkins · 2014-05-28T05:42:59Z

Merged build triggered.

AmplabJenkins · 2014-05-28T05:43:08Z

Merged build started.

AmplabJenkins · 2014-05-28T05:44:34Z

Merged build finished.

AmplabJenkins · 2014-05-28T05:44:34Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15244/

mateiz · 2014-05-29T00:14:15Z

python/pyspark/rdd.py

You seem to confuse T and V here (same in the other doc comments)

My bad. Uploaded a patch that fixes this.

AmplabJenkins · 2014-05-29T04:57:58Z

Merged build triggered.

AmplabJenkins · 2014-05-29T04:58:07Z

Merged build started.

AmplabJenkins · 2014-05-29T04:59:34Z

Merged build finished.

AmplabJenkins · 2014-05-29T04:59:35Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15280/

mateiz · 2014-06-03T01:54:51Z

@sryza Looks like you have a style error, see https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15280/console. You can run sbt scalastyle to do these tests locally.

sryza · 2014-06-03T02:27:52Z

My bad - fixed these.

AmplabJenkins · 2014-06-03T02:27:58Z

Build triggered.

AmplabJenkins · 2014-06-03T02:28:08Z

Build started.

AmplabJenkins · 2014-06-03T02:29:05Z

Build finished.

AmplabJenkins · 2014-06-03T02:29:05Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15367/

mateiz · 2014-06-03T20:24:30Z

Jenkins, retest this please

AmplabJenkins · 2014-06-03T20:28:06Z

Build triggered.

AmplabJenkins · 2014-06-03T20:28:13Z

Build started.

AmplabJenkins · 2014-06-03T20:38:17Z

Build finished.

AmplabJenkins · 2014-06-03T20:38:18Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15401/

mateiz · 2014-06-03T20:58:52Z

Jenkins, retest this please

AmplabJenkins · 2014-06-03T21:02:59Z

Build triggered.

AmplabJenkins · 2014-06-03T21:08:54Z

Build started.

AmplabJenkins · 2014-06-03T21:09:55Z

Build finished.

pwendell · 2014-06-11T03:23:33Z

@sryza looks like the python tests are failing

sryza · 2014-06-12T00:46:19Z

Ok, the test should be fixed for real this time.

AmplabJenkins · 2014-06-12T00:47:05Z

Merged build triggered.

AmplabJenkins · 2014-06-12T00:47:15Z

Merged build started.

AmplabJenkins · 2014-06-12T01:28:50Z

Merged build finished.

AmplabJenkins · 2014-06-12T01:28:51Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15696/

sryza · 2014-06-12T02:12:38Z

I'm noticing a MIMA error. I think this must be a false positive? As the patch only adds APIs.

pwendell · 2014-06-12T06:32:50Z

@ScrapCodes - do you have any idea on this one? This doesn't make much sense to me at all! @sryza for now you can add an exclude in project/MimaExcludes this is clearly a false positive.

ScrapCodes · 2014-06-12T07:13:39Z

This happened because in the byte code there was a method as follows
public final java.lang.Object org$apache$spark$rdd$PairRDDFunctions$$createZero$1(byte[], scala.runtime.ObjectRef, scala.runtime.VolatileByteRef);

Which got changed to public final java.lang.Object org$apache$spark$rdd$PairRDDFunctions$$createZero$1(scala.reflect.ClassTag, byte[], scala.runtime.ObjectRef, scala.runtime.VolatileByteRef);

After this patch, not very sure if we could have avoided it. Since this method was an inner method.

AmplabJenkins · 2014-06-12T08:17:06Z

Merged build triggered.

AmplabJenkins · 2014-06-12T08:17:11Z

Merged build started.

ScrapCodes · 2014-06-12T08:21:11Z

I just tried extracting createZero out and making it private, since the method is used in two places now. But making it private did not help, the mima warning is still there.

AmplabJenkins · 2014-06-12T09:00:59Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-12T09:00:59Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15707/

pwendell · 2014-06-12T15:12:35Z

@ScrapCodes. Ah I see - there was a name collision with an inner function that was used elsewhere.

pwendell · 2014-06-12T15:13:48Z

Thanks Sandy - I'm merging this!

Author: Sandy Ryza <[email protected]> Closes apache#705 from sryza/sandy-spark-554 and squashes the following commits: 2302b8f [Sandy Ryza] Add MIMA exclude f52e0ad [Sandy Ryza] Fix Python tests for real 2f3afa3 [Sandy Ryza] Fix Python test 0b735e9 [Sandy Ryza] Fix line lengths ae56746 [Sandy Ryza] Fix doc (replace T with V) c2be415 [Sandy Ryza] Java and Python aggregateByKey 23bf400 [Sandy Ryza] SPARK-554. Add aggregateByKey.

…ache#705) Co-authored-by: Jeremy Liu <[email protected]>

…ath option (apache#705) * [SPARK-51281][SQL] DataFrameWriterV2 should respect the path option Unlike `DataFrameWriter.saveAsTable` where we explicitly get the "path" option and treat it as table location, `DataFrameWriterV2` doesn't do it and treats the "path" option as a normal option which doesn't have any real impact. This PR fixes it, and adds a legacy config to restore the old behavior. bug fix Yes, now `DataFrameWriterV2` can correctly write data to the specified path for file source tables. new test no Closes apache#50040 from cloud-fan/prop. Lead-authored-by: Wenchen Fan <[email protected]> Co-authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit a3671e5) Signed-off-by: Wenchen Fan <[email protected]>

mateiz reviewed May 29, 2014
View reviewed changes

sryza added 6 commits June 11, 2014 16:26

SPARK-554. Add aggregateByKey.

23bf400

Java and Python aggregateByKey

c2be415

Fix doc (replace T with V)

ae56746

Fix line lengths

0b735e9

Fix Python test

2f3afa3

Fix Python tests for real

f52e0ad

Add MIMA exclude

2302b8f

asfgit closed this in ce92a9c Jun 12, 2014

Alexis-D pushed a commit to Alexis-D/spark that referenced this pull request Nov 16, 2020

Get preInstalledPath value in computation in sparkR package paths (ap…

0c76d64

…ache#705) Co-authored-by: Jeremy Liu <[email protected]>

SPARK-554. Add aggregateByKey. #705

SPARK-554. Add aggregateByKey. #705

Uh oh!

Conversation

sryza commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

mateiz commented May 27, 2014

Uh oh!

sryza commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

mateiz May 29, 2014

Choose a reason for hiding this comment

Uh oh!

sryza May 29, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 29, 2014

Uh oh!

AmplabJenkins commented May 29, 2014

Uh oh!

AmplabJenkins commented May 29, 2014

Uh oh!

AmplabJenkins commented May 29, 2014

Uh oh!

mateiz commented Jun 3, 2014

Uh oh!

sryza commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

mateiz commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

mateiz commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

pwendell commented Jun 11, 2014

Uh oh!

sryza commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 12, 2014

Uh oh!