[SPARK-2808][Streaming][Kafka] update kafka to 0.8.2 #4537

koeninger · 2015-02-11T20:34:06Z

i don't think this should be merged until after 1.3.0 is final

…8.2-upgrade

srowen · 2015-02-11T20:41:17Z

ok to test

SparkQA · 2015-02-11T21:52:46Z

Test build #27301 has finished for PR 4537 at commit 6953429.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zzcclp · 2015-02-13T02:13:15Z

Will this RP be merged into 1.3.0?

koeninger · 2015-02-13T03:28:51Z

I believe that it will not be merged into 1.3
On Feb 12, 2015 8:13 PM, "zzcclp" [email protected] wrote:

Will this RP be merged into 1.3.0?

—
Reply to this email directly or view it on GitHub
#4537 (comment).

lausobo · 2015-02-20T04:10:01Z

@koeninger what's the reason for not putting this into 1.3.0? Kafka 0.8.2 adds support for scala 2.11. That plus non-experimental scala 2.11 support in spark is an interesting combo for scala 2.11 projects.

Thanks in any case.

koeninger · 2015-02-20T04:18:48Z

Spark 1.3.0 was already supposed to be frozen a while back as far as I know.

My personal gut feeling is that it'd be better to wait for kafka 0.8.2.1
bugfixes and more of a chance to test.

On Thu, Feb 19, 2015 at 10:10 PM, Luis Solano [email protected]
wrote:

@koeninger https://github.com/koeninger what's the reason for not
putting this into 1.3.0? Kafka 0.8.2 adds support for scala 2.11. That plus non-experimental
scala 2.11 support in spark
https://issues.apache.org/jira/browse/SPARK-5850 is an interesting
combo for scala 2.11 projects.

Thanks in any case.

—
Reply to this email directly or view it on GitHub
#4537 (comment).

zzcclp · 2015-03-18T08:28:39Z

@koeninger , kafka 0.8.2.1 was released, can it be upgrade this version?

SparkQA · 2015-03-18T15:13:39Z

Test build #28805 has finished for PR 4537 at commit 407382e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

koeninger · 2015-03-18T15:28:17Z

Those tests have passed locally for me 3 times in a row... if I get time later I'll try to dig in

zeitos · 2015-03-19T15:08:10Z

+1 I just tested it with 2.10 and 2.11 and it also passed looks like a flaky test :s

srowen · 2015-03-19T15:12:49Z

Jenkins, retest this please

SparkQA · 2015-03-19T16:11:32Z

Test build #28872 has finished for PR 4537 at commit 407382e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zzcclp · 2015-04-01T03:19:04Z

@koeninger , no progress yet?

koeninger · 2015-04-03T14:18:21Z

@zzcclp if you want to help try and figure out how reproduce this test failure outside of Jenkins, go for it.

zzcclp · 2015-04-05T00:40:51Z

@koeninger , I can't visit this url , it's 404. ??

srowen · 2015-04-05T06:41:54Z

Jenkins, retest this please

zzcclp · 2015-04-09T06:57:33Z

@koeninger , I merge this RP into master and build, it is OK. then run tests as follows:

mvn -Pscala-2.10 -Phadoop-2.3 -Pyarn -Dyarn.version=2.3.0-cdh5.1.2 -Dhadoop.version=2.3.0-cdh5.1.2 -pl core,streaming,external/kafka -DwildcardSuites=org.apache.spark.streaming.kafka.KafkaClusterSuite test

mvn -Pscala-2.10 -Phadoop-2.3 -Pyarn -Dyarn.version=2.3.0-cdh5.1.2 -Dhadoop.version=2.3.0-cdh5.1.2 -pl core,streaming,external/kafka -DwildcardSuites=org.apache.spark.streaming.kafka.KafkaStreamSuite test

all are successful.

I am new to Spark and only do little help.

koeninger · 2015-04-15T16:19:50Z

As far as I can tell from jenkins output, it failed during MiMa checks.

But if I run

sbt '; project streaming-kafka; mima-report-binary-issues'

I get

[info] spark-streaming-kafka: found 0 potential binary incompatibilities (filtered 1)

srowen · 2015-04-15T16:24:08Z

Jenkins claims ...

[error]  * method setConsumerOffsetMetadata(java.lang.String,scala.collection.immutable.Map)scala.util.Either in class org.apache.spark.streaming.kafka.KafkaCluster does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.streaming.kafka.KafkaCluster.setConsumerOffsetMetadata")
[error]  * method setConsumerOffsets(java.lang.String,scala.collection.immutable.Map)scala.util.Either in class org.apache.spark.streaming.kafka.KafkaCluster does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.streaming.kafka.KafkaCluster.setConsumerOffsets")
[error]  * method getConsumerOffsetMetadata(java.lang.String,scala.collection.immutable.Set)scala.util.Either in class org.apache.spark.streaming.kafka.KafkaCluster does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.streaming.kafka.KafkaCluster.getConsumerOffsetMetadata")
[error]  * method getConsumerOffsets(java.lang.String,scala.collection.immutable.Set)scala.util.Either in class org.apache.spark.streaming.kafka.KafkaCluster does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.streaming.kafka.KafkaCluster.getConsumerOffsets")

koeninger · 2015-04-15T16:33:23Z

That maybe makes a certain amount of sense... I'll try replacing the default arguments with multiple overloaded methods, see if that passes mima.

…to overloaded method, for binary compat

zzcclp · 2015-04-22T02:04:35Z

@koeninger , I test streaming + kakfa 0.8.2.1 with this RP, it works well, 😄

koeninger · 2015-04-22T13:38:56Z

Glad to hear it works for you. Fixing the default arguments for mima was straightforward, but there's a lot that has changed in the test code. Work's been busy, haven't had time to fix the merge conflicts yet

SparkQA · 2015-04-30T22:07:42Z

Test build #31448 has finished for PR 4537 at commit 1d896e2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

tdas · 2015-05-01T03:54:43Z

Hey @koeninger , here is a trick I am trying out to to do targeted debugging in Jenkins. I disabled all the other tests and running only streaming tests. The whole thing runs in less than 20 minutes.

#5824

Try this on your PR to see whether you can pin point the issue?

…ython 3 syntax

SparkQA · 2015-05-01T15:38:25Z

Test build #31555 has finished for PR 4537 at commit 30d991d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-01T15:56:37Z

Test build #31558 has finished for PR 4537 at commit d4267e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-05-01T18:41:15Z

python/pyspark/streaming/tests.py

If the waitUntilLeaderOffset has been added to Scala, shouldnt they be added to Python as well? Maybe that then this arbitrary sleep wont be needed.

…all, call it from python tests as well

SparkQA · 2015-05-01T19:21:12Z

Test build #31587 has finished for PR 4537 at commit 1770abc.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

koeninger · 2015-05-01T19:25:01Z

scalastyle passes locally for me...

… jenkins again

koeninger · 2015-05-01T20:28:50Z

At any rate, I tested it out locally against 0.8.1.1 server install, with the IdempotentExample job from my blogpost. Seemed to work ok.

tdas · 2015-05-01T20:55:20Z

external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaTestUtils.scala

Could you put this big piece of code in a separate def (can be inside waitUntilMetadata..) outside the eventually, and then the assert just calls that function? Much easier to read.

And also explain in a comment what this code does? For example, I am not sure what is isr in leaderAndIsr. And what are all these conditions &&-ed together.

SparkQA · 2015-05-01T22:06:38Z

Test build #31602 has finished for PR 4537 at commit e6dfaf6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-05-01T22:12:27Z

Running this couple of times to test for flakiness.

SparkQA · 2015-05-01T23:18:47Z

Test build #31608 has finished for PR 4537 at commit 803aa2c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-01T23:53:12Z

Test build #756 has finished for PR 4537 at commit 803aa2c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-01T23:55:54Z

Test build #757 has finished for PR 4537 at commit 803aa2c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-05-02T00:52:47Z

Great! Merging this! Thanks @koeninger @helena !

i don't think this should be merged until after 1.3.0 is final Author: cody koeninger <[email protected]> Author: Helena Edelson <[email protected]> Closes apache#4537 from koeninger/wip-2808-kafka-0.8.2-upgrade and squashes the following commits: 803aa2c [cody koeninger] [SPARK-2808][Streaming][Kafka] code cleanup per TD e6dfaf6 [cody koeninger] [SPARK-2808][Streaming][Kafka] pointless whitespace change to trigger jenkins again 1770abc [cody koeninger] [SPARK-2808][Streaming][Kafka] make waitUntilLeaderOffset easier to call, call it from python tests as well d4267e9 [cody koeninger] [SPARK-2808][Streaming][Kafka] fix stderr redirect in python test script 30d991d [cody koeninger] [SPARK-2808][Streaming][Kafka] remove stderr prints since it breaks python 3 syntax 1d896e2 [cody koeninger] [SPARK-2808][Streaming][Kafka] add even even more logging to python test 4c4557f [cody koeninger] [SPARK-2808][Streaming][Kafka] add even more logging to python test 115aeee [cody koeninger] Merge branch 'master' into wip-2808-kafka-0.8.2-upgrade 2712649 [cody koeninger] [SPARK-2808][Streaming][Kafka] add more logging to python test, see why its timing out in jenkins 2b92d3f [cody koeninger] [SPARK-2808][Streaming][Kafka] wait for leader offsets in the java test as well 3824ce3 [cody koeninger] [SPARK-2808][Streaming][Kafka] naming / comments per tdas 61b3464 [cody koeninger] [SPARK-2808][Streaming][Kafka] delay for second send in boundary condition test af6f3ec [cody koeninger] [SPARK-2808][Streaming][Kafka] delay test until latest leader offset matches expected value 9edab4c [cody koeninger] [SPARK-2808][Streaming][Kafka] more shots in the dark on jenkins failing test c70ee43 [cody koeninger] [SPARK-2808][Streaming][Kafka] add more asserts to test, try to figure out why it fails on jenkins but not locally 1d10751 [cody koeninger] Merge branch 'master' into wip-2808-kafka-0.8.2-upgrade ed02d2c [cody koeninger] [SPARK-2808][Streaming][Kafka] move default argument for api version to overloaded method, for binary compat 407382e [cody koeninger] [SPARK-2808][Streaming][Kafka] update kafka to 0.8.2.1 77de6c2 [cody koeninger] Merge branch 'master' into wip-2808-kafka-0.8.2-upgrade 6953429 [cody koeninger] [SPARK-2808][Streaming][Kafka] update kafka to 0.8.2 2e67c66 [Helena Edelson] #SPARK-2808 Update to Kafka 0.8.2.0 GA from beta. d9dc2bc [Helena Edelson] Merge remote-tracking branch 'upstream/master' into wip-2808-kafka-0.8.2-upgrade e768164 [Helena Edelson] apache#2808 update kafka to version 0.8.2

Helena Edelson and others added 4 commits December 7, 2014 10:50

apache#2808 update kafka to version 0.8.2

e768164

Merge remote-tracking branch 'upstream/master' into wip-2808-kafka-0.…

d9dc2bc

…8.2-upgrade

#SPARK-2808 Update to Kafka 0.8.2.0 GA from beta.

2e67c66

[SPARK-2808][Streaming][Kafka] update kafka to 0.8.2

6953429

koeninger mentioned this pull request Feb 11, 2015

#SPARK-2808 update kafka to version 0.8.2 #3631

Closed

koeninger added 2 commits March 18, 2015 07:05

Merge branch 'master' into wip-2808-kafka-0.8.2-upgrade

77de6c2

[SPARK-2808][Streaming][Kafka] update kafka to 0.8.2.1

407382e

[SPARK-2808][Streaming][Kafka] move default argument for api version …

ed02d2c

…to overloaded method, for binary compat

tdas mentioned this pull request May 1, 2015

[DO NO MERGE] Debugging Kafka 0.8.2 test hang in Jenkins #5828

Closed

koeninger added 2 commits May 1, 2015 08:55

[SPARK-2808][Streaming][Kafka] remove stderr prints since it breaks p…

30d991d

…ython 3 syntax

[SPARK-2808][Streaming][Kafka] fix stderr redirect in python test script

d4267e9

tdas reviewed May 1, 2015
View reviewed changes

[SPARK-2808][Streaming][Kafka] make waitUntilLeaderOffset easier to c…

1770abc

…all, call it from python tests as well

[SPARK-2808][Streaming][Kafka] pointless whitespace change to trigger…

e6dfaf6

… jenkins again

tdas reviewed May 1, 2015
View reviewed changes

[SPARK-2808][Streaming][Kafka] code cleanup per TD

803aa2c

asfgit closed this in 4786484 May 2, 2015

[SPARK-2808][Streaming][Kafka] update kafka to 0.8.2 #4537

[SPARK-2808][Streaming][Kafka] update kafka to 0.8.2 #4537

Uh oh!

Conversation

koeninger commented Feb 11, 2015

Uh oh!

srowen commented Feb 11, 2015

Uh oh!

SparkQA commented Feb 11, 2015

Uh oh!

zzcclp commented Feb 13, 2015

Uh oh!

koeninger commented Feb 13, 2015

Uh oh!

lausobo commented Feb 20, 2015

Uh oh!

koeninger commented Feb 20, 2015

Uh oh!

zzcclp commented Mar 18, 2015

Uh oh!

SparkQA commented Mar 18, 2015

Uh oh!

koeninger commented Mar 18, 2015

Uh oh!

zeitos commented Mar 19, 2015

Uh oh!

srowen commented Mar 19, 2015

Uh oh!

SparkQA commented Mar 19, 2015

Uh oh!

zzcclp commented Apr 1, 2015

Uh oh!

koeninger commented Apr 3, 2015

Uh oh!

zzcclp commented Apr 5, 2015

Uh oh!

srowen commented Apr 5, 2015

Uh oh!

zzcclp commented Apr 9, 2015

Uh oh!

koeninger commented Apr 15, 2015

Uh oh!

srowen commented Apr 15, 2015

Uh oh!

koeninger commented Apr 15, 2015

Uh oh!

zzcclp commented Apr 22, 2015

Uh oh!

koeninger commented Apr 22, 2015

Uh oh!

SparkQA commented Apr 30, 2015

Uh oh!

tdas commented May 1, 2015

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

tdas May 1, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

koeninger commented May 1, 2015

Uh oh!

koeninger commented May 1, 2015

Uh oh!

tdas May 1, 2015

Choose a reason for hiding this comment

Uh oh!

tdas May 1, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

tdas commented May 1, 2015

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

SparkQA commented May 1, 2015