[SPARK-6211][Streaming] Add Python Kafka API unit test

[jerryshao]

Refactor the Kafka unit test and add Python API support. CC @tdas @davies please help to review, thanks a lot.

[SparkQA]

Test build #28427 has started for PR 4961 at commit d4f2e5b.

• This patch merges cleanly.

[SparkQA]

Test build #28427 has finished for PR 4961 at commit d4f2e5b.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class EmbeddedZookeeper(val zkConnect: String)

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28427/
Test FAILed.

[SparkQA]

Test build #28429 has started for PR 4961 at commit 4898694.

• This patch merges cleanly.

[SparkQA]

Test build #28429 has finished for PR 4961 at commit 4898694.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class EmbeddedZookeeper(val zkConnect: String)

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28429/
Test FAILed.

[jerryshao]

Jenkins, retest this please.

[SparkQA]

Test build #28431 has started for PR 4961 at commit 4898694.

• This patch merges cleanly.

[SparkQA]

Test build #28431 has finished for PR 4961 at commit 4898694.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class EmbeddedZookeeper(val zkConnect: String)

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28431/
Test FAILed.

[SparkQA]

Test build #28458 has started for PR 4961 at commit f66a067.

• This patch merges cleanly.

[SparkQA]

Test build #28458 has finished for PR 4961 at commit f66a067.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class EmbeddedZookeeper(val zkConnect: String)

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28458/
Test FAILed.

[jerryshao]

Looks like maven command in run-tests script do not generate kafka-assembly fat jar, only a single module empty jar. I took a look at the pom file under kafka-assembly, actually has shade plugin and enabled under package, I'm not sure why mvn package from parent module cannot generate kafka-assembly fat jar, only one assembly jar in module assembly is generated, is it a bug or just the right behavior?

Currently what I did is to use mvn -am -pl "external/kafka-assembly" package to generate a fat jar, do you have any suggestions, @tdas. Thanks a lot.

[tdas]

This seems like a bug that was fixed couple of weeks ago. Are you sure you have merged with master recently. In fact the PR has merge conflicts.

[tdas]

Can you add one line to say why this is in /main and not /test.

[tdas]

Other than the assembly issue, its looks quite good. Only a few minor comments.

[jerryshao]

Thanks a lot @tdas for your review, actually I rebased the code in my local environment and tried to test again, seems a recent merged PR #3916 changed the behavior of pyspark, so previously --jars argument that can be worked in unit test are failed to work now, I'm trying to figure out a way to enable this in unit test.

[jerryshao]

Seems recent merged PR #5261 changed the script and make my test fail, I will fix it.

[SparkQA]

Test build #29999 has started for PR 4961 at commit ee4b919.

[SparkQA]

Test build #29999 has finished for PR 4961 at commit ee4b919.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class FlattenedValuesSerializer(BatchedSerializer):
  • class ExternalList(object):
  • class ExternalListOfList(ExternalList):
  • class GroupByKey(object):
  • class ExternalGroupBy(ExternalMerger):
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29999/
Test PASSed.

[jerryshao]

Jenkins, retest this please.

[SparkQA]

Test build #30006 has started for PR 4961 at commit ee4b919.

[tdas]

Jenkins, retest this please.
On Apr 9, 2015 6:52 PM, "Apache Spark QA" [email protected] wrote:

Test build #29991 has finished
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29991/consoleFull
for PR 4961 at commit 1cda32a
1cda32a
.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

—
Reply to this email directly or view it on GitHub
#4961 (comment).

[tdas]

LGTM. Will merge if it passes.

[SparkQA]

Test build #30006 has finished for PR 4961 at commit ee4b919.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30006/
Test PASSed.

[tdas]

Merging this.

[tdas]

Thanks @jerryshao for doing this!