Skip to content

Conversation

@theckang
Copy link

@theckang theckang commented Nov 8, 2015

  1. kafkaStreams is a list. The list should be unpacked when passing it into the streaming context union method, which accepts a variable number of streams.
  2. print() should be pprint() for pyspark.

This contribution is my original work, and I license the work to the project under the project's open source license.

@srowen
Copy link
Member

srowen commented Nov 8, 2015

LGTM

@SparkQA
Copy link

SparkQA commented Nov 8, 2015

Test build #2010 has finished for PR 9545 at commit 8020f13.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@C-kang actually I don't know Python syntax well enough to be sure about this -- this is varargs syntax?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@srowen right. The method is: def union(self, *dstreams), so it expects a variable number of dstreams passed in as arguments. If you pass in a list of dstreams without unpacking it, then vargs= [kafkaStreams, ] as opposed to vargs = [stream1, stream2, stream3, ...]. In the method then, len(vargs) == 1, so it just returns [kafkaStreams] as the unifiedStream without actually performing the union. And then unifiedStream.pprint() will fail because unifiedStream is not a dstream but a list.

asfgit pushed a commit that referenced this pull request Nov 9, 2015
1) kafkaStreams is a list.  The list should be unpacked when passing it into the streaming context union method, which accepts a variable number of streams.
2) print() should be pprint() for pyspark.

This contribution is my original work, and I license the work to the project under the project's open source license.

Author: chriskang90 <[email protected]>

Closes #9545 from c-kang/streaming_python_typo.

(cherry picked from commit 874cd66)
Signed-off-by: Sean Owen <[email protected]>
@srowen
Copy link
Member

srowen commented Nov 9, 2015

Merged to master/1.6/1.5

asfgit pushed a commit that referenced this pull request Nov 9, 2015
1) kafkaStreams is a list.  The list should be unpacked when passing it into the streaming context union method, which accepts a variable number of streams.
2) print() should be pprint() for pyspark.

This contribution is my original work, and I license the work to the project under the project's open source license.

Author: chriskang90 <[email protected]>

Closes #9545 from c-kang/streaming_python_typo.

(cherry picked from commit 874cd66)
Signed-off-by: Sean Owen <[email protected]>
@asfgit asfgit closed this in 874cd66 Nov 9, 2015
@theckang theckang deleted the streaming_python_typo branch April 27, 2016 19:03
@theckang theckang restored the streaming_python_typo branch April 27, 2016 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants