[SPARK-1981] Add AWS Kinesis streaming support #1434

cfregly · 2014-07-16T07:35:16Z

No description provided.

AmplabJenkins · 2014-07-16T07:36:35Z

Can one of the admins verify this patch?

Resolved conflict: project/SparkBuild.scala

srosenthal · 2014-07-18T22:29:41Z

project/SparkBuild.scala

Looks like a copy-paste error? Did you mean to use "spark-kinesis-asl" here instead of "spark-ganglia-lgpl"?

good catch, stephen. thanks!

mateiz · 2014-07-26T22:22:23Z

Jenkins, this is ok to test

mateiz · 2014-07-26T22:22:59Z

Chris, is this new code from scratch, or is it based on Parviz's old pull request?

SparkQA · 2014-07-26T22:28:38Z

QA tests have started for PR 1434. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17230/consoleFull

cfregly · 2014-07-27T01:54:05Z

@mateiz -

this is a completely brand-new, from-scratch implementation.

parviz's old code was actually a Scala port of the Java-based Kinesis sample app found here: https://github.com/aws/aws-sdk-java/blob/master/src/samples/AmazonKinesisApplication/SampleRecordProcessor.java

this was fine for a quick/dirty sample of kinesis functionality, but my goal was to make this code more reusable, testable, readable, configurable, production-ready, and well-documented. the old code did not support the new Streaming 1.0 API and extras/ build structure (due to ASL-license). i also updated the AWS Java SDK and Kinesis Client Libraries to their latest versions.

here is parviz's PR for reference/comparison: #223.
i've addressed all of the comments provided in the old PR to speed up the acceptance of this new PR.

thanks and i look forward to getting this merged!

-chris

cfregly · 2014-07-27T01:55:37Z

also, can someone address the questions i have here regarding the ec2 scripts and other peripheral aspects of this PR: https://issues.apache.org/jira/browse/SPARK-1981?focusedCommentId=14072761

thanks!

-chris

cfregly · 2014-07-27T01:57:16Z

this PR worked for @srosenthal , btw.

tdas · 2014-07-29T03:24:06Z

bin/run-kinesis-example

Why is there a need for a complete script to run the kinesis example? Shouldnt the standard bin/run-example be sufficient?
The thing is that we want to have one streamlined way for running all the examples. The more special cases we make, the more we have to explicitly maintain in the future.

@pdeyhim explained me that this script is necessary as the jar for kinesis is not included in the spark jar. That makes sense.
But its best not to put such module-specific scripts in the main spark/bin directory. Its best to put this in extras/kinesis/bin/ .

yup, yup. this is a special circumstance due to the licensing. i didn't want to complicate the examples build by depending on an optional package, so this was the best workaround.

i put the scripts alongside the other scripts to reduce confusion, but i can move it to extras/kinesis/bin for sure. i'll update the docs accordingly.

mateiz · 2014-07-29T07:26:20Z

@pdeyhim can you take a look over this too when you have a chance?

tdas · 2014-07-29T17:43:15Z

Jenkins, this is ok to test.

tdas · 2014-07-29T17:43:22Z

Jenkins, test this please.

SparkQA · 2014-07-29T17:48:58Z

QA tests have started for PR 1434. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17365/consoleFull

SparkQA · 2014-07-29T19:27:39Z

QA results for PR 1434:
- This patch PASSES unit tests.

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17365/consoleFull

tdas · 2014-07-29T20:06:38Z

Hey @cfregly,

My apologies for being so late to review this. Seems like the PR requires merging with the master, probably because of changes to pom.xml. Would you get a chance to merge with the master once again?

tdas · 2014-07-29T20:17:24Z

extras/spark-kinesis-asl/pom.xml

Can you rename is the directory spark-kinesis-asl to just kinesis. Its already in spark code base, so does not quite need another spark in the directory name. :)

Aah, I understand, you must have followed the existing spark-ganglia-gpl directory name. Let me figure out why it was named so.

Okay, I spoke to @pwendell and there is not need to have spark- as part of the name directory name and profile name. So can you make it kinesis-asl at all places.

makes sense. i'll update.

tdas · 2014-08-01T22:43:07Z

examples/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java

Inline comment style should /* ... / .... /* is reserved only for scala docs.

updated the build to only include kinesis-asl inside the examples jar when -Pkinesis-asl is specified

SparkQA · 2014-08-02T02:29:26Z

QA tests have started for PR 1434. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17736/consoleFull

tdas · 2014-08-02T02:30:59Z

extras/kinesis-asl/pom.xml

Nit: Change this, this will be super confusing in future.

oh yikes. yeah, that's just plain wrong.

Correction, this is not a nit. Its already confusing ... please change this!

Haha, didnt see your response. Good 👍

SparkQA · 2014-08-02T03:25:01Z

QA results for PR 1434:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
public final class JavaKinesisWordCountASL {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17736/consoleFull

tdas · 2014-08-02T03:28:16Z

extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala

This does not need to extend Logging.

removed and cleaned up imports

retries around store() method

SparkQA · 2014-08-02T07:14:28Z

QA tests have started for PR 1434. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17760/consoleFull

SparkQA · 2014-08-02T08:14:25Z

QA results for PR 1434:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
public final class JavaKinesisWordCountASL {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17760/consoleFull

tdas · 2014-08-02T20:37:06Z

Thanks you very much @cfregly! I have merged this!!

Author: Chris Fregly <[email protected]> Closes #1434 from cfregly/master and squashes the following commits: 4774581 [Chris Fregly] updated docs, renamed retry to retryRandom to be more clear, removed retries around store() method 0393795 [Chris Fregly] moved Kinesis examples out of examples/ and back into extras/kinesis-asl 691a6be [Chris Fregly] fixed tests and formatting, fixed a bug with JavaKinesisWordCount during union of streams 0e1c67b [Chris Fregly] Merge remote-tracking branch 'upstream/master' 74e5c7c [Chris Fregly] updated per TD's feedback. simplified examples, updated docs e33cbeb [Chris Fregly] Merge remote-tracking branch 'upstream/master' bf614e9 [Chris Fregly] per matei's feedback: moved the kinesis examples into the examples/ dir d17ca6d [Chris Fregly] per TD's feedback: updated docs, simplified the KinesisUtils api 912640c [Chris Fregly] changed the foundKinesis class to be a publically-avail class db3eefd [Chris Fregly] Merge remote-tracking branch 'upstream/master' 21de67f [Chris Fregly] Merge remote-tracking branch 'upstream/master' 6c39561 [Chris Fregly] parameterized the versions of the aws java sdk and kinesis client 338997e [Chris Fregly] improve build docs for kinesis 828f8ae [Chris Fregly] more cleanup e7c8978 [Chris Fregly] Merge remote-tracking branch 'upstream/master' cd68c0d [Chris Fregly] fixed typos and backward compatibility d18e680 [Chris Fregly] Merge remote-tracking branch 'upstream/master' b3b0ff1 [Chris Fregly] [SPARK-1981] Add AWS Kinesis streaming support (cherry picked from commit 91f9504) Signed-off-by: Tathagata Das <[email protected]>

nchammas · 2014-08-02T21:26:13Z

This was an epic pull request. Nice work, people.

pdeyhim · 2014-08-02T21:27:51Z

Awsome work!! @tdas @cfregly

Author: Chris Fregly <[email protected]> Closes apache#1434 from cfregly/master and squashes the following commits: 4774581 [Chris Fregly] updated docs, renamed retry to retryRandom to be more clear, removed retries around store() method 0393795 [Chris Fregly] moved Kinesis examples out of examples/ and back into extras/kinesis-asl 691a6be [Chris Fregly] fixed tests and formatting, fixed a bug with JavaKinesisWordCount during union of streams 0e1c67b [Chris Fregly] Merge remote-tracking branch 'upstream/master' 74e5c7c [Chris Fregly] updated per TD's feedback. simplified examples, updated docs e33cbeb [Chris Fregly] Merge remote-tracking branch 'upstream/master' bf614e9 [Chris Fregly] per matei's feedback: moved the kinesis examples into the examples/ dir d17ca6d [Chris Fregly] per TD's feedback: updated docs, simplified the KinesisUtils api 912640c [Chris Fregly] changed the foundKinesis class to be a publically-avail class db3eefd [Chris Fregly] Merge remote-tracking branch 'upstream/master' 21de67f [Chris Fregly] Merge remote-tracking branch 'upstream/master' 6c39561 [Chris Fregly] parameterized the versions of the aws java sdk and kinesis client 338997e [Chris Fregly] improve build docs for kinesis 828f8ae [Chris Fregly] more cleanup e7c8978 [Chris Fregly] Merge remote-tracking branch 'upstream/master' cd68c0d [Chris Fregly] fixed typos and backward compatibility d18e680 [Chris Fregly] Merge remote-tracking branch 'upstream/master' b3b0ff1 [Chris Fregly] [SPARK-1981] Add AWS Kinesis streaming support

rxin · 2014-09-30T19:29:40Z

extras/kinesis-asl/src/main/resources/log4j.properties

question: why do we need this log4j file here?

[SPARK-1981] Add AWS Kinesis streaming support

b3b0ff1

Merge remote-tracking branch 'upstream/master'

d18e680

Resolved conflict: project/SparkBuild.scala

srosenthal reviewed Jul 18, 2014
View reviewed changes

cfregly added 5 commits July 18, 2014 20:16

fixed typos and backward compatibility

cd68c0d

Merge remote-tracking branch 'upstream/master'

e7c8978

more cleanup

828f8ae

improve build docs for kinesis

338997e

parameterized the versions of the aws java sdk and kinesis client

6c39561

tdas reviewed Jul 29, 2014
View reviewed changes

tdas reviewed Aug 1, 2014
View reviewed changes

moved Kinesis examples out of examples/ and back into extras/kinesis-asl

0393795

updated the build to only include kinesis-asl inside the examples jar when -Pkinesis-asl is specified

tdas reviewed Aug 2, 2014
View reviewed changes

cfregly mentioned this pull request Aug 2, 2014

SPARK-2770: Rename spark-ganglia-lgpl to ganglia-lgpl #1730

Closed

tdas reviewed Aug 2, 2014
View reviewed changes

updated docs, renamed retry to retryRandom to be more clear, removed

4774581

retries around store() method

asfgit closed this in 91f9504 Aug 2, 2014

rxin reviewed Sep 30, 2014
View reviewed changes

extras/kinesis-asl/src/main/resources/log4j.properties

Copy link

Contributor

rxin Sep 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: why do we need this log4j file here?

[SPARK-1981] Add AWS Kinesis streaming support #1434

[SPARK-1981] Add AWS Kinesis streaming support #1434

Uh oh!

Conversation

cfregly commented Jul 16, 2014

Uh oh!

AmplabJenkins commented Jul 16, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateiz commented Jul 26, 2014

Uh oh!

mateiz commented Jul 26, 2014

Uh oh!

SparkQA commented Jul 26, 2014

Uh oh!

cfregly commented Jul 27, 2014

Uh oh!

cfregly commented Jul 27, 2014

Uh oh!

cfregly commented Jul 27, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateiz commented Jul 29, 2014

Uh oh!

tdas commented Jul 29, 2014

Uh oh!

tdas commented Jul 29, 2014

Uh oh!

SparkQA commented Jul 29, 2014

Uh oh!

SparkQA commented Jul 29, 2014

Uh oh!

tdas commented Jul 29, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 2, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 2, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 2, 2014

Uh oh!

SparkQA commented Aug 2, 2014

Uh oh!

tdas commented Aug 2, 2014

Uh oh!

nchammas commented Aug 2, 2014

Uh oh!

pdeyhim commented Aug 2, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!