[SPARK-12269][Streaming][Kinesis] Update aws-java-sdk version #10256

BrianLondon · 2015-12-10T21:01:38Z

The current Spark Streaming kinesis connector references a quite old version 1.9.40 of the AWS Java SDK (1.10.40 is current). Numerous AWS features including Kinesis Firehose are unavailable in 1.9. Those two versions of the AWS SDK in turn require conflicting versions of Jackson (2.4.4 and 2.5.3 respectively) such that one cannot include the current AWS SDK in a project that also uses the Spark Streaming Kinesis ASL.

holdenk · 2015-12-10T22:54:42Z

Have you tested the current kinesis and works with the new API? Looking online it seems like we might also need to bump our kinesis client library if we upgrade the general aws sdk version.

holdenk · 2015-12-10T22:56:55Z

Note we also depend depend on com.fasterxml.jackson.core 2.4.4 in Spark

BrianLondon · 2015-12-10T23:06:59Z

I ran the demo with the following two commands against a stream (test-stream1) on my AWS account. It performed as expected.

bin/run-example streaming.KinesisWordProducerASL test-stream1 https://kinesis.us-east-1.amazonaws.com 100 10
bin/run-example streaming.KinesisWordCountASL testapp test-stream1 https://kinesis.us-east-1.amazonaws.com

I happy to bump the KCL library as well. With the bumped com.fasterxml.jackson.core version, the project build and passed all tests except docker-integration, which I can't run. AWS Java SDK 1.10.40 calls a function that isn't provided by Jackson 2.4.x. If raising that to 2.5.x isn't an option, then we can find whatever the last version of AWS Java SDK to not use it is. I know it works with 1.10.24, so somewhere between 1.10.24 and 1.10.40 could be a temporary solution.

holdenk · 2015-12-10T23:11:57Z

Cool - if we don't need to upgrade the KCL library then thats probably good. Maybe good to see if @tdas or someone similar can take a look?

JoshRosen · 2015-12-10T23:32:04Z

/cc @brkyvz RE: the KCL library.

zsxwing · 2015-12-10T23:34:14Z

pom.xml

Is it dangerous to upgrade jackson? It's also used by SQL.

Yeah, this makes me nervous. I'll run the tests to see what happens.

@srowen did you manage to run the tests? We'd need to be sure nothing breaks with this version bump!

@MLnick the tests seem to have run OK in https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2203/consoleFull (although jenkins seems to be having some issues)

@holdenk Unfortunately the tests that actually mattered didn't run, they were ignored :(

[info] WithAggregationKinesisStreamSuite: [info] - KinesisUtils API (202 milliseconds) [info] - RDD generation (24 milliseconds) [info] - basic operation [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - custom message handling [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - failure recovery [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Prepare KinesisTestUtils [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] WithoutAggregationKinesisStreamSuite: [info] - KinesisUtils API (5 milliseconds) [info] - RDD generation (6 milliseconds) [info] - basic operation [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - custom message handling [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - failure recovery [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Prepare KinesisTestUtils [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] WithAggregationKinesisBackedBlockRDDSuite: [info] - Basic reading from Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Read data available in both block manager and Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Read data available only in block manager, not in Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Read data available only in Kinesis, not in block manager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Read data available partially in block manager, rest in Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Test isBlockValid skips block fetching from block manager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Test whether RDD is valid after removing blocks from block anager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] WithoutAggregationKinesisBackedBlockRDDSuite: [info] - Basic reading from Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Read data available in both block manager and Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Read data available only in block manager, not in Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Read data available only in Kinesis, not in block manager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Read data available partially in block manager, rest in Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Test isBlockValid skips block fetching from block manager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!! [info] - Test whether RDD is valid after removing blocks from block anager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!

Those tests are i think enabled only in the default jenkins PR builder.

brkyvz · 2015-12-11T01:17:26Z

@BrianLondon What happens if you change the version of KCL to 1.6.1? Could you also please add a new line to anywhere in extras/kinesis-asl? Otherwise Kinesis tests don't run and we have no idea whether this is a safe change.

Thanks!

SparkQA · 2015-12-11T14:17:10Z

Test build #2203 has finished for PR 10256 at commit 47d0a59.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

BrianLondon · 2015-12-11T16:38:40Z

@brkyvz Pushed up the KCL version and added a newline to KinesisReciever. Tests passed for me locally on a hadoop 2.6 build and the kinesis demos ran against a live AWS stream.

BrianLondon · 2015-12-14T15:23:55Z

@holdenk bump

holdenk · 2015-12-14T19:05:13Z

Does anyone working on SQL (e.g. @marmbrus ) have any concerns about the Jackson upgrade?

MLnick · 2015-12-14T19:09:35Z

@brkyvz @holdenk @tdas any particular reason we depend on aws-sdk-java explicitly? The KCL brings in those dependencies (in fact from 1.5.0 KCL only depends on the necessary AWS SDK components, i.e. core, kinesis, dynamodb and cloudwatch). Depending only on KCL would make things cleaner and avoid potential issues such as not bumping versions at the same time (e.g. #8957 and related mailing list discussion)

holdenk · 2015-12-14T19:33:25Z

I'm not really sure why - skimming the git logs it seems the aws-sdk-java sdk dependency was added at the same time as the kcl dependency.

BrianLondon · 2015-12-14T20:44:03Z

It appears kcl pulls in 1.10.20. I'm all for dropping the explicit dependency. That probably wouldn't have prevented the regression in Spark 1.5.2 though. At any rate, the jackson bump is necessary if someone wants to use the latest aws sdk (e.g. for Firehose).

BrianLondon · 2015-12-15T17:29:53Z

I removed the explicit dependence on the AWS Java SDK. There's a newline that was added to KinesisReceiver.scala since the last test run. I think if tests are re-run the kinesis part should be included, although someone more knowledgable than me about Travis can weigh in.

Tests and the kinesis example worked locally for me for what it's worth.

JoshRosen · 2015-12-15T23:16:49Z

FYI, a bunch of spark-redshift users have run into the Jackson dependency conflict issue that's discussed here; see databricks/spark-redshift#133 and databricks/spark-redshift#121. Not commenting either way w.r.t. the dep. concerns themselves, but just wanted to point out that the pain of the Jackson conflict has impacts for a bunch of other applications too.

BrianLondon · 2015-12-16T03:16:33Z

Yeah, that looks like the same underlying issue of using AWS Java SDK 1.10.0 or later with Spark. I believe this version bump will fix it. You can sometimes get around it by adding the following to your sbt build:

dependencyOverrides ++= Set(
  "com.fasterxml.jackson.core" % "jackson-core" % "2.4.4",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4",
  "com.fasterxml.jackson.core" % "jackson-annotations" % "2.4.4"
)

That doesn't work however if the calls to the AWS library use any Jackson 2.5.x functionality and some of them do.

MLnick · 2015-12-18T18:32:43Z

I've also run the Kinesis tests locally (against a live AWS account) against this PR and all tests pass. I'm +1 on this subject to agreement for the jackson dependency bump, for which I defer to SQL folks e.g. @marmbrus

marmbrus · 2015-12-18T22:03:35Z

If the SQL/Hive tests are passing I'm good. @srowen usually has good insight on dependency changes.

srowen · 2015-12-21T11:52:35Z

Can this wait until 2.0? I think it's going to anyway now. Then the dependency issue is fine.

SparkQA · 2015-12-30T21:13:46Z

Test build #2271 has finished for PR 10256 at commit 64858df.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

BrianLondon · 2016-01-08T20:12:44Z

Rebased against current master, ran tests (except for docker) locally, and tested kinesis example against aws hosted stream. All worked.

My preference would be to get the version bump in as soon as practicable for two reasons. First, and mostly selfishly, the inability to link any library that requires Jackson 2.5.x features is causing us problems in our production environment. Second, we currently can just bump the version number because the Spark codebase appears compatible with both Jackson versions at present, but the longer we wait the more likely someone introduces an incompatible change.

That said, I fully understand the reluctance to replace a heavily used library mid release cycle.

SparkQA · 2016-01-08T20:27:54Z

Test build #2354 has finished for PR 10256 at commit 6f3bbfd.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-01-08T20:34:17Z

Ah right, you'll need to acknowledge the dep changes in the build file:

To update the manifest file, run './dev/test-dependencies.sh --replace-manifest'.
diff --git a/dev/deps/spark-deps-hadoop-2.2 b/dev/pr-deps/spark-deps-hadoop-2.2
index e4373f7..cd3ff29 100644
--- a/dev/deps/spark-deps-hadoop-2.2
+++ b/dev/pr-deps/spark-deps-hadoop-2.2
@@ -84,13 +84,13 @@ hadoop-yarn-server-web-proxy-2.2.0.jar
 httpclient-4.3.2.jar
 httpcore-4.3.2.jar
 ivy-2.4.0.jar
-jackson-annotations-2.4.4.jar
-jackson-core-2.4.4.jar
+jackson-annotations-2.5.3.jar
+jackson-core-2.5.3.jar
 jackson-core-asl-1.9.13.jar
-jackson-databind-2.4.4.jar
+jackson-databind-2.5.3.jar
 jackson-jaxrs-1.9.13.jar
 jackson-mapper-asl-1.9.13.jar
-jackson-module-scala_2.10-2.4.4.jar
+jackson-module-scala_2.10-2.5.3.jar
 jackson-xc-1.9.13.jar
 janino-2.7.8.jar
 jansi-1.4.jar

This LGTM but I do think we'll have to keep it for 2.x only.

BrianLondon · 2016-01-09T19:06:25Z

Manifest updated. Is there documented somewhere what all the tests Travis runs are? It seems to differ from what's described at http://spark.apache.org/docs/latest/building-spark.html

srowen · 2016-01-10T10:44:55Z

@BrianLondon it should be running dev/run-tests, so that's something you can inspect or run locally if you need to. The doc there describes how to run one test.

SparkQA · 2016-01-10T12:33:43Z

Test build #2358 has finished for PR 10256 at commit a1f75c1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-01-10T13:47:00Z

I'm going to go for this one since I think we may want to, in general, consider updating Spark dependencies in 2.x anyway. 2.5 is even old for Jackson, so good to bump up here as a side effect I think.

srowen · 2016-01-11T09:32:24Z

Merged to master

cowtowncoder · 2016-01-28T17:46:34Z

I know this issue is closed, but one suggestion I have would be to specify the latest Jackson 2.5 version, 2.5.5 instead of 2.5.3. While I understand caution in general, Jackson patch versions try to limit inclusion of fixes to safe ones and specifically there are no API changes. This also means that while it is good to keep even patch versions in sync across components there is very low risk of incompatibilities even if patch versions do differ.

In this specific case, jackson-databind 2.5.4 has 10 bug fixes -- see https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.5.4 -- this is the main reason I would recommend use of latest version.

srowen · 2016-01-28T17:56:10Z

I think that's OK. I was intending to propose a series of dependency updates for 2.x like this sort of thing.

cowtowncoder · 2016-01-28T18:07:50Z

@srowen Ah. Yes, just wanted to mention it; I don't have enough context so there may be specific reasons for picking up particular versions. Glad to see version upgrades coming along. My interests are selfish, being both the main author of Jackson, and a (relative new) Spark user at work. :)

With that, if there are problems from Jackson side I could help, do not hesitate to cc me and/or file issues at https://github.com/FasterXML/jackson-databind (or other specific components). I know version compatibility is an annoying issue, and would like to help if possible.

zsxwing reviewed Dec 10, 2015
View reviewed changes

JoshRosen mentioned this pull request Dec 15, 2015

java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.ClientConfiguration databricks/spark-redshift#133

Closed

BrianLondon added 2 commits January 7, 2016 23:29

Updating aws sdk and jackson versions

ab5b066

Increase KCL version and ping the kinesis-asl

6f3bbfd

BrianLondon force-pushed the master branch from 64858df to 6f3bbfd Compare January 8, 2016 04:33

Updating manifest files

a1f75c1

asfgit closed this in 8fe928b Jan 11, 2016

[SPARK-12269][Streaming][Kinesis] Update aws-java-sdk version #10256

[SPARK-12269][Streaming][Kinesis] Update aws-java-sdk version #10256

Uh oh!

Conversation

BrianLondon commented Dec 10, 2015

Uh oh!

holdenk commented Dec 10, 2015

Uh oh!

holdenk commented Dec 10, 2015

Uh oh!

BrianLondon commented Dec 10, 2015

Uh oh!

holdenk commented Dec 10, 2015

Uh oh!

JoshRosen commented Dec 10, 2015

Uh oh!

zsxwing Dec 10, 2015

Choose a reason for hiding this comment

Uh oh!

srowen Dec 11, 2015

Choose a reason for hiding this comment

Uh oh!

MLnick Dec 14, 2015

Choose a reason for hiding this comment

Uh oh!

holdenk Dec 14, 2015

Choose a reason for hiding this comment

Uh oh!

brkyvz Dec 14, 2015

Choose a reason for hiding this comment

Uh oh!

holdenk Dec 15, 2015

Choose a reason for hiding this comment

Uh oh!

tdas Dec 15, 2015

Choose a reason for hiding this comment

Uh oh!

brkyvz commented Dec 11, 2015

Uh oh!

SparkQA commented Dec 11, 2015

Uh oh!

BrianLondon commented Dec 11, 2015

Uh oh!

BrianLondon commented Dec 14, 2015

Uh oh!

holdenk commented Dec 14, 2015

Uh oh!

MLnick commented Dec 14, 2015

Uh oh!

holdenk commented Dec 14, 2015

Uh oh!

BrianLondon commented Dec 14, 2015

Uh oh!

BrianLondon commented Dec 15, 2015

Uh oh!

JoshRosen commented Dec 15, 2015

Uh oh!

BrianLondon commented Dec 16, 2015

Uh oh!

MLnick commented Dec 18, 2015

Uh oh!

marmbrus commented Dec 18, 2015

Uh oh!

srowen commented Dec 21, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

BrianLondon commented Jan 8, 2016

Uh oh!

SparkQA commented Jan 8, 2016

Uh oh!

srowen commented Jan 8, 2016

Uh oh!

BrianLondon commented Jan 9, 2016

Uh oh!

srowen commented Jan 10, 2016

Uh oh!

SparkQA commented Jan 10, 2016

Uh oh!

srowen commented Jan 10, 2016