Skip to content

[SPARK-32495][2.4] Update jackson versions to a maintained release, to fix various security vulnerabilities. #29334

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

[SPARK-32495][2.4] Update jackson versions to a maintained release, to fix various security vulnerabilities. #29334

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
08af11f
Update jackson versions to a maintained release, to fix various secur…
ScrapCodes Aug 3, 2020
75b8155
update scala-reflect dependency on core, as in the newer version of j…
ScrapCodes Aug 3, 2020
7cff657
NON_NULL -> NON_ABSENT annotation change.
ScrapCodes Aug 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,10 @@
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-reflect</artifactId>
Copy link
Contributor

@Fokko Fokko Aug 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this one needed?

Copy link
Member Author

@ScrapCodes ScrapCodes Aug 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is required because, jackson-scala-module underwent this change:

FasterXML/jackson-module-scala@d6071ac#diff-fdc3abdfd754eeb24090dbd90aeec2ce

And Spark-core uses scala-reflect library at one place.

Copy link
Member Author

@ScrapCodes ScrapCodes Aug 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

org.apache.spark.deploy.security.HBaseDelegationTokenProvider

</dependency>
<dependency>
<groupId>org.json4s</groupId>
<artifactId>json4s-jackson_${scala.binary.version}</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/deploy/rest/SubmitRestProtocolMessage.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ import org.apache.spark.util.Utils
* (2) the Spark version of the client / server
* (3) an optional message
*/
@JsonInclude(Include.NON_NULL)
@JsonInclude(Include.NON_ABSENT)
@JsonAutoDetect(getterVisibility = Visibility.ANY, setterVisibility = Visibility.ANY)
@JsonPropertyOrder(alphabetic = true)
private[rest] abstract class SubmitRestProtocolMessage {
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ import org.apache.spark.internal.Logging
* There is no particular relationship between an operation scope and a stage or a job.
* A scope may live inside one stage (e.g. map) or span across multiple jobs (e.g. take).
*/
@JsonInclude(Include.NON_NULL)
@JsonInclude(Include.NON_ABSENT)
@JsonPropertyOrder(Array("id", "name", "parent"))
private[spark] class RDDOperationScope(
val name: String,
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/status/KVUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ private[spark] object KVUtils extends Logging {
private[spark] class KVStoreScalaSerializer extends KVStoreSerializer {

mapper.registerModule(DefaultScalaModule)
mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL)
mapper.setSerializationInclusion(JsonInclude.Include.NON_ABSENT)

}

Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/status/api/v1/JacksonMessageWriter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ private[v1] class JacksonMessageWriter extends MessageBodyWriter[Object]{
}
mapper.registerModule(com.fasterxml.jackson.module.scala.DefaultScalaModule)
mapper.enable(SerializationFeature.INDENT_OUTPUT)
mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL)
mapper.setSerializationInclusion(JsonInclude.Include.NON_ABSENT)
mapper.setDateFormat(JacksonMessageWriter.makeISODateFormat)

override def isWriteable(
Expand Down
16 changes: 8 additions & 8 deletions dev/deps/spark-deps-hadoop-2.6
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,16 +86,16 @@ htrace-core/3.0.4//htrace-core-3.0.4.jar
httpclient/4.5.6//httpclient-4.5.6.jar
httpcore/4.4.10//httpcore-4.4.10.jar
ivy/2.4.0//ivy-2.4.0.jar
jackson-annotations/2.6.7//jackson-annotations-2.6.7.jar
jackson-annotations/2.9.10//jackson-annotations-2.9.10.jar
jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
jackson-core/2.6.7//jackson-core-2.6.7.jar
jackson-databind/2.6.7.3//jackson-databind-2.6.7.3.jar
jackson-dataformat-yaml/2.6.7//jackson-dataformat-yaml-2.6.7.jar
jackson-core/2.9.10//jackson-core-2.9.10.jar
jackson-databind/2.9.10.5//jackson-databind-2.9.10.5.jar
jackson-dataformat-yaml/2.9.10//jackson-dataformat-yaml-2.9.10.jar
jackson-jaxrs/1.9.13//jackson-jaxrs-1.9.13.jar
jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations/2.6.7//jackson-module-jaxb-annotations-2.6.7.jar
jackson-module-paranamer/2.7.9//jackson-module-paranamer-2.7.9.jar
jackson-module-scala_2.11/2.6.7.1//jackson-module-scala_2.11-2.6.7.1.jar
jackson-module-jaxb-annotations/2.9.10//jackson-module-jaxb-annotations-2.9.10.jar
jackson-module-paranamer/2.9.10//jackson-module-paranamer-2.9.10.jar
jackson-module-scala_2.11/2.9.10//jackson-module-scala_2.11-2.9.10.jar
jackson-xc/1.9.13//jackson-xc-1.9.13.jar
janino/3.0.16//janino-3.0.16.jar
javassist/3.18.1-GA//javassist-3.18.1-GA.jar
Expand Down Expand Up @@ -178,7 +178,7 @@ shapeless_2.11/2.3.2//shapeless_2.11-2.3.2.jar
shims/0.7.45//shims-0.7.45.jar
slf4j-api/1.7.16//slf4j-api-1.7.16.jar
slf4j-log4j12/1.7.16//slf4j-log4j12-1.7.16.jar
snakeyaml/1.15//snakeyaml-1.15.jar
snakeyaml/1.23//snakeyaml-1.23.jar
snappy-java/1.1.7.5//snappy-java-1.1.7.5.jar
snappy/0.2//snappy-0.2.jar
spire-macros_2.11/0.13.0//spire-macros_2.11-0.13.0.jar
Expand Down
16 changes: 8 additions & 8 deletions dev/deps/spark-deps-hadoop-2.7
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,16 +86,16 @@ htrace-core/3.1.0-incubating//htrace-core-3.1.0-incubating.jar
httpclient/4.5.6//httpclient-4.5.6.jar
httpcore/4.4.10//httpcore-4.4.10.jar
ivy/2.4.0//ivy-2.4.0.jar
jackson-annotations/2.6.7//jackson-annotations-2.6.7.jar
jackson-annotations/2.9.10//jackson-annotations-2.9.10.jar
jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
jackson-core/2.6.7//jackson-core-2.6.7.jar
jackson-databind/2.6.7.3//jackson-databind-2.6.7.3.jar
jackson-dataformat-yaml/2.6.7//jackson-dataformat-yaml-2.6.7.jar
jackson-core/2.9.10//jackson-core-2.9.10.jar
jackson-databind/2.9.10.5//jackson-databind-2.9.10.5.jar
jackson-dataformat-yaml/2.9.10//jackson-dataformat-yaml-2.9.10.jar
jackson-jaxrs/1.9.13//jackson-jaxrs-1.9.13.jar
jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations/2.6.7//jackson-module-jaxb-annotations-2.6.7.jar
jackson-module-paranamer/2.7.9//jackson-module-paranamer-2.7.9.jar
jackson-module-scala_2.11/2.6.7.1//jackson-module-scala_2.11-2.6.7.1.jar
jackson-module-jaxb-annotations/2.9.10//jackson-module-jaxb-annotations-2.9.10.jar
jackson-module-paranamer/2.9.10//jackson-module-paranamer-2.9.10.jar
jackson-module-scala_2.11/2.9.10//jackson-module-scala_2.11-2.9.10.jar
jackson-xc/1.9.13//jackson-xc-1.9.13.jar
janino/3.0.16//janino-3.0.16.jar
javassist/3.18.1-GA//javassist-3.18.1-GA.jar
Expand Down Expand Up @@ -179,7 +179,7 @@ shapeless_2.11/2.3.2//shapeless_2.11-2.3.2.jar
shims/0.7.45//shims-0.7.45.jar
slf4j-api/1.7.16//slf4j-api-1.7.16.jar
slf4j-log4j12/1.7.16//slf4j-log4j12-1.7.16.jar
snakeyaml/1.15//snakeyaml-1.15.jar
snakeyaml/1.23//snakeyaml-1.23.jar
snappy-java/1.1.7.5//snappy-java-1.1.7.5.jar
snappy/0.2//snappy-0.2.jar
spire-macros_2.11/0.13.0//spire-macros_2.11-0.13.0.jar
Expand Down
16 changes: 8 additions & 8 deletions dev/deps/spark-deps-hadoop-3.1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,17 +86,17 @@ htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
httpclient/4.5.6//httpclient-4.5.6.jar
httpcore/4.4.10//httpcore-4.4.10.jar
ivy/2.4.0//ivy-2.4.0.jar
jackson-annotations/2.6.7//jackson-annotations-2.6.7.jar
jackson-annotations/2.9.10//jackson-annotations-2.9.10.jar
jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
jackson-core/2.6.7//jackson-core-2.6.7.jar
jackson-databind/2.6.7.3//jackson-databind-2.6.7.3.jar
jackson-dataformat-yaml/2.6.7//jackson-dataformat-yaml-2.6.7.jar
jackson-core/2.9.10//jackson-core-2.9.10.jar
jackson-databind/2.9.10.5//jackson-databind-2.9.10.5.jar
jackson-dataformat-yaml/2.9.10//jackson-dataformat-yaml-2.9.10.jar
jackson-jaxrs-base/2.7.8//jackson-jaxrs-base-2.7.8.jar
jackson-jaxrs-json-provider/2.7.8//jackson-jaxrs-json-provider-2.7.8.jar
jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations/2.6.7//jackson-module-jaxb-annotations-2.6.7.jar
jackson-module-paranamer/2.7.9//jackson-module-paranamer-2.7.9.jar
jackson-module-scala_2.11/2.6.7.1//jackson-module-scala_2.11-2.6.7.1.jar
jackson-module-jaxb-annotations/2.9.10//jackson-module-jaxb-annotations-2.9.10.jar
jackson-module-paranamer/2.9.10//jackson-module-paranamer-2.9.10.jar
jackson-module-scala_2.11/2.9.10//jackson-module-scala_2.11-2.9.10.jar
janino/3.0.16//janino-3.0.16.jar
javassist/3.18.1-GA//javassist-3.18.1-GA.jar
javax.annotation-api/1.2//javax.annotation-api-1.2.jar
Expand Down Expand Up @@ -199,7 +199,7 @@ shapeless_2.11/2.3.2//shapeless_2.11-2.3.2.jar
shims/0.7.45//shims-0.7.45.jar
slf4j-api/1.7.16//slf4j-api-1.7.16.jar
slf4j-log4j12/1.7.16//slf4j-log4j12-1.7.16.jar
snakeyaml/1.15//snakeyaml-1.15.jar
snakeyaml/1.23//snakeyaml-1.23.jar
snappy-java/1.1.7.5//snappy-java-1.1.7.5.jar
snappy/0.2//snappy-0.2.jar
spire-macros_2.11/0.13.0//spire-macros_2.11-0.13.0.jar
Expand Down
6 changes: 3 additions & 3 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,9 +158,9 @@
<scala.version>2.11.12</scala.version>
<scala.binary.version>2.11</scala.binary.version>
<codehaus.jackson.version>1.9.13</codehaus.jackson.version>
<fasterxml.jackson.version>2.6.7</fasterxml.jackson.version>
<fasterxml.jackson-module-scala.version>2.6.7.1</fasterxml.jackson-module-scala.version>
<fasterxml.jackson.databind.version>2.6.7.3</fasterxml.jackson.databind.version>
<fasterxml.jackson.version>2.9.10</fasterxml.jackson.version>
<fasterxml.jackson-module-scala.version>2.9.10</fasterxml.jackson-module-scala.version>
<fasterxml.jackson.databind.version>2.9.10.5</fasterxml.jackson.databind.version>
<snappy.version>1.1.7.5</snappy.version>
<netlib.java.version>1.1.2</netlib.java.version>
<calcite.version>1.2.0-incubating</calcite.version>
Expand Down
2 changes: 1 addition & 1 deletion project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -553,7 +553,7 @@ object DependencyOverrides {
lazy val settings = Seq(
dependencyOverrides += "com.google.guava" % "guava" % "14.0.1",
dependencyOverrides += "commons-io" % "commons-io" % "2.4",
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.7.3",
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.10.5",
dependencyOverrides += "jline" % "jline" % "2.14.6")
}

Expand Down
41 changes: 23 additions & 18 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmarks.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,13 @@ object JSONBenchmarks {
}

/*
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
Java HotSpot(TM) 64-Bit Server VM 1.8.0_241-b07 on Mac OS X 10.15.5
Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz

JSON schema inferring: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------
No encoding 38902 / 39282 2.6 389.0 1.0X
UTF-8 is set 56959 / 57261 1.8 569.6 0.7X
JSON schema inferring: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------
No encoding 7664 / 7686 13.0 76.6 1.0X
UTF-8 is set 9981 / 10180 10.0 99.8 0.8X
*/
benchmark.run()
}
Expand Down Expand Up @@ -113,12 +114,13 @@ object JSONBenchmarks {
}

/*
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
Java HotSpot(TM) 64-Bit Server VM 1.8.0_241-b07 on Mac OS X 10.15.5
Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz

JSON per-line parsing: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------
No encoding 25947 / 26188 3.9 259.5 1.0X
UTF-8 is set 46319 / 46417 2.2 463.2 0.6X
JSON per-line parsing: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------
No encoding 5832 / 5915 17.1 58.3 1.0X
UTF-8 is set 8687 / 8884 11.5 86.9 0.7X
*/
benchmark.run()
}
Expand Down Expand Up @@ -161,12 +163,13 @@ object JSONBenchmarks {
}

/*
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
Java HotSpot(TM) 64-Bit Server VM 1.8.0_241-b07 on Mac OS X 10.15.5
Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz

JSON parsing of wide lines: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------
No encoding 45543 / 45660 0.2 4554.3 1.0X
UTF-8 is set 65737 / 65957 0.2 6573.7 0.7X
JSON parsing of wide lines: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------
No encoding 9367 / 9475 1.1 936.7 1.0X
UTF-8 is set 14211 / 14397 0.7 1421.1 0.7X
*/
benchmark.run()
}
Expand Down Expand Up @@ -196,12 +199,14 @@ object JSONBenchmarks {
}

/*
Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
Java HotSpot(TM) 64-Bit Server VM 1.8.0_241-b07 on Mac OS X 10.15.5
Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz

Count a dataset with 10 columns: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------
Select 10 columns + count() 9961 / 10006 1.0 996.1 1.0X
Select 1 column + count() 8355 / 8470 1.2 835.5 1.2X
Select 10 columns + count() 2074 / 2112 4.8 207.4 1.0X
Select 1 column + count() 1753 / 1792 5.7 175.3 1.2X

*/
benchmark.run()
}
Expand Down