Skip to content

Revert "[SPARK-24601][SPARK-27051][BACKPORT][CORE] Update to Jackson 2.9.8 #24493

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Revert "[SPARK-24601][SPARK-27051][BACKPORT][CORE] Update to Jackson 2.9.8 #24493

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,10 +224,6 @@
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-reflect</artifactId>
</dependency>
<dependency>
<groupId>org.json4s</groupId>
<artifactId>json4s-jackson_${scala.binary.version}</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/deploy/rest/SubmitRestProtocolMessage.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ import org.apache.spark.util.Utils
* (2) the Spark version of the client / server
* (3) an optional message
*/
@JsonInclude(Include.NON_ABSENT)
@JsonInclude(Include.NON_NULL)
@JsonAutoDetect(getterVisibility = Visibility.ANY, setterVisibility = Visibility.ANY)
@JsonPropertyOrder(alphabetic = true)
private[rest] abstract class SubmitRestProtocolMessage {
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ import org.apache.spark.internal.Logging
* There is no particular relationship between an operation scope and a stage or a job.
* A scope may live inside one stage (e.g. map) or span across multiple jobs (e.g. take).
*/
@JsonInclude(Include.NON_ABSENT)
@JsonInclude(Include.NON_NULL)
@JsonPropertyOrder(Array("id", "name", "parent"))
private[spark] class RDDOperationScope(
val name: String,
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/status/KVUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ private[spark] object KVUtils extends Logging {
private[spark] class KVStoreScalaSerializer extends KVStoreSerializer {

mapper.registerModule(DefaultScalaModule)
mapper.setSerializationInclusion(JsonInclude.Include.NON_ABSENT)
mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL)

}

Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/status/api/v1/JacksonMessageWriter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ private[v1] class JacksonMessageWriter extends MessageBodyWriter[Object]{
}
mapper.registerModule(com.fasterxml.jackson.module.scala.DefaultScalaModule)
mapper.enable(SerializationFeature.INDENT_OUTPUT)
mapper.setSerializationInclusion(JsonInclude.Include.NON_ABSENT)
mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL)
mapper.setDateFormat(JacksonMessageWriter.makeISODateFormat)

override def isWriteable(
Expand Down
16 changes: 8 additions & 8 deletions dev/deps/spark-deps-hadoop-2.6
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,16 +86,16 @@ htrace-core-3.0.4.jar
httpclient-4.5.6.jar
httpcore-4.4.10.jar
ivy-2.4.0.jar
jackson-annotations-2.9.8.jar
jackson-core-2.9.8.jar
jackson-annotations-2.6.7.jar
jackson-core-2.6.7.jar
jackson-core-asl-1.9.13.jar
jackson-databind-2.9.8.jar
jackson-dataformat-yaml-2.9.8.jar
jackson-databind-2.6.7.1.jar
jackson-dataformat-yaml-2.6.7.jar
jackson-jaxrs-1.9.13.jar
jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations-2.9.8.jar
jackson-module-paranamer-2.9.8.jar
jackson-module-scala_2.11-2.9.8.jar
jackson-module-jaxb-annotations-2.6.7.jar
jackson-module-paranamer-2.7.9.jar
jackson-module-scala_2.11-2.6.7.1.jar
jackson-xc-1.9.13.jar
janino-3.0.9.jar
javassist-3.18.1-GA.jar
Expand Down Expand Up @@ -178,7 +178,7 @@ shapeless_2.11-2.3.2.jar
shims-0.7.45.jar
slf4j-api-1.7.16.jar
slf4j-log4j12-1.7.16.jar
snakeyaml-1.23.jar
snakeyaml-1.15.jar
snappy-0.2.jar
snappy-java-1.1.7.3.jar
spire-macros_2.11-0.13.0.jar
Expand Down
16 changes: 8 additions & 8 deletions dev/deps/spark-deps-hadoop-2.7
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,16 +86,16 @@ htrace-core-3.1.0-incubating.jar
httpclient-4.5.6.jar
httpcore-4.4.10.jar
ivy-2.4.0.jar
jackson-annotations-2.9.8.jar
jackson-core-2.9.8.jar
jackson-annotations-2.6.7.jar
jackson-core-2.6.7.jar
jackson-core-asl-1.9.13.jar
jackson-databind-2.9.8.jar
jackson-dataformat-yaml-2.9.8.jar
jackson-databind-2.6.7.1.jar
jackson-dataformat-yaml-2.6.7.jar
jackson-jaxrs-1.9.13.jar
jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations-2.9.8.jar
jackson-module-paranamer-2.9.8.jar
jackson-module-scala_2.11-2.9.8.jar
jackson-module-jaxb-annotations-2.6.7.jar
jackson-module-paranamer-2.7.9.jar
jackson-module-scala_2.11-2.6.7.1.jar
jackson-xc-1.9.13.jar
janino-3.0.9.jar
javassist-3.18.1-GA.jar
Expand Down Expand Up @@ -179,7 +179,7 @@ shapeless_2.11-2.3.2.jar
shims-0.7.45.jar
slf4j-api-1.7.16.jar
slf4j-log4j12-1.7.16.jar
snakeyaml-1.23.jar
snakeyaml-1.15.jar
snappy-0.2.jar
snappy-java-1.1.7.3.jar
spire-macros_2.11-0.13.0.jar
Expand Down
16 changes: 8 additions & 8 deletions dev/deps/spark-deps-hadoop-3.1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,17 +86,17 @@ htrace-core4-4.1.0-incubating.jar
httpclient-4.5.6.jar
httpcore-4.4.10.jar
ivy-2.4.0.jar
jackson-annotations-2.9.8.jar
jackson-core-2.9.8.jar
jackson-annotations-2.6.7.jar
jackson-core-2.6.7.jar
jackson-core-asl-1.9.13.jar
jackson-databind-2.9.8.jar
jackson-dataformat-yaml-2.9.8.jar
jackson-databind-2.6.7.1.jar
jackson-dataformat-yaml-2.6.7.jar
jackson-jaxrs-base-2.7.8.jar
jackson-jaxrs-json-provider-2.7.8.jar
jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations-2.9.8.jar
jackson-module-paranamer-2.9.8.jar
jackson-module-scala_2.11-2.9.8.jar
jackson-module-jaxb-annotations-2.6.7.jar
jackson-module-paranamer-2.7.9.jar
jackson-module-scala_2.11-2.6.7.1.jar
janino-3.0.9.jar
javassist-3.18.1-GA.jar
javax.annotation-api-1.2.jar
Expand Down Expand Up @@ -199,7 +199,7 @@ shapeless_2.11-2.3.2.jar
shims-0.7.45.jar
slf4j-api-1.7.16.jar
slf4j-log4j12-1.7.16.jar
snakeyaml-1.23.jar
snakeyaml-1.15.jar
snappy-0.2.jar
snappy-java-1.1.7.3.jar
spire-macros_2.11-0.13.0.jar
Expand Down
7 changes: 4 additions & 3 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,8 @@
<scala.version>2.11.12</scala.version>
<scala.binary.version>2.11</scala.binary.version>
<codehaus.jackson.version>1.9.13</codehaus.jackson.version>
<fasterxml.jackson.version>2.9.8</fasterxml.jackson.version>
<fasterxml.jackson.version>2.6.7</fasterxml.jackson.version>
<fasterxml.jackson.databind.version>2.6.7.1</fasterxml.jackson.databind.version>
<snappy.version>1.1.7.3</snappy.version>
<netlib.java.version>1.1.2</netlib.java.version>
<calcite.version>1.2.0-incubating</calcite.version>
Expand Down Expand Up @@ -633,7 +634,7 @@
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${fasterxml.jackson.version}</version>
<version>${fasterxml.jackson.databind.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
Expand All @@ -645,7 +646,7 @@
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
<version>${fasterxml.jackson.version}</version>
<version>${fasterxml.jackson.databind.version}</version>
<exclusions>
<exclusion>
<groupId>com.google.guava</groupId>
Expand Down
7 changes: 0 additions & 7 deletions ...alyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,13 +244,6 @@ class JsonExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper with
"1234")
}

test("some big value") {
val value = "x" * 3000
checkEvaluation(
GetJsonObject(NonFoldableLiteral((s"""{"big": "$value"}""")),
NonFoldableLiteral("$.big")), value)
}

val jsonTupleQuery = Literal("f1") ::
Literal("f2") ::
Literal("f3") ::
Expand Down
33 changes: 15 additions & 18 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmarks.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,13 +72,12 @@ object JSONBenchmarks {
}

/*
Java HotSpot(TM) 64-Bit Server VM 1.8.0_172-b11 on Mac OS X 10.13.5
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz

JSON schema inferring: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------
No encoding 45908 / 46480 2.2 459.1 1.0X
UTF-8 is set 68469 / 69762 1.5 684.7 0.7X
JSON schema inferring: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------
No encoding 38902 / 39282 2.6 389.0 1.0X
UTF-8 is set 56959 / 57261 1.8 569.6 0.7X
*/
benchmark.run()
}
Expand Down Expand Up @@ -114,13 +113,12 @@ object JSONBenchmarks {
}

/*
Java HotSpot(TM) 64-Bit Server VM 1.8.0_172-b11 on Mac OS X 10.13.5
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz

JSON per-line parsing: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------
No encoding 9982 / 10237 10.0 99.8 1.0X
UTF-8 is set 16373 / 16806 6.1 163.7 0.6X
JSON per-line parsing: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------
No encoding 25947 / 26188 3.9 259.5 1.0X
UTF-8 is set 46319 / 46417 2.2 463.2 0.6X
*/
benchmark.run()
}
Expand Down Expand Up @@ -163,13 +161,12 @@ object JSONBenchmarks {
}

/*
Java HotSpot(TM) 64-Bit Server VM 1.8.0_172-b11 on Mac OS X 10.13.5
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz

JSON parsing of wide lines: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------
No encoding 26038 / 26386 0.4 2603.8 1.0X
UTF-8 is set 28343 / 28557 0.4 2834.3 0.9X
JSON parsing of wide lines: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------
No encoding 45543 / 45660 0.2 4554.3 1.0X
UTF-8 is set 65737 / 65957 0.2 6573.7 0.7X
*/
benchmark.run()
}
Expand Down