[SPARK-35870][BUILD] Upgrade Jetty to 9.4.42 #33053

sarutak · 2021-06-24T06:49:15Z

What changes were proposed in this pull request?

This PR upgrades Jetty to 9.4.42.
In the current master, 9.4.40 is used.
9.4.41 and 9.4.42 include the following updates.
https://github.com/eclipse/jetty.project/releases/tag/jetty-9.4.41.v20210516
https://github.com/eclipse/jetty.project/releases/tag/jetty-9.4.42.v20210604

Why are the changes needed?

Mainly for CVE-2021-28169.
https://nvd.nist.gov/vuln/detail/CVE-2021-28169
This CVE might little affect Spark, but just in case.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

CI.

…-jetty-9.4.42

dongjoon-hyun

+1 (Pending CIs).
Yes, it's good to catch up the dependency for Apache Spark 3.2.0.

SparkQA · 2021-06-24T07:38:03Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44775/

SparkQA · 2021-06-24T10:04:17Z

Test build #140246 has finished for PR 33053 at commit d290b38.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2021-06-24T18:34:16Z

Merged to master. Thank you @dongjoon-hyun for the review!

HyukjinKwon · 2021-06-25T02:06:29Z

Hmmm .. @sarutak, I don't know why and how but seems like JDBC docker test is flaky after this commit (although the test passed in this PR ... huh?) https://github.com/apache/spark/runs/2910056787:


[info] - SPARK-33034: ALTER TABLE ... add new columns (173 milliseconds)
[info] - SPARK-33034: ALTER TABLE ... drop column *** FAILED *** (126 milliseconds)
[info]   "Cannot delete missing field bad_column in postgresql.alt_table schema: root
[info]    |-- C2: string (nullable = true)
[info]   ; line 1 pos 0;
[info]   'AlterTableDropColumns [unresolvedfieldname(bad_column)]
[info]   +- ResolvedTable org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog@7f4b7516, alt_table, JDBCTable(alt_table,StructType(StructField(C2,StringType,true)),org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions@5842301d), [C2#1879]
[info]   " did not contain "Cannot delete missing field bad_column in alt_table schema" (V2JDBCTest.scala:106)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info]   at org.apache.spark.sql.jdbc.v2.V2JDBCTest.$anonfun$$init$$6(V2JDBCTest.scala:106)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1461)
[info]   at org.apache.spark.sql.test.SQLTestUtilsBase.withTable(SQLTestUtils.scala:305)
[info]   at org.apache.spark.sql.test.SQLTestUtilsBase.withTable$(SQLTestUtils.scala:303)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.withTable(DockerJDBCIntegrationSuite.scala:95)
[info]   at org.apache.spark.sql.jdbc.v2.V2JDBCTest.$anonfun$$init$$5(V2JDBCTest.scala:95)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:190)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:188)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:200)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:200)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:182)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62)
[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
[info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:233)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:431)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:233)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:232)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1563)
[info]   at org.scalatest.Suite.run(Suite.scala:1112)
[info]   at org.scalatest.Suite.run$(Suite.scala:1094)
[info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1563)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:237)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:237)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:236)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:62)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:62)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:748)
[info] - SPARK-33034: ALTER TABLE ... update column type (122 milliseconds)
[info] - SPARK-33034: ALTER TABLE ... rename column (93 milliseconds)
[info] - SPARK-33034: ALTER TABLE ... update column nullability (92 milliseconds)
[info] - CREATE TABLE with table comment (38 milliseconds)
[info] - CREATE TABLE with table property (52 milliseconds)
[info] MySQLIntegrationSuite:
[info] - Basic test (61 milliseconds)
[info] - Numeric types (67 milliseconds)
[info] - Date types (59 milliseconds)
[info] - String types (50 milliseconds)
[info] - Basic write test (216 milliseconds)
[info] - query JDBC option (64 milliseconds)
[info] Run completed in 19 minutes, 43 seconds.
[info] Total number of tests run: 89
[info] Suites: completed 14, aborted 0
[info] Tests: succeeded 84, failed 5, canceled 0, ignored 0, pending 0
[info] *** 5 TESTS FAILED ***
[error] Failed tests:
[error] 	org.apache.spark.sql.jdbc.v2.OracleIntegrationSuite
[error] 	org.apache.spark.sql.jdbc.v2.MsSqlServerIntegrationSuite
[error] 	org.apache.spark.sql.jdbc.v2.DB2IntegrationSuite
[error] 	org.apache.spark.sql.jdbc.v2.MySQLIntegrationSuite
[error] 	org.apache.spark.sql.jdbc.v2.PostgresIntegrationSuite
[error] (docker-integration-tests / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1223 s (20:23), completed Jun 25, 2021 1:31:04 AM

Mind taking a look when you find some time?

sarutak · 2021-06-25T02:28:40Z

@HyukjinKwon Hmm, it's strange. Please let me investigate.

sarutak · 2021-06-25T02:36:56Z

@HyukjinKwon I run v2.PostgresIntegrationSuite on the commit hash 14b1836 which is the parent of this commit but this issue still happens.

[info] PostgresIntegrationSuite:
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[info] No tests to run for avro / Test / testOnly
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[info] No tests to run for sql-kafka-0-10 / Test / testOnly
[info] - SPARK-33034: ALTER TABLE ... add new columns (3 seconds, 979 milliseconds)
[info] - SPARK-33034: ALTER TABLE ... drop column *** FAILED *** (314 milliseconds)
[info]   "Cannot delete missing field bad_column in postgresql.alt_table schema: root
[info]    |-- C2: string (nullable = true)
[info]   ; line 1 pos 0;
[info]   'AlterTableDropColumns [unresolvedfieldname(bad_column)]
[info]   +- ResolvedTable org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog@35f3548a, alt_table, JDBCTable(alt_table,StructType(StructField(C2,StringType,true)),org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions@698bf9c2), [C2#39]
[info]   " did not contain "Cannot delete missing field bad_column in alt_table schema" (V2JDBCTest.scala:106)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info]   at org.apache.spark.sql.jdbc.v2.V2JDBCTest.$anonfun$$init$$6(V2JDBCTest.scala:106)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1461)
[info]   at org.apache.spark.sql.test.SQLTestUtilsBase.withTable(SQLTestUtils.scala:305)
[info]   at org.apache.spark.sql.test.SQLTestUtilsBase.withTable$(SQLTestUtils.scala:303)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.withTable(DockerJDBCIntegrationSuite.scala:95)
[info]   at org.apache.spark.sql.jdbc.v2.V2JDBCTest.$anonfun$$init$$5(V2JDBCTest.scala:95)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:190)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:188)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:200)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:200)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:182)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62)
[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
[info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:233)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:431)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:233)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:232)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1563)
[info]   at org.scalatest.Suite.run(Suite.scala:1112)
[info]   at org.scalatest.Suite.run$(Suite.scala:1094)
[info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1563)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:237)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:237)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:236)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:62)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:62)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:748)

The root cause may be elsewhere.

sarutak · 2021-06-25T02:45:13Z

I found 5b4816c would be the root cause.
That change didn't trigger docker-integration-tests on GA so we couldn't notice the problem.
Let me open a follow-up PR.

sarutak added 2 commits June 24, 2021 15:30

Upgrade Jetty.

4c84508

Merge branch 'master' of https://github.com/apache/spark into upgrade…

d290b38

…-jetty-9.4.42

github-actions bot added the BUILD label Jun 24, 2021

dongjoon-hyun approved these changes Jun 24, 2021

View reviewed changes

sarutak closed this in 7b78d56 Jun 24, 2021

sunchao mentioned this pull request Jul 7, 2021

[SPARK-35959][BUILD][test-maven][test-hadoop3.2][test-java11] Add a new Maven profile "no-shaded-hadoop-client" for Hadoop versions older than 3.2.2/3.3.1 #33160

Closed

sunchao mentioned this pull request Sep 5, 2021

[SPARK-36669][BUILD] Revert to non-shaded Hadoop client library #33913

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[SPARK-35870][BUILD] Upgrade Jetty to 9.4.42 #33053

[SPARK-35870][BUILD] Upgrade Jetty to 9.4.42 #33053

Uh oh!

sarutak commented Jun 24, 2021

Uh oh!

dongjoon-hyun left a comment

Uh oh!

SparkQA commented Jun 24, 2021

Uh oh!

SparkQA commented Jun 24, 2021

Uh oh!

sarutak commented Jun 24, 2021

Uh oh!

HyukjinKwon commented Jun 25, 2021

Uh oh!

sarutak commented Jun 25, 2021

Uh oh!

sarutak commented Jun 25, 2021

Uh oh!

sarutak commented Jun 25, 2021 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

[SPARK-35870][BUILD] Upgrade Jetty to 9.4.42 #33053

[SPARK-35870][BUILD] Upgrade Jetty to 9.4.42 #33053

Uh oh!

Conversation

sarutak commented Jun 24, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 24, 2021

Uh oh!

SparkQA commented Jun 24, 2021

Uh oh!

sarutak commented Jun 24, 2021

Uh oh!

HyukjinKwon commented Jun 25, 2021

Uh oh!

sarutak commented Jun 25, 2021

Uh oh!

sarutak commented Jun 25, 2021

Uh oh!

sarutak commented Jun 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sarutak commented Jun 25, 2021 •

edited

Loading