[SPARK-29567][TESTS] Update JDBC Integration Test Docker Images #26224

dongjoon-hyun · 2019-10-23T08:59:35Z

What changes were proposed in this pull request?

This PR updates JDBC Integration Test DBMS Docker Images.

DBMS	Docker Image Tag	Release
MySQL	mysql:5.7.28	Oct 13, 2019
PostgreSQL	postgres:12.0-alpine	Oct 3, 2019

For MySQL, SET GLOBAL sql_mode = '' is added to disable all strict modes because test("Basic write test") creates a table like the following. The latest MySQL rejects 0000-00-00 00:00:00 as TIMESTAMP and causes the test case failure.

mysql> desc datescopy;
+-------+-----------+------+-----+---------------------+-----------------------------+
| Field | Type      | Null | Key | Default             | Extra                       |
+-------+-----------+------+-----+---------------------+-----------------------------+
| d     | date      | YES  |     | NULL                |                             |
| t     | timestamp | NO   |     | CURRENT_TIMESTAMP   | on update CURRENT_TIMESTAMP |
| dt    | timestamp | NO   |     | 0000-00-00 00:00:00 |                             |
| ts    | timestamp | NO   |     | 0000-00-00 00:00:00 |                             |
| yr    | date      | YES  |     | NULL                |                             |
+-------+-----------+------+-----+---------------------+-----------------------------+

For PostgreSQL, I chose the smallest image in 12 releases. It reduces the image size a lot, 312MB -> 72.8MB. This is good for CI/CI testing environment.

$ docker images | grep postgres
postgres      12.0-alpine          5b681acb1cfc        2 days ago          72.8MB
postgres      11.4                 53912975086f        3 months ago        312MB

Note that

For MsSqlServer, we are using 2017-GA-ubuntu and the next version 2019-CTP3.2-ubuntu is still Community Technology Preview status.
For DB2 and Oracle, the official images are not available.

Why are the changes needed?

This is to make it sure we are testing with the latest DBMS images during preparing 3.0.0.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Since this is the integration test, we need to run this manually.

build/mvn install -DskipTests
build/mvn -Pdocker-integration-tests -pl :spark-docker-integration-tests_2.12 test

dongjoon-hyun · 2019-10-23T09:47:15Z

There is some suspicious part. I'll reopen this later.

SparkQA · 2019-10-23T11:55:17Z

Test build #112527 has finished for PR 26224 at commit 324c8d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-23T12:25:26Z

Test build #112528 has finished for PR 26224 at commit c31d669.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya

seems fine.

dongjoon-hyun · 2019-10-23T21:47:37Z

Thank you for review, @viirya !

SparkQA · 2019-10-23T23:00:33Z

Test build #112562 has finished for PR 26224 at commit c31d669.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2019-10-23T23:02:49Z

retest this please

maropu · 2019-10-23T23:28:29Z

...ocker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala

  }

  override def dataPreparation(conn: Connection): Unit = {
+    // Since MySQL 5.7.14+, we need to disable strict mode


just out of curiosity; why we need to disable the mode in this mysql?

As I mentioned in the PR description, MySQL baned 0000-00-00 00:00:00. There is a SQL mode for that. I enumerated and tested all versions. It starts to ban on 5.7.14.

oh, I missed that. thanks.

maropu · 2019-10-23T23:29:30Z

...er-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala

 class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite {
  override val db = new DatabaseOnDocker {
-    override val imageName = "postgres:11.4"
+    override val imageName = "postgres:12.0-alpine"


12.0-alpine instead of 12.0?

Yes. This is another improvement. It reduces the image size a lot, 312MB -> 72.8MB. This is good for CI/CI testing environment.

$ docker images | grep postgres postgres 12.0-alpine 5b681acb1cfc 2 days ago 72.8MB postgres 11.4 53912975086f 3 months ago 312MB

I'll add this to the PR description.

Thanks! Yea, it looks nice.

dongjoon-hyun · 2019-10-23T23:33:21Z

Thank you for review, @maropu !

maropu · 2019-10-23T23:36:16Z

Anyway, I checked this pr worked correctly on my local env.

dongjoon-hyun · 2019-10-23T23:37:26Z

Thank you again, @maropu !

dongjoon-hyun · 2019-10-23T23:43:45Z

Thank you, @viirya and @maropu .
Since this is the integration test suite and the integration test itself is verified manually, I'll merge this. (Also, the last commit passed the test twice last night) Merged to master!

SparkQA · 2019-10-24T01:48:20Z

Test build #112567 has finished for PR 26224 at commit c31d669.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

[SPARK-29567][TESTS] Update JDBC Integration Test Docker Images

c31d669

dongjoon-hyun closed this Oct 23, 2019

dongjoon-hyun reopened this Oct 23, 2019

viirya reviewed Oct 23, 2019

View reviewed changes

dongjoon-hyun added the TESTS label Oct 23, 2019

maropu reviewed Oct 23, 2019

View reviewed changes

maropu approved these changes Oct 23, 2019

View reviewed changes

dongjoon-hyun closed this in 7ecf968 Oct 23, 2019

dongjoon-hyun deleted the SPARK-29567 branch October 23, 2019 23:45

[SPARK-29567][TESTS] Update JDBC Integration Test Docker Images #26224

[SPARK-29567][TESTS] Update JDBC Integration Test Docker Images #26224

Uh oh!

Conversation

dongjoon-hyun commented Oct 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Oct 23, 2019

Uh oh!

SparkQA commented Oct 23, 2019

Uh oh!

SparkQA commented Oct 23, 2019

Uh oh!

viirya left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Oct 23, 2019

Uh oh!

SparkQA commented Oct 23, 2019

Uh oh!

maropu commented Oct 23, 2019

Uh oh!

maropu Oct 23, 2019

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maropu Oct 23, 2019

Choose a reason for hiding this comment

Uh oh!

maropu Oct 23, 2019

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 23, 2019

Choose a reason for hiding this comment

Uh oh!

maropu Oct 23, 2019

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Oct 23, 2019

Uh oh!

maropu commented Oct 23, 2019

Uh oh!

dongjoon-hyun commented Oct 23, 2019

Uh oh!

dongjoon-hyun commented Oct 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Oct 24, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dongjoon-hyun commented Oct 23, 2019 •

edited

Loading

dongjoon-hyun Oct 23, 2019 •

edited

Loading

dongjoon-hyun Oct 23, 2019 •

edited

Loading

dongjoon-hyun commented Oct 23, 2019 •

edited

Loading