[SPARK-33858][SQL][TESTS] Unify v1 and v2 ALTER TABLE .. RENAME PARTITION tests #30863

MaxGekk · 2020-12-20T10:48:04Z

What changes were proposed in this pull request?

Move the ALTER TABLE .. RENAME PARTITION parsing tests to AlterTableRenamePartitionParserSuite
Place the v1 tests for ALTER TABLE .. RENAME PARTITION from DDLSuite to v1.AlterTableRenamePartitionSuite and v2 tests from AlterTablePartitionV2SQLSuite to v2.AlterTableRenamePartitionSuite, so, the tests will run for V1, Hive V1 and V2 DS.

Why are the changes needed?

The unification will allow to run common ALTER TABLE .. RENAME PARTITION tests for both DSv1 and Hive DSv1, DSv2
We can detect missing features and differences between DSv1 and DSv2 implementations.

Does this PR introduce any user-facing change?

No

How was this patch tested?

By running new test suites:

$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *AlterTableRenamePartitionParserSuite"
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *AlterTableRenamePartitionSuite"

SparkQA · 2020-12-20T12:08:43Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37701/

SparkQA · 2020-12-20T12:36:58Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37701/

SparkQA · 2020-12-20T15:25:25Z

Test build #133101 has finished for PR 30863 at commit e8d2307.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

...rc/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableRenamePartitionSuite.scala

MaxGekk · 2020-12-20T15:39:47Z

@cloud-fan @HyukjinKwon Please, review this PR.

SparkQA · 2020-12-20T16:41:00Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37707/

SparkQA · 2020-12-20T17:13:45Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37707/

SparkQA · 2020-12-20T18:07:38Z

Test build #133107 has finished for PR 30863 at commit aa602e0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-20T20:05:43Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37713/

SparkQA · 2020-12-20T20:34:35Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37713/

SparkQA · 2020-12-21T00:05:33Z

Test build #133114 has finished for PR 30863 at commit 6db2b3e.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-12-21T08:20:36Z

@MaxGekk can you fix conflicts?

…ion-tests # Conflicts: # sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala # sql/core/src/test/scala/org/apache/spark/sql/connector/AlterTablePartitionV2SQLSuite.scala # sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala # sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

MaxGekk · 2020-12-21T10:35:38Z

.../test/scala/org/apache/spark/sql/hive/execution/command/AlterTableRenamePartitionSuite.scala

+      createSinglePartTable(t)
+      sql(s"INSERT INTO $t PARTITION (id = 2) SELECT 'def'")
+      checkPartitions(t, Map("id" -> "1"), Map("id" -> "2"))
+      // TODO(SPARK-33862): Throw PartitionAlreadyExistsException


This PR #30866 solves the issue.

@dongjoon-hyun merged #30866, I am going to rebase this PR on the master and try to move this test to the common v1 trait.

SparkQA · 2020-12-21T11:11:26Z

Test build #133146 has finished for PR 30863 at commit 2b47331.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-12-21T11:14:56Z

jenkins, retest this, please

SparkQA · 2020-12-21T11:18:55Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37745/

SparkQA · 2020-12-21T11:54:52Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37745/

…ion-tests # Conflicts: # sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala

SparkQA · 2020-12-21T12:20:29Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37747/

SparkQA · 2020-12-21T12:51:32Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37747/

SparkQA · 2020-12-21T13:29:34Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37751/

SparkQA · 2020-12-21T14:02:04Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37751/

SparkQA · 2020-12-21T16:00:37Z

Test build #133148 has finished for PR 30863 at commit 2b47331.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-21T16:58:31Z

Test build #133152 has finished for PR 30863 at commit b11c07b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…ion-tests

SparkQA · 2020-12-22T15:43:39Z

Test build #133220 has finished for PR 30863 at commit 1284c37.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-22T16:43:31Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37829/

SparkQA · 2020-12-22T17:14:48Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37829/

SparkQA · 2020-12-22T21:10:15Z

Test build #133232 has finished for PR 30863 at commit 0eb89ae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-12-23T01:00:29Z

...rc/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableRenamePartitionSuite.scala

+import org.apache.spark.sql.execution.command
+import org.apache.spark.sql.internal.SQLConf
+
+trait AlterTableRenamePartitionSuiteBase extends command.AlterTableRenamePartitionSuiteBase {


It would be helpful if you add a comment before this PR about the new hierarchy of the test suites, @MaxGekk .

this is a good idea. Shall we do it for all the new test suites?

Let me do that separately for all unified test suites, and to don't block this PR.

Let me do that separately for all unified test suites, and to don't block this PR.

Since this is merged, please proceed the comment PR, @MaxGekk .

Here is the PR #30929 , please, review it.

cloud-fan · 2020-12-23T12:19:04Z

thanks, merging to master!

MaxGekk added 8 commits December 20, 2020 12:08

Add AlterTableRenamePartitionParserSuite

6edc035

Add v2.AlterTableRenamePartitionSuite

08ee277

Add TODO for v2

4786c65

Port tests from DDLSuite

9254164

checkAnswer

b15e830

test with location

603d2fd

test for renaming of a multi part partition

edfd77f

Add Hive tests

e8d2307

github-actions bot added the SQL label Dec 20, 2020

Remove testRenamePartitions()

a1ba687

Move v1 tests

aa602e0

MaxGekk commented Dec 20, 2020

View reviewed changes

...rc/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableRenamePartitionSuite.scala Show resolved Hide resolved

MaxGekk mentioned this pull request Dec 20, 2020

[SPARK-33856][SQL] Migrate ALTER TABLE ... RENAME TO PARTITION to use UnresolvedTable to resolve the identifier #30862

Closed

Test that target partition exists

6db2b3e

MaxGekk added 3 commits December 21, 2020 12:52

Fix merge issues in AlterTableRenamePartitionParserSuite

27c353d

Fix AlterTableRenamePartitionSuite

2b47331

MaxGekk commented Dec 21, 2020

View reviewed changes

MaxGekk mentioned this pull request Dec 21, 2020

[SPARK-33862][SQL] Throw PartitionAlreadyExistsException if the target partition exists while renaming #30866

Closed

Merge remote-tracking branch 'origin/master' into unify-rename-partit…

8ebb812

…ion-tests # Conflicts: # sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala

Move the test "target partition exists" to the common v1 test suite

b11c07b

MaxGekk added 2 commits December 22, 2020 15:53

Merge remote-tracking branch 'origin/master' into unify-rename-partit…

17d42ff

…ion-tests

Move v1 test to the common trait

1284c37

cloud-fan approved these changes Dec 22, 2020

View reviewed changes

Trigger build

0eb89ae

dongjoon-hyun reviewed Dec 23, 2020

View reviewed changes

cloud-fan closed this in cc23581 Dec 23, 2020

MaxGekk deleted the unify-rename-partition-tests branch February 19, 2021 15:04

[SPARK-33858][SQL][TESTS] Unify v1 and v2 ALTER TABLE .. RENAME PARTITION tests #30863

[SPARK-33858][SQL][TESTS] Unify v1 and v2 ALTER TABLE .. RENAME PARTITION tests #30863

Uh oh!

Conversation

MaxGekk commented Dec 20, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Dec 20, 2020

Uh oh!

SparkQA commented Dec 20, 2020

Uh oh!

SparkQA commented Dec 20, 2020

Uh oh!

Uh oh!

MaxGekk commented Dec 20, 2020

Uh oh!

SparkQA commented Dec 20, 2020

Uh oh!

SparkQA commented Dec 20, 2020

Uh oh!

SparkQA commented Dec 20, 2020

Uh oh!

SparkQA commented Dec 20, 2020

Uh oh!

SparkQA commented Dec 20, 2020

Uh oh!

SparkQA commented Dec 21, 2020

Uh oh!

cloud-fan commented Dec 21, 2020

Uh oh!

MaxGekk Dec 21, 2020

Choose a reason for hiding this comment

Uh oh!

MaxGekk Dec 21, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 21, 2020

Uh oh!

MaxGekk commented Dec 21, 2020

Uh oh!

SparkQA commented Dec 21, 2020

Uh oh!

SparkQA commented Dec 21, 2020

Uh oh!

SparkQA commented Dec 21, 2020

Uh oh!

SparkQA commented Dec 21, 2020

Uh oh!

SparkQA commented Dec 21, 2020

Uh oh!

SparkQA commented Dec 21, 2020

Uh oh!

SparkQA commented Dec 21, 2020

Uh oh!

SparkQA commented Dec 21, 2020

Uh oh!

SparkQA commented Dec 22, 2020

Uh oh!

SparkQA commented Dec 22, 2020

Uh oh!

SparkQA commented Dec 22, 2020

Uh oh!

SparkQA commented Dec 22, 2020

Uh oh!

dongjoon-hyun Dec 23, 2020

Choose a reason for hiding this comment

Uh oh!

cloud-fan Dec 23, 2020

Choose a reason for hiding this comment

Uh oh!

MaxGekk Dec 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Dec 23, 2020

Choose a reason for hiding this comment

Uh oh!

MaxGekk Dec 23, 2020 •

edited

Loading