Skip to content

[SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuite #33510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 12 commits into from
Closed

[SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuite #33510

wants to merge 12 commits into from

Conversation

@jerqi
Copy link
Contributor

@jerqi jerqi commented Jul 24, 2021 •
edited
Loading

What changes were proposed in this pull request?

In current github actions we run TPCDSQueryTestSuite for tpcds benchmark. But it's only tested under default configurations. Since we have added the spark.sql.join.forceApplyShuffledHashJoin config. Now we can test all 3 join strategies in TPCDS to improve the coverage.

Why are the changes needed?

Improve the coverage of join strategies in the TPCDS.

Does this PR introduce any user-facing change?

No, only for testing.

How was this patch tested?

No need.

@github-actions github-actions bot added the SQL label Jul 24, 2021
@jerqi jerqi changed the title [SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuite [WIP][SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuite Jul 25, 2021
@jerqi jerqi force-pushed the SPARK-36223 branch 7 times, most recently from a603ffc to 46bbf1f Compare July 25, 2021 16:11
@jerqi jerqi changed the title [WIP][SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuite [SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuite Jul 25, 2021
@jerqi jerqi force-pushed the SPARK-36223 branch 4 times, most recently from 42bb832 to 763c492 Compare August 6, 2021 13:58
cloud-fan
cloud-fan reviewed Nov 10, 2021
View reviewed changes
@cloud-fan
Copy link
Contributor

cloud-fan commented Nov 10, 2021

ok to test

@SparkQA
Copy link

SparkQA commented Nov 10, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49535/

@SparkQA
Copy link

SparkQA commented Nov 10, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49535/

@SparkQA
Copy link

SparkQA commented Nov 10, 2021

Test build #145066 has finished for PR 33510 at commit 8e2f4fb.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 11, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49589/

@SparkQA
Copy link

SparkQA commented Nov 11, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49590/

@SparkQA
Copy link

SparkQA commented Nov 11, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49589/

@SparkQA
Copy link

SparkQA commented Nov 11, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49590/

@SparkQA
Copy link

SparkQA commented Nov 12, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49640/

@SparkQA
Copy link

SparkQA commented Nov 12, 2021

Test build #145167 has finished for PR 33510 at commit cb0d3bc.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 12, 2021

Test build #145170 has finished for PR 33510 at commit cf38cab.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 15, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49692/

@SparkQA
Copy link

SparkQA commented Nov 15, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49692/

@SparkQA
Copy link

SparkQA commented Nov 15, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49693/

@cloud-fan
Copy link
Contributor

cloud-fan commented Nov 15, 2021

thanks, merging to master!

@cloud-fan cloud-fan closed this in 7070eb5 Nov 15, 2021
@SparkQA
Copy link

SparkQA commented Nov 15, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49693/

@jerqi
Copy link
Contributor Author

jerqi commented Nov 15, 2021

Thank you @cloud-fan @linhongliu-db and @HyukjinKwon for review.

@jerqi jerqi deleted the SPARK-36223 branch November 15, 2021 08:19
@SparkQA
Copy link

SparkQA commented Nov 15, 2021

Test build #145222 has finished for PR 33510 at commit c4ff4ab.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 15, 2021

Test build #145223 has finished for PR 33510 at commit 510fa61.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon HyukjinKwon mentioned this pull request Nov 24, 2021
Closed
HyukjinKwon
HyukjinKwon reviewed Nov 24, 2021
View reviewed changes
sql/core/src/test/scala/org/apache/spark/sql/TPCDSQueryTestSuite.scala
assertResult(expectedSchema, s"Schema did not match\n$queryString") { schema }
assertResult(expectedOutput, s"Result did not match\n$queryString") { outputString }
}
val joinConfSet: Set[Map[String, String]] =
Copy link
Member

@HyukjinKwon HyukjinKwon Nov 24, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, why is this set? Then joinConfSet.head won't be deterministic below, and there would be no point of needSort.

Copy link
Member

@HyukjinKwon HyukjinKwon Nov 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me fix it together at #34698

Copy link
Contributor Author

@jerqi jerqi Nov 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, seq may be better.

HyukjinKwon added a commit that referenced this pull request Nov 25, 2021
@HyukjinKwon
[SPARK-37453][INFRA][SQL][TESTS] Split TPC-DS build in GitHub Actions
27eb687 
### What changes were proposed in this pull request?

This is kind of a followup for #33510 and #34641. This PR proposes to split TPC-DS build in GitHub Actions.

### Why are the changes needed?

Running these queries easily causes out-of-memory in GitHub Actions machines, and make the build flaky. We should deflake it.

### Does this PR introduce _any_ user-facing change?

No, dev-only.

### How was this patch tested?

GitHub Actions in this PR should test it out.

Closes #34698 from HyukjinKwon/split-tpcds.

Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon left review comments

@cloud-fan cloud-fan cloud-fan approved these changes

+1 more reviewer

@linhongliu-db linhongliu-db linhongliu-db left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jerqi @cloud-fan @SparkQA @linhongliu-db @HyukjinKwon