[SPARK-53066][SQL] Improve EXPLAIN output for DSv2 Join pushdown #51781

PetarVasiljevic-DB · 2025-08-01T17:28:39Z

What changes were proposed in this pull request?

Prior to this change, EXPLAIN FORMATTED on query e.g.:

SELECT *
FROM catalog.tbl1 t1
    JOIN catalog.tbl2 t2 ON t1.id1 = t2.id2
    JOIN catalog.tbl3 t3 ON t2.id2 = t3.id3
    JOIN catalog.tbl4 t4 ON t3.id3 = t4.id4;

looked like:

PushedJoins: [join_pushdown_catalog.tbl1, join_pushdown_catalog.tbl2, join_pushdown_catalog.tlb3, join_pushdown_catalog.tlb4]

With the change from PR, the output of EXPLAIN FORMATTED would be:

 == Physical Plan ==
   *(1) Project [ID#x, AMOUNT#x, ADDRESS#x, ID_1#x AS ID#x, NEXT_ID#x, SALARY#x, SURNAME#x, id_3#x AS id#x, id_1_2#x AS id_1#x, id_2#x, id_1_1#x, sid#x, id_4#x AS id#x, id_1_3#x AS id_1#x, id_2_2#x AS id_2#x, id_2_1#x, Sid_1#x AS Sid#x]
   +- *(1) Scan JDBC v1 Relation from v2 scan join_pushdown_catalog.JOIN_SCHEMA.JOIN_TABLE_1[ID#x,AMOUNT#x,ADDRESS#x,ID_1#x,NEXT_ID#x,SALARY#x,SURNAME#x,id_3#x,id_1_2#x,id_2#x,id_1_1#x,sid#x,id_4#x,id_1_3#x,id_2_2#x,id_2_1#x,Sid_1#x] PushedFilters: [id_3 = (id_4 + 1)], PushedJoins: 
   [L]: PushedFilters: [ID_1 = (id_3 + 1)]
        PushedJoins:
        [L]: PushedFilters: [ID = (ID_1 + 1)]
             PushedJoins:
             [L]: Relation: join_pushdown_catalog.JOIN_SCHEMA.JOIN_TABLE_1
                  PushedFilters: [ID IS NOT NULL]
             [R]: Relation: join_pushdown_catalog.JOIN_SCHEMA.JOIN_TABLE_2
                  PushedFilters: [ID IS NOT NULL]
        [R]: Relation: join_pushdown_catalog.JOIN_SCHEMA.JOIN_TABLE_3
             PushedFilters: [id IS NOT NULL]
   [R]: Relation: join_pushdown_catalog.JOIN_SCHEMA.JOIN_TABLE_4
        PushedFilters: [id IS NOT NULL]
   , ReadSchema: struct<ID:int,AMOUNT:decimal(10,2),ADDRESS:string,ID_1:int,NEXT_ID:int,SALARY:decimal(10,2),SURNAME:string,id_3:int,id_1_2:int,id_2:int,id_1_1:int,sid:int,id_4:int,id_1_3:int,id_2_2:int,id_2_1:int,Sid_1:int>

PushedFilters on top of PushedJoins are actually join conditions.
It can be seen that the name of Scan JDBC v1 Relation from v2 scan is catalog.tbl1. This should be fixed as well, but it won't be a part of this PR.

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

PetarVasiljevic-DB · 2025-08-01T18:04:02Z

cc: @cloud-fan

cloud-fan · 2025-08-04T09:49:02Z

@PetarVasiljevic-DB can you fix the merge conflicts?

cloud-fan · 2025-08-04T16:03:14Z

sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala

it's always 0 and 1, shall we use LEFT and RIGHT? or L and R?

it is always 0 and 1 yes. I can change to L and R as the indentation will stay the same. With LEFT and RIGHT we would have some indentation issues.

PetarVasiljevic-DB · 2025-08-05T09:21:18Z

@cloud-fan can we merge?

cloud-fan · 2025-08-06T13:14:47Z

sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala

+   * The exmaple of resulting string is the following:
+   *
+   * PushedJoins:
+   * [0]: [PushedFilters: [ID_1 = (ID_2 + 1)],


let's update the comment

cloud-fan · 2025-08-06T13:15:22Z

sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala

+   * [1]: [Relation: join_pushdown_catalog.tbl4, PushedFilters: [ID IS NOT NULL]]
+   */
+  private def getPushedJoinString(
+      joinedPushedDownOperators: Seq[PushedDownOperators],


it's always two elements? shall we just have two parameters?

yes, we can have 2 parameters only.

cloud-fan · 2025-08-06T13:17:20Z

...ore/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala

  }
+
+  // TODO: test currently doesn't assert on anything. We can assert on pushedJoins from explain
+  // when https://github.com/apache/spark/pull/51686 is checked in, since we would have


it's merged already, can we update?

cloud-fan · 2025-08-06T13:18:28Z

...ore/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala

+      checkJoinPushed(
+        df,
+        s"""PushedFilters: [id_3 = (id_4 + 1)], PushedJoins: [
+          |[L]: [PushedFilters: [${caseConvert("id_1")} = (id_3 + 1)],


I think the string looks better if there is no wrapping [] around pushed filters.

so [L]: PushedFilters: [${caseConvert("id_1")} = (id_3 + 1)?
what about the bottom one with relation and pushed filters:

[L]: [Relation: $catalogAndNamespace.${caseConvert(joinTableName1)}, PushedFilters: [${caseConvert("id")} IS NOT NULL]],

Would you prefer omitting brackets here as well, or should we keep them?

how about we unify the leaf and non-leaf nodes a bit more

[L]: PushedFilters: ... PushedJoins: [L]: PushedFilters: ... Relation: ...

The topmost filter can't really by put in new line, because we are doing metadata.toSeq.sorted when generating the string from metadata Map, so the order would be different if I put Map("\nPushedJoins" -> ...) instead of Map("PushedJoins" -> ...).

I would still put [] in pushed filters because that is what we did before this PR for pushed filters. Also, semantically, this is an array of filters so it makes sense to wrap it into brackets.

Before committing anything, are you fine with the following structure? Because of the formatting, there is a weird formatting at the end (, ReadSchema) but it is due to how we generate the string from metadata Map

== Physical Plan == *(1) Project [ID#x, AMOUNT#x, ADDRESS#x, ID_1#x AS ID#x, NEXT_ID#x, SALARY#x, SURNAME#x, id_3#x AS id#x, id_1_2#x AS id_1#x, id_2#x, id_1_1#x, sid#x, id_4#x AS id#x, id_1_3#x AS id_1#x, id_2_2#x AS id_2#x, id_2_1#x, Sid_1#x AS Sid#x] +- *(1) Scan JDBC v1 Relation from v2 scan join_pushdown_catalog.JOIN_SCHEMA.JOIN_TABLE_1[ID#x,AMOUNT#x,ADDRESS#x,ID_1#x,NEXT_ID#x,SALARY#x,SURNAME#x,id_3#x,id_1_2#x,id_2#x,id_1_1#x,sid#x,id_4#x,id_1_3#x,id_2_2#x,id_2_1#x,Sid_1#x] PushedFilters: [id_3 = (id_4 + 1)], PushedJoins: [ [L]: PushedFilters: [ID_1 = (id_3 + 1)] PushedJoins: [L]: PushedFilters: [ID = (ID_1 + 1)] PushedJoins: [L]: Relation: join_pushdown_catalog.JOIN_SCHEMA.JOIN_TABLE_1 PushedFilters: [ID IS NOT NULL] [R]: Relation: join_pushdown_catalog.JOIN_SCHEMA.JOIN_TABLE_2 PushedFilters: [ID IS NOT NULL] [R]: Relation: join_pushdown_catalog.JOIN_SCHEMA.JOIN_TABLE_3 PushedFilters: [id IS NOT NULL] [R]: Relation: join_pushdown_catalog.JOIN_SCHEMA.JOIN_TABLE_4 PushedFilters: [id IS NOT NULL]] , ReadSchema: struct<ID:int,AMOUNT:decimal(10,2),ADDRESS:string,ID_1:int,NEXT_ID:int,SALARY:decimal(10,2),SURNAME:string,id_3:int,id_1_2:int,id_2:int,id_1_1:int,sid:int,id_4:int,id_1_3:int,id_2_2:int,id_2_1:int,Sid_1:int>

ah this looks better!

Cool! Pushed the commit

cloud-fan · 2025-08-06T13:19:21Z

...ore/src/test/scala/org/apache/spark/sql/jdbc/v2/JDBCV2JoinPushdownIntegrationSuiteBase.scala


    withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") {
      val df = sql(sqlQuery)
-      checkJoinPushed(


why do we remove all these checks?

I find these useless now. There is a separate test for testing the output of explain command. I can get them back if it's a blocker.

cloud-fan · 2025-08-07T10:01:00Z

The failed test passed locally, thanks, merging to master!

github-actions bot added the SQL label Aug 1, 2025

PetarVasiljevic-DB force-pushed the improve_explain_command_for_dsv2_join_pushdown branch 4 times, most recently from 3563693 to 897ce34 Compare August 1, 2025 18:03

PetarVasiljevic-DB added 2 commits August 4, 2025 11:53

improve explain for join pushdown

99fb56a

change the explain format

05ae972

PetarVasiljevic-DB force-pushed the improve_explain_command_for_dsv2_join_pushdown branch 2 times, most recently from 9aa996a to c3d304d Compare August 4, 2025 14:43

cloud-fan reviewed Aug 4, 2025

View reviewed changes

cloud-fan approved these changes Aug 4, 2025

View reviewed changes

adapt on simplified join column names

9a05c5f

PetarVasiljevic-DB force-pushed the improve_explain_command_for_dsv2_join_pushdown branch from c3d304d to 9a05c5f Compare August 4, 2025 16:27

PetarVasiljevic-DB added 2 commits August 4, 2025 18:51

change to L and R

db1fdf6

fix oracle

d7c33e7

cloud-fan reviewed Aug 6, 2025

View reviewed changes

resolve comments

9723be9

PetarVasiljevic-DB force-pushed the improve_explain_command_for_dsv2_join_pushdown branch 2 times, most recently from 411e8c2 to bd25ef4 Compare August 6, 2025 17:53

pretify formatted string

0531a70

PetarVasiljevic-DB force-pushed the improve_explain_command_for_dsv2_join_pushdown branch from bd25ef4 to 0531a70 Compare August 6, 2025 23:12

cloud-fan approved these changes Aug 7, 2025

View reviewed changes

cloud-fan closed this in 88dbe42 Aug 7, 2025

[SPARK-53066][SQL] Improve EXPLAIN output for DSv2 Join pushdown #51781

[SPARK-53066][SQL] Improve EXPLAIN output for DSv2 Join pushdown #51781

Uh oh!

Conversation

PetarVasiljevic-DB commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

PetarVasiljevic-DB commented Aug 1, 2025

Uh oh!

cloud-fan commented Aug 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PetarVasiljevic-DB commented Aug 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Aug 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PetarVasiljevic-DB commented Aug 1, 2025 •

edited

Loading