Skip to content

Conversation

@HyukjinKwon
Copy link
Member

What changes were proposed in this pull request?

This PR is a followup of #47157 that moves connect into sql/connect.

Why are the changes needed?

The reasons are as follow:

  • There was a bit of question about moving connect as a standalone top level ([SPARK-48763][CONNECT][BUILD] Move connect server and common to builtin module #47157 (comment)).
  • Technically all Spark Connect related code have to placed under sql just like Hive thrift server.
    • Spark Connect server is 99% SQL dedicated code for now
    • Spark Connect server already is using a lot of spark.sql configurations, e.g., spark.sql.connect.serverStacktrace.enabled
    • Spark Connect common is only for SQL module. If other components have to be implemented, that common has to be placed within that directory.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

CI in this PR should verify it.

Was this patch authored or co-authored using generative AI tooling?

No.

@HyukjinKwon
Copy link
Member Author

cc @hvanhovell

Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although this is a massive change, +1 for the direction for Apache Spark 4.0.0 . Thank you, @HyukjinKwon .

@HyukjinKwon
Copy link
Member Author

Thank you!!!

@HyukjinKwon
Copy link
Member Author

Merged to master.

HyukjinKwon added a commit that referenced this pull request Aug 5, 2024
…tionTestSuite

### What changes were proposed in this pull request?

This PR is a followup of #47579 that updates the Spark Connect location in the test `PlanGenerationTestSuite`.

### Why are the changes needed?

Just for completeness.

### Does this PR introduce _any_ user-facing change?

No, test-only.

### How was this patch tested?

Manually ran it via:

```bash
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "connect-client-jvm/testOnly org.apache.spark.sql.PlanGenerationTestSuite"
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #47605 from HyukjinKwon/SPARK-48763-followup2.

Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants