[SPARK-42371][CONNECT] Add scripts to start and stop Spark Connect server

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to scripts to start and stop the Spark Connect server.

Why are the changes needed?

Currently, there is no proper way to start and stop the Spark Connect server. Now it requires you to start it with, for example, a Spark shell:

# For development,
./bin/spark-shell \
   --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar` \
  --conf spark.plugins=org.apache.spark.sql.connect.SparkConnectPlugin

# For released Spark versions
./bin/spark-shell \
  --packages org.apache.spark:spark-connect_2.12:3.4.0 \
  --conf spark.plugins=org.apache.spark.sql.connect.SparkConnectPlugin

which is awkward.

Does this PR introduce any user-facing change?

Yes, it adds new scripts to start and stop Spark Connect server.

How was this patch tested?

Manually tested:

# For released Spark versions,
#`sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.4.0`
sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`

bin/pyspark --remote sc://localhost:15002
...

sbin/stop-connect-server.sh
ps -fe | grep Spark

[dongjoon-hyun]

+1, LGTM

[pan3793]

Maybe sbin/start-connect-server.sh, sbin/stop-connect-server.sh?

Currently all scripts under sbin use - except to start-thriftserver.sh, stop-thriftserver.sh

[HyukjinKwon]

sure sgtm

[HyukjinKwon]

Merged to master and branch-3.4.

Actually CI doesn't verify none of the changes here (except the linter job that passed now).

[tgravescs]

it feels a bit odd to me to have a script in the sbin that user has to manually specify the location of the connect jars. What is the plan for distribution, are connect jars going to be in the main distribution jars/ directory so it won't be needed at that point?
I don't think we do that with any of the other connectors but wasn't sure the plan and if it was mostly for initial development.
@HyukjinKwon

[HyukjinKwon]

it feels a bit odd to me to have a script in the sbin that user has to manually specify the location of the connect jars.

What is the plan for distribution, are connect jars going to be in the main distribution jars/ directory so it won't be needed at that point?

Yes, the eventual plan is to move the whole connect jars to the main distribution jars/ around the next release (Apache Spark 3.5.0). For now, the Spark connect project is separated into the external project. It is located in connector/ - that was external/ before (see also #35874, I think actually we might have to revisit the top level name here).

For a bit of more context, there are a couple of plans such as replacing Py4J to Spark Connect (so we can block arbitrary JVM access from Python side for security purpose), and I personally am thinking about replacing Thrift server in the far future (and don't use Hive's Thrift server). There are also some more context in this blog post that I like: https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html

I plan to send another email to explain the whole context in the dev mailing list around .. right after the Spark 3.4 release.