[SPARK-3734] DriverRunner should not read SPARK_HOME from submitter's environment

[JoshRosen]

When using spark-submit in cluster mode to submit a job to a Spark Standalone
cluster, if the JAVA_HOME environment variable was set on the submitting
machine then DriverRunner would attempt to use the submitter's JAVA_HOME to
launch the driver process (instead of the worker's JAVA_HOME), causing the
driver to fail unless the submitter and worker had the same Java location.

This commit fixes this by reading JAVA_HOME from sys.env instead of
command.environment.

[SparkQA]

QA tests have started for PR 2586 at commit e9513d9.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2586 at commit e9513d9.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

LGTM. Merging into master and 1.1

[thanhdanh1803]

When using spark 3.3.0, I see this issue again, when run spark-submit in cluster mode, the worker show this message:
INFO DriverRunner: Launch Command: "/Library/Java/JavaVirtualMachines/temurin-8.jdk/Contents/Home/bin/java" "-cp" "/opt/spark3.3.0/conf:/opt/spark/jars/*" "-Xmx1024M" "-Dspark.executor.memory=2g" "-Dspark.driver.supervise=false" "-Dspark.app.submitTime=1706455734988" "-Dspark.submit.deployMode=cluster" "-Dspark.master=spark://127.0.0.1:7077" "-Dspark.app.name=spark_example" "-Dspark.submit.pyFiles=" "-Dspark.executor.cores=1" "-Dspark.driver.memory=1g" "-Dspark.rpc.askTimeout=10s" "-Dspark.jars=" "org.apache.spark.deploy.worker.DriverWrapper" "spark://[email protected]:7000" "/opt/spark/work/driver-20240128152856-0001/spark_example.jar" "example.SparkExample" "-n" "1000"
Can I open this issue again