Build: Bump pyspark from 3.4.2 to 3.5.0

[dependabot]

Bumps pyspark from 3.4.2 to 3.5.0.

Commits

• ce5ddad Preparing Spark release v3.5.0-rc5
• 05b57bc [SPARK-45075][SQL] Fix alter table with invalid default value will not report...
• b2b2ba9 [SPARK-44805][SQL] getBytes/getShorts/getInts/etc. should work in a column ve...
• 695be32 [SPARK-45106][SQL] PercentileCont should check user supplied input
• 0662175 [SPARK-45098][DOCS] Custom jekyll-rediect-from redirect.html template to fix ...
• 8f730e7 [SPARK-45100][SQL] Fix an internal error from reflect()on NULL class and ...
• 959d93a [SPARK-44508][PYTHON][DOCS] Add user guide for Python user-defined table func...
• 3ceec3b [SPARK-44835][CONNECT] Make INVALID_CURSOR.DISCONNECTED a retriable error
• a9d601c [SPARK-44640][PYTHON][FOLLOW-UP][3.5] Update UDTF error messages to include m...
• 916b6f5 [SPARK-45050][SQL][CONNECT] Improve error message for UNKNOWN io.grpc.StatusR...
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[HonahX]

Mark: we also need to update the iceberg-spark-runtime used here:

iceberg-python/tests/integration/test_writes.py

Lines 231 to 237 in 8f7927b

	@pytest.fixture(scope="session")
	def spark() -> SparkSession:
	import os
	os.environ["PYSPARK_SUBMIT_ARGS"] = (
	"--packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.4.0,org.apache.iceberg:iceberg-aws-bundle:1.4.0 pyspark-shell"
	)

[Fokko]

@HonahX Good one! Maybe we should pass that in through an environment variable?

[HonahX]

Hi @Fokko.

Maybe we should pass that in through an environment variable

Do you mean putting it in a .env file? Also, I am thinking if it is possible to link the iceberg-spark-runtime 's version with our pyspark version in pyproject.toml.

spark_version = ".".join(importlib.metadata.version("pyspark").split(".")[:2])

    os.environ["PYSPARK_SUBMIT_ARGS"] = (
        f"--packages org.apache.iceberg:iceberg-spark-runtime-{spark_version}_2.12:1.4.0,org.apache.iceberg:iceberg-aws-bundle:1.4.0 pyspark-shell"
    )

So that next time the dependabot can automatically update pyspark or fail if iceberg does not support the version yet. Does this sound good to you?

[Fokko]

@HonahX I was thinking of adding this to the Dockerfile through an ENV step. Ah wait, the tests run outside of the container. I like your suggestion a lot with using the importlib 👍

[HonahX]

@Fokko I opened a new PR to manually update the Pyspark version and use importlib to fetch version:
#303

[dependabot]

Looks like pyspark is up-to-date now, so this is no longer needed.