[WIP][SPARK-46051][INFRA] Cache python deps for linter and documentation

[zhengruifeng]

What changes were proposed in this pull request?

Cache python deps for linter and documentation

Why are the changes needed?

1, to avoid unnecessary installation: some packages were installed multiple times;
2, to centralize the installations: should only modify dockerfile in the future

Does this PR introduce any user-facing change?

no, infra-only

How was this patch tested?

ci

Was this patch authored or co-authored using generative AI tooling?

no

[zhengruifeng]

this step is also used in 3.3/3.4/3.5, so move it to Install dependencies for documentation generation for branch-3.3, branch-3.4, branch-3.5

[dongjoon-hyun]

+1 for this another attempt. Thank you for keeping improving this, @zhengruifeng .

cc @LuciferYang , too, because he found the last outage on the release branches.

[LuciferYang]

+1, LGTM

[zhengruifeng]

please hold on, it seems there are env conflicts between pyspark and lint
I convert it to draft, and will look into it

[zhengruifeng]

unfortunately, there is conflict in jinja2 and cause multiple pyspark tests fail.
I plan to test miniconda later.

[nchammas]

Why are we listing out individual dependencies here vs. listing them in a requirements file (or equivalent) that we can use across Docker, GitHub Actions, and local build scripts? Don't we want our build and test dependencies to be consistent?

I made a past attempt at this over in #27928. It failed because, in addition to building a shared set of build and test dependencies, it also pinned transitive build and test dependencies, which the reviewers weren't keen on. But we can separate the two ideas from each other.

IMO there should be a single requirements file for build and test dependencies (whether or not it pins transitive dependencies is a separate issue), and that file should be used everywhere. What do you think?

I also don't follow why we need to pull in conda. What is it getting us over vanilla pip?

[zhengruifeng]

hi @nchammas actually, I wanna give up this PR to cache the dependencies for linter and documentation, since there is still weird issue in building documentation with conda in CI (while it works well in my local).

The reason in this PR to try conda was that there were env conflicts between PySpark and lint/doc, that is, installation of lint/doc dependencies will break some tests.

As to why not use requirement file in CI, I guess a problem maybe, the modification in requirement file won't automatically refresh the cached testing image?

[nchammas]

As to why not use requirement file in CI, I guess a problem maybe, the modification in requirement file won't automatically refresh the cached testing image?

That shouldn't be the case. Assuming you COPY the requirements file into the image, changing the file will invalidate the cache:

The first encountered COPY instruction will invalidate the cache for all following instructions from the Dockerfile if the contents of <src> have changed. This includes invalidating the cache for RUN instructions.

Also:

For the ADD and COPY instructions, the modification time and size file metadata is used to determine whether cache is valid. During cache lookup, cache is invalidated if the file metadata has changed for any of the files involved.

[zhengruifeng]

good to know this. You are right.