Figure out why pydata-sphinx-theme is parallel-write-unsafe, and fix it

[drammock]

In working on scipy/scipy#16660 I discovered that our theme is parallel-write unsafe (see esp. this comment, and the PR description of #1642 where I mark the theme as parallel-write unsafe).

This is a reminder to try to figure out why, and hopefully fix it so that we can remove the parallel-write-unsafe flag.

[RobPasMue]

This is a rather unpleasant issue on our side because we are forcing warnings to raise as errors (in order to properly maintain our docs). Having to switch to single-threaded mode is also inconvenient. It'd be great if we could get this fix soon in the next release 😄! Upvoting the issue!!

[stinodego]

I ran into this when trying to upgrade from version 0.14.1 to version 0.15.2. We use Sphinx version 7.2.4.

Eagerly awaiting a fix so that we may upgrade to the latest version of the theme!

[drammock]

@RobPasMue and @stinodego can you tell us how big of an impact this is having for you? I.e., for the project(s) you're working on, how long is the doc build with -j1 and how long with -jN (for whatever N is typical in your workflow)? I'm trying to gauge how to prioritize this; is it "single-threaded builds is a crippling non-starter" or more like "single-threaded builds is inconvenient"?

For context, the changes that precipitated the parallel-write-unsafe marking were discovered when building the SciPy docs (which are fairly large) and even there the single-threaded build is only around ~12 minutes (which in the context of local dev/docs work is bad, but for CI servers on pull requests not so bad). Without the change, SciPy was seeing build times of ~30 minutes (even with -j2). It would be helpful to have similar timing numbers for other projects.

[stinodego]

I just ran a quick local benchmark with -j auto vs no parallelization. With parallelization it's 2:37 to build the docs, without it's 11:14.

So that's quite significant. In practice, that goes from "let me quickly build the docs to see the effect of this change" to "let's just hope it renders correctly".

At this point, we will likely wait for a new version that supports parallization, rather than upgrading to a version without parallelization that may have some nice new features. I think a lot of other projects will have parallel building enabled as well (free performance) and will run into this limitation. So I would say it's quite a critical fix.

For reference, I work on Polars, and we've been happy users of the theme for a long time now!
https://github.com/pola-rs/polars

[RobPasMue]

Hi @drammock - In my case I work with the PyAnsys ecosystem, where we are consumers of the pydata sphinx theme for over 3 years. I gave it a try with out library PyAnsys Geometry https://github.com/ansys/pyansys-geometry (without building the examples with sphinx gallery) and the results were as follows.

Using:

Sphinx==7.2.6
pydata-sphinx-theme==0.14.4

With parallelization (-j auto): 1:01
No parallelization (-j1): 2:00

Our build times are not that big in any case, but the impact of parallelization is significant. However, the main issue we are facing is that we enable parallelization by default throughout the ecosystem. And we are forcing warnings to behave as errors so that we ensure a proper build process. We have a wrapped version of the pydata sphinx theme (i.e. https://github.com/ansys/ansys-sphinx-theme) which we have tailored for our brand. We had to limit the version to the latest one with parallelization but as @stinodego mentioned, we would love to still use the latest and greatest features you guys are working on.