sagemathgh-35373: Fix workflow "Build documentation (PDF)"
    
<!-- Please provide a concise, informative and self-explanatory title.
-->
<!-- Don't put issue numbers in the title. Put it in the Description
below. -->
<!-- For example, instead of "Fixes sagemath#12345", use "Add a new method to
multiply two integers" -->

### 📚 Description

Follow-up on sagemath#35169.

The container `sage-docker-fedora-31-maximal-with-targets` used for the
PDF docbuild turned out to be not reliable.
Here we replace it by `ubuntu-focal-standard-with-targets` and install
texlive in it.

We also copy over the incremental build from doc-build.yml and the
method to get CI fixes from blocker tickets from sagemath#36338.

This workflow is currently disabled in sagemath/sage.
Example run:
https://github.com/mkoeppe/sage/actions/runs/6318741659/job/17158468016

<!-- Describe your changes here in detail. -->
<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. It should be `[x]` not `[x
]`. -->

- [x] The title is concise, informative, and self-explanatory.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
- Depends on sagemath#36338
- Depends on sagemath#36348

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#35373
Reported by: Matthias Köppe
Reviewer(s): Dima Pasechnik

Author: Release Manager

.github/workflows/build.yml

@@ -41,7 +41,9 @@ jobs:
       - name: Store CI fixes in upstream artifact
         run: |
           mkdir -p upstream
-          git format-patch --stdout test_base > upstream/ci_fixes.patch
+          if git format-patch --stdout test_base > ci_fixes.patch; then
+            cp ci_fixes.patch upstream/
+          fi
       - uses: actions/upload-artifact@v3
         with:
           path: upstream

.github/workflows/doc-build-pdf.yml

@@ -5,43 +5,136 @@ on:
   push:
   workflow_dispatch:
     # Allow to run manually
+    inputs:
+      platform:
+        description: 'Platform'
+        required: true
+        default: 'ubuntu-focal-standard'
+      docker_tag:
+        description: 'Docker tag'
+        required: true
+        default: 'dev'
 
 concurrency:
   # Cancel previous runs of this workflow for the same branch
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
 
 jobs:
-  build-docs:
+  get_ci_fixes:
     runs-on: ubuntu-latest
-    # Use "maximal" so that texlive is installed
-    # Use "fedora-31" for build diversity
-    container: ghcr.io/sagemath/sage/sage-docker-fedora-31-maximal-with-targets:dev
     steps:
       - name: Checkout
+        id: checkout
         uses: actions/checkout@v4
-
       - name: Merge CI fixes from sagemath/sage
         run: |
           .ci/merge-fixes.sh
         env:
           GH_TOKEN: ${{ github.token }}
+      - name: Store CI fixes in upstream artifact
+        run: |
+          mkdir -p upstream
+          if git format-patch --stdout test_base > ci_fixes.patch; then
+            cp ci_fixes.patch upstream/
+          fi
+      - uses: actions/upload-artifact@v3
+        with:
+          path: upstream
+          name: upstream
 
-      - name: Prepare
+  build-docs-pdf:
+    runs-on: ubuntu-latest
+    container: ghcr.io/sagemath/sage/sage-${{ github.event.inputs.platform || 'ubuntu-focal-standard' }}-with-targets:${{ github.event.inputs.docker_tag || 'dev'}}
+    needs: [get_ci_fixes]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Update system packages
+        run: |
+          export PATH="build/bin:$PATH"
+          eval $(sage-print-system-package-command auto update)
+          eval $(sage-print-system-package-command auto --yes --no-install-recommends install zip)
+          eval $(sage-print-system-package-command auto --spkg --yes --no-install-recommends install git texlive)
+
+
+      - name: Add prebuilt tree as a worktree
+        id: worktree
         run: |
-          apt-get update && apt-get install -y zip
-          # Reuse built SAGE_LOCAL contained in the Docker image
-          ./bootstrap
-          ./configure --enable-build-as-root --prefix=/sage/local --with-sage-venv --enable-download-from-upstream-url
+          set -ex
+          git config --global user.email "[email protected]"
+          git config --global user.name "Build & Test workflow"
+          git config --global --add safe.directory $(pwd)
+          # If actions/checkout downloaded our source tree using the GitHub REST API
+          # instead of with git (because do not have git installed in our image),
+          # we first make the source tree a repo.
+          if [ ! -d .git ]; then git init && git add -A && git commit --quiet -m "new"; fi
+          # Tag this state of the source tree "new". This is what we want to build and test.
+          git tag -f new
+          # Our container image contains a source tree in /sage with a full build of Sage.
+          # But /sage is not a git repository.
+          # We make /sage a worktree whose index is at tag "new".
+          # We then commit the current sources and set the tag "old". (This keeps all mtimes unchanged.)
+          # Then we update worktree and index with "git reset --hard new".
+          # (This keeps mtimes of unchanged files unchanged and mtimes of changed files newer than unchanged files.)
+          # Finally we reset the index to "old". (This keeps all mtimes unchanged.)
+          # The changed files now show up as uncommitted changes.
+          # The final "git add -N" makes sure that files that were added in "new" do not show
+          # as untracked files, which would be removed by "git clean -fx".
+          git worktree add --detach worktree-image
+          rm -rf /sage/.git && mv worktree-image/.git /sage/
+          rm -rf worktree-image && ln -s /sage worktree-image
+          if [ ! -f worktree-image/.gitignore ]; then cp .gitignore worktree-image/; fi
+          (cd worktree-image && git add -A && git commit --quiet --allow-empty -m "old" -a && git tag -f old && git reset --hard new && git reset --quiet old && git add -N . && git status)
+          # Keep track of changes to built HTML
+          new_version=$(cat src/VERSION.txt); (cd /sage/local/share/doc/sage/html/en && find . -name "*.html" | xargs sed -i '/class="sidebar-brand-text"/s/Sage [0-9a-z.]* /Sage '$new_version' /'; git init && (echo "*.svg binary"; echo "*.pdf binary") >> .gitattributes && (echo ".buildinfo"; echo '*.inv'; echo '.git*'; echo '*.svg'; echo '*.pdf'; echo '*.png'; echo 'searchindex.js') > .gitignore; git add -A && git commit --quiet -m "old")
+
+      - name: Download upstream artifact
+        uses: actions/download-artifact@v3
+        with:
+          path: upstream
+          name: upstream
+
+      - name: Apply CI fixes from sagemath/sage
+        # After applying the fixes, make sure all changes are marked as uncommitted changes.
+        run: |
+          if [ -r upstream/ci_fixes.patch ]; then
+            (cd worktree-image && git commit -q -m "current changes" --allow-empty -a && git am; git reset --quiet old; git add -N .) < upstream/ci_fixes.patch
+          fi
+
+      - name: Incremental build
+        id: incremental
+        run: |
+          # Now re-bootstrap and build. The build is incremental because we were careful with the timestamps.
+          ./bootstrap && make build
+        working-directory: ./worktree-image
+        env:
+          MAKE: make -j2
+          SAGE_NUM_THREADS: 2
+
+      - name: Build (fallback to non-incremental)
+        id: build
+        if: always() && steps.worktree.outcome == 'success' && steps.incremental.outcome != 'success'
+        run: |
+          set -ex
+          make doc-clean doc-uninstall sagelib-clean && git clean -fx src/sage && ./config.status && make build
+        working-directory: ./worktree-image
+        env:
+          MAKE: make -j2
+          SAGE_NUM_THREADS: 2
 
-      - name: Build
+      - name: Build docs (PDF)
+        id: docbuild
+        if: always() && (steps.incremental.outcome == 'success' || steps.build.outcome == 'success')
         run: make build V=0 && make doc-pdf
+        working-directory: ./worktree-image
         env:
           MAKE: make -j2
           SAGE_NUM_THREADS: 2
-          TEXMFHOME: /sage/texmf
 
       - name: Copy docs
+        if: always() && steps.docbuild.outcome == 'success'
         run: |
           # For some reason the deploy step below cannot find /sage/...
           # So copy everything from there to local folder
@@ -51,6 +144,7 @@ jobs:
           zip -r docs-pdf.zip docs
 
       - name: Upload docs
+        if: always() && steps.copy.outcome == 'success'
         uses: actions/upload-artifact@v3
         with:
           name: docs-pdf

.github/workflows/doc-build.yml

@@ -30,7 +30,9 @@ jobs:
       - name: Store CI fixes in upstream artifact
         run: |
           mkdir -p upstream
-          git format-patch --stdout test_base > upstream/ci_fixes.patch
+          if git format-patch --stdout test_base > ci_fixes.patch; then
+            cp ci_fixes.patch upstream/
+          fi
       - uses: actions/upload-artifact@v3
         with:
           path: upstream