ci: parallelize the mdbook build <language> process #2772
Conversation
…issions and publish steps
|
Down from 25 minutes to 11 minutes. Less than expected, but also less than half the previous run time |
|
Nice!! |
|
This now finished (even with missing rust caches!) in 21 minutes. https://github.com/google/comprehensive-rust/actions/runs/15612132409 With caching this would get down quite a lot. @mgeisler can you review the comprehensive-rust-all artifact if that is the correct structure? This would show that uploading the intermediate steps and artifact downloading with merge works as expected. The workflow still needs to be refactored a little bit as there is quite some duplication now but this only serves as a POC as of now |
.github/workflows/publish.yml
Outdated
| - name: Update Rust | ||
| run: rustup update | ||
|
|
||
| - name: Setup Rust cache | ||
| uses: ./.github/workflows/setup-rust-cache | ||
|
|
||
| - name: Install Gettext | ||
| run: | | ||
| sudo apt update | ||
| sudo apt install gettext | ||
|
|
||
| - name: Install mdbook | ||
| uses: ./.github/workflows/install-mdbook |
There was a problem hiding this comment.
Small question: do we need to install Rust, mdbook, and the Gettext tools here? I would guess not since everything is built already, no? We're just putting the pieces together and uploading it all?
There was a problem hiding this comment.
No... but we do ;)
We actually only need i18n-report, that is installed in the install-mdbook step cd922c4
There was a problem hiding this comment.
The code changed over time a bit but essentially cargo xtask install-tools installs the necessary tool at this moment. For simplification and POC this is just installed as is. This needs to be refactored
There was a problem hiding this comment.
I don't know exactly if i18n-report requires gettext.
I could upload the .cargo/bin/i18n-report binary in the previous job then download it in all jobs to use use that. This has the potential to be a bit more obscure in the future. But it probably is already pretty obscure where this is from at this point.
There was a problem hiding this comment.
while doing this and already making this a bit more obscure, I could create a tool-building phase that uploads all rust tools so we can upload the entire .cargo/bin/ directory and just use this in every translation
| - name: Deploy to GitHub Pages | ||
| id: deployment | ||
| uses: actions/deploy-pages@v4 |
There was a problem hiding this comment.
I'm surprised this went away? Is this not the most important part, so to speak? 😄
There was a problem hiding this comment.
yes, it is! This was removed for development and testing purposes to not accidentally deploy stuff
I think this is a great approach, well done, @michael-kerscher! Originaly, we only had a simple As for the caching, if building |
michael-kerscher
force-pushed
the
parallel-build
branch
from
97e83de to
45218fc
Compare
…steps. This should be a massive performance gain in the publish step as we don't build several rust tools from scratch anymore for each language
… build/install logs
…ng that does not preserve permissions)
This needs to be refactored later
michael-kerscher
force-pushed
the
parallel-build
branch
from
2ee0b73 to
8805afc
Compare
Make the mdbook build process parallel. This is a simple approach by just cloning the entire repository into new directories and running the process in parallel. This is a demonstration for #2767.
The process appears to be mostly CPU bound and not memory or disk heavy so the amount of cores on the CI runner matter.
mdbook build does not seem to use more than 1 core and a typical github CI machine has 4 cores. So the expected speedup would be 4x.
Currently 30 minutes are used for publish, 25 minutes of this is build process, this would be reduced to 5 + 6.25 = ~12 minutes, thus more than halving the process.