doc: fix incremental build by fixing doxygen output mtime

[marc-hb]

Doxygen doesn't support incremental builds. It could be ok because it
doesn't take much time in this codebase. However it's not because it
makes old output look new which has a cascade effect on sphinx:
https://sourceforge.net/p/doxygen/mailman/message/36580807/

Make doxygen artificially smarter by saving and restoring modify
timestamps on the filesystem for doxygen output files that haven't
changed.

On my system this brings down the time to run an incremental "make
htmldocs" from 75s down to 8s (cmake -DKCONFIG_TURBO_MODE=1 used in both
cases)

This optimization speeds-up non-doxygen documentation work only.

Signed-off-by: Marc Herbert [email protected]

[codecov-io]

Codecov Report

Merging #13159 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #13159   +/-   ##
=======================================
  Coverage   48.73%   48.73%           
=======================================
  Files         315      315           
  Lines       46540    46540           
  Branches    10743    10743           
=======================================
  Hits        22679    22679           
  Misses      19349    19349           
  Partials     4512     4512

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a7b45d...e22807e. Read the comment docs.

[marc-hb]

sphinx-html was just mentioned to me. Unlike sphinx-html, this doesn't take any shortcut hence safely regenerates everything that's needed; slowly in case of some actual doxygen change and faster otherwise.

[carlescufi]

Hey thank you! This had been in my list to look into for a while. I have no objections to this, but have you tested on Windows? I will try to give it a go myself.

[dcpleung]

LGTM

[marc-hb]

Note there's a much simpler solution if rsync becomes an acceptable Windows requirement, no need for the new python script https://sourceforge.net/p/doxygen/mailman/message/36582383/

According to cmake's documentation, "copy_if_different" copies files and not directories recursively:
https://cmake.org/cmake/help/v3.13/manual/cmake.1.html#command-line-tool-mode
I trusted the documentation and I haven't tried.

[dbkinder]

The patch isn't as dramatic a performance gain on windows, but still quite significant: goes from 5 minute rebuild without this patch to 2 mins.

On a Linux box, it's worth noting that this patch gives a doc rebuild time of about 12 seconds for a full doc build (including all the configuration docs), so for developers building local documents a few times to validate changes, this is a big win for subsequent rebuilds.

[dbkinder]

👍

[marc-hb]

According to cmake's documentation, "copy_if_different" copies files and not directories recursively

I just tested cmake -E copy_if_different and I confirm it doesn't copy directories. It (silently!) ignores all directory arguments and copies files only.

So the only alternative solution left is requesting Windows users to install rsync; however there's a high risk of making Windows a better place https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6

[marc-hb]

So this fixes the incremental build time for pure .rst changes but not yet for doxygen changes, there's another and unrelated issue when doing doxygen work.

I observed that changing 1 doxygen XML file has the exact same effect as changing all doxygen files. I investigated further and found that the sphinx extension for doxygen "breathe" wrongly believes that "everything depends on everything". I reported this unrelated breathe issue in breathe-doc/breathe#420

[carlescufi]

@marc-hb rsync will never be acceptable on Windows because we support native Windows Command Prompt, not Windows Subsystems for Linux or any emulation layer like Cygwin or MSYS.

[marc-hb]

There is at least one native rsync build for Windows too: DeltaCopy https://serverfault.com/questions/35336/why-hasnt-rsync-caught-on-in-the-windows-world
(I have no idea how much pain and effort a DeltaCopy requirement would add for Windows users, mentioning just for completeness)

[marc-hb]

I just tested this on a Macbook Pro with Mojave. It brings the time to run make htmldocs-fast SPHINXOPTS=-v from 2 minutes down to 30s (only on the second run / incremental build of course)
Using SPHINXOPTS=-v and staring at the logs and "top" shows that the 30 remaining seconds are entirely spent running doxygen, the 1 min 30 s spent in sphinx is gone. @dbkinder it would be interesting if you could repeat your Windows measurement and observe the same (task manager or resmon.exe)

[marc-hb]

The patch isn't as dramatic a performance gain on windows, but still quite significant: goes from 5 minute rebuild without this patch to 2 mins.

I suspect most of these 2 mins are spent re-running doxygen: fix submitted in PR #13442 and tested on Linux and macOS. Curious how it fares on Windows.

[marc-hb]

For the record there is - at least in theory - a simpler, one-way solution: https://sourceforge.net/p/doxygen/mailman/message/36582383/

I'm just suggesting that you use Doxygen to generate a "scratch" copy that
you never deploy (and could delete immediately after rsyncing) and using a
one-way rsync to another location where you store the "real" copy for
use/deployment/processing/etc. Sphinx would run on the "real" copy, not the
scratch one
...
Even though I modified all 3 files, rsync only copied the two with changed content. The file with no changes keeps its old timestamp.