doc: turbo mode for kconfig options #12631

nashif · 2019-01-21T21:56:35Z

Building the complete documentation is a time consuming task due to the amount
of auto-generated pages we create during the documentation build process. When
making major changes to the documentation and rebuilding the documentation very
often it is possible to skip the Kconfig option generation and just enable
generation empty option definition to satisfy references.

To build in turbo mode, use

make htmldocs-fast

zephyrbot · 2019-01-21T22:01:15Z

All checks are passing now.

Review history of this comment for details about previous failed status.
Note that some checks might have not completed yet.

codecov-io · 2019-01-21T22:42:18Z

Codecov Report

Merging #12631 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #12631   +/-   ##
=======================================
  Coverage   53.94%   53.94%           
=======================================
  Files         242      242           
  Lines       27654    27654           
  Branches     6717     6717           
=======================================
  Hits        14917    14917           
  Misses       9932     9932           
  Partials     2805     2805

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b43aa5...6823467. Read the comment docs.

dbkinder

When I tried to build documents with KCONFIG_DOC_TURBO_MODE unset, the build fails within 10 seconds with only the message ninja: build stopped: subcommand failed:

~/zephyr$ echo $KCONFIG_DOC_TURBO_MODE

~/zephyr$ source zephyr-env.sh
~/zephyr$ time make htmldocs
mkdir -p doc/_build && cmake -GNinja -DDOC_TAG=development -DSPHINXOPTS=-q -Bdoc/_build -Hdoc/ && ninja -C doc/_build htmldocs
-- Zephyr base: /home/dbkinder/zephyr
Zephyr version: 1.13.99
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.7", minimum required is "3.4")
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.13") found components:  doxygen
-- Found LATEX: /usr/bin/latex
-- Configuring done
-- Generating done
-- Build files have been written to: /home/dbkinder/zephyr/doc/_build
ninja: Entering directory `doc/_build'
[1/4] cd /home/dbkinder/zephyr/doc && /home/dbkinder/.local/lib/python3.6/site-packages/cmake/data/bin/cmake -...CH=x86 /usr/bin/python3 scripts/genrest.py Kconfig /home/dbkinder/zephyr/doc/_build/rst/doc/reference/kconfig/
FAILED: CMakeFiles/kconfig
cd /home/dbkinder/zephyr/doc && /home/dbkinder/.local/lib/python3.6/site-packages/cmake/data/bin/cmake -E make_directory /home/dbkinder/zephyr/doc/_build/rst/doc/reference/kconfig && /home/dbkinder/.local/lib/python3.6/site-packages/cmake/data/bin/cmake -E env PYTHONPATH="/home/dbkinder/zephyr/scripts/kconfig:" srctree=/home/dbkinder/zephyr KERNELVERSION=0x10D6300 BOARD_DIR=boards/*/*/ ARCH=* SOC_DIR=soc/ SRCARCH=x86 /usr/bin/python3 scripts/genrest.py Kconfig /home/dbkinder/zephyr/doc/_build/rst/doc/reference/kconfig/
[3/4] cd /home/dbkinder/zephyr/doc/_build && /home/dbkinder/.local/lib/python3.6/site-packages/cmake/data/bin/...xy.log -DWORKING_DIRECTORY=/home/dbkinder/zephyr/doc -P /home/dbkinder/zephyr/cmake/util/execute_process.cmake
ninja: build stopped: subcommand failed.
Makefile:16: recipe for target 'htmldocs' failed
make: *** [htmldocs] Error 1

real    0m9.599s
user    0m8.611s
sys     0m1.475s

dbkinder · 2019-01-22T00:12:08Z

An interesting optimization, but I'm not sure this PR is really needed for developers.

Today, generating full documentation in a clean doc build environment (with an empty doc/_build folder) takes about 7m40s (on my Ubuntu box). We've implemented optimizations so a rebuild of documentation with some reST file changes takes only 1m38s (when the doc/_build folder already exists). Sphinx knows not to regenerate HTML if the corresponding reST file didn't change, and the Kconfig automation doesn't update a config option reST file if it would be the same as was previously generated.

With this PR, a doc build without generating the Kconfig options (in a clean doc build environment) takes about 2m33s and a rebuild (with an existing doc/_build folder) takes 1m30s), but the Kconfig option descriptions are missing.

If the intent was to expedite developers rebuilding documentation in a working environment where they're making reST changes, checking the output, and rebuilding docs again, what we have today seems to work just fine, after the first doc build.

That said, our current CI process always builds documentation in a clean environment (with an empty doc/_build folder), so this PR would indeed reduce the CI doc build processing by about 6 minutes, but would not be verifying the full docs would build properly (if there were errors in the Kconfig option documentation).

nashif · 2019-01-22T01:01:03Z

An interesting optimization, but I'm not sure this PR is really needed for developers.

I need that and and anyone doing anything serious with the documentation would need that. The rebuild performance is useless for many cases and is error prone, we copy files around and we end up with inconsistent results, especially when doing some major restructuring of the docs.

But regardless, waiting the initial 7m40sis is 5 minutes wasted that I will never get back, why should I wait ~8 minutes when I can get things done in ~2?. Obviously whoever is going to use that need to know what they are doing.

nashif · 2019-01-22T01:03:40Z

When I tried to build documents with KCONFIG_DOC_TURBO_MODE unset, the build fails within 10 seconds with only the message ninja: build stopped: subcommand failed:

yeah, leftover debug code.

dbkinder · 2019-01-22T02:30:02Z

doc/README.rst

To position this a bit differently and fix some typos, how about:

Developer-mode Document Building ******************************** Building the documentation for all the Kconfig options significantly adds to the total doc build time. When making and testing major changes to the documentation, we provide an option to temporarily stub-out the auto-generated configuration documentation so the doc build process runs much faster. To enable this "turbo" doc generation mode, before building the documents, export the following environment variable (on Linux and macOS):: export KCONFIG_DOC_TURBO_MODE=1 and then build the documentation as usual. .. note:: Be sure to unset the KCONFIG_DOC_TURBO_MODE variable before building the documentation for publication purposes.

dbkinder · 2019-01-22T02:33:02Z

doc/scripts/genrest.py

I think this will run in TURBO_MODE if KCONFIG_DOC_TURBO_MODE has any value, right?

carlescufi · 2019-01-22T12:03:28Z

I am fine with the change, but I suggest we do not use environment variables since those are terrible on Windows. Can we use instead either:

a -DKCONFIG_TURBO_MODE CMake variable
or
a new target ninja html-turbo
?

dbkinder

OK.

nashif · 2019-01-22T17:25:15Z

OK.

I can feel the excitement.

dbkinder · 2019-01-22T18:52:50Z

OK!!!! :)

nashif · 2019-01-22T19:04:09Z

@carlescufi

I am fine with the change, but I suggest we do not use environment variables since those are terrible on Windows. Can we use instead either:

addressed

ulfalizer

Could the turbo_mode stuff just be a single if turbo_mode? That way it wouldn't get tangled up in the normal path.

Could get rid of a variable that way too maybe, like this:

if os.environ.get("KCONFIG_TURBO_MODE") == "1":
    # Generate dummy index page, etc.
else:
    ...

doc/scripts/genrest.py

ulfalizer · 2019-01-23T08:50:49Z

doc/scripts/genrest.py

Could reuse the same index_rst variable in both cases here maybe. It's just different RST being generated for the index page, if I'm reading it right.

ulfalizer · 2019-01-23T08:57:33Z

doc/scripts/genrest.py

Why is a different file being output here, in addition to index.rst?

need to create the definition of the option in RST, otherwise we will have broken references.

ulfalizer · 2019-01-23T08:59:10Z

doc/scripts/genrest.py

Could index_rst be reused here instead?

no, need to generate two files.

carlescufi

LGTM

Building the documentation for all the Kconfig options significantly adds to the total doc build time. When making and testing major changes to the documentation, we provide an option to temporarily stub-out the auto-generated configuration documentation so the doc build process runs much faster. To enable this mode, set the following option when invoking cmake -DKCONFIG_TURBO_MODE=1 Signed-off-by: Anas Nashif <[email protected]>

addressed

marc-hb · 2019-02-08T01:04:39Z

a rebuild of documentation with some reST file changes takes only 1m38s (when the doc/_build folder already exists).

And now TURBO_TURBO mode: 8 seconds with PR #13159

marc-hb · 2019-02-15T21:28:54Z

TURBO x3: do not re-run doxygen for no reason PR #13442

nashif requested review from carlescufi and dbkinder as code owners January 21, 2019 21:56

nashif force-pushed the turbo_html branch from 9f35325 to 6bdc3f7 Compare January 21, 2019 21:58

nashif force-pushed the turbo_html branch from 6bdc3f7 to 4546a0d Compare January 21, 2019 22:07

dbkinder suggested changes Jan 21, 2019

View reviewed changes

nashif force-pushed the turbo_html branch 2 times, most recently from 182847a to d1dd9c3 Compare January 22, 2019 01:03

dbkinder suggested changes Jan 22, 2019

View reviewed changes

nashif force-pushed the turbo_html branch from d1dd9c3 to 71c5acb Compare January 22, 2019 14:11

nashif requested review from galak and ulfalizer January 22, 2019 15:36

dbkinder approved these changes Jan 22, 2019

View reviewed changes

ulfalizer previously requested changes Jan 23, 2019

View reviewed changes

carlescufi approved these changes Jan 23, 2019

View reviewed changes

nashif force-pushed the turbo_html branch from 71c5acb to 6823467 Compare January 23, 2019 13:33

nashif merged commit 940a931 into zephyrproject-rtos:master Jan 23, 2019

doc: turbo mode for kconfig options #12631

doc: turbo mode for kconfig options #12631

Uh oh!

Conversation

nashif commented Jan 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zephyrbot commented Jan 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-io commented Jan 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dbkinder left a comment

Choose a reason for hiding this comment

Uh oh!

dbkinder commented Jan 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nashif commented Jan 22, 2019

Uh oh!

nashif commented Jan 22, 2019

Uh oh!

dbkinder Jan 22, 2019

Choose a reason for hiding this comment

Uh oh!

dbkinder Jan 22, 2019

Choose a reason for hiding this comment

Uh oh!

carlescufi commented Jan 22, 2019

Uh oh!

dbkinder left a comment

Choose a reason for hiding this comment

Uh oh!

nashif commented Jan 22, 2019

Uh oh!

dbkinder commented Jan 22, 2019

Uh oh!

nashif commented Jan 22, 2019

Uh oh!

ulfalizer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ulfalizer Jan 23, 2019

Choose a reason for hiding this comment

Uh oh!

ulfalizer Jan 23, 2019

Choose a reason for hiding this comment

Uh oh!

nashif Jan 23, 2019

Choose a reason for hiding this comment

Uh oh!

ulfalizer Jan 23, 2019

Choose a reason for hiding this comment

Uh oh!

nashif Jan 23, 2019

Choose a reason for hiding this comment

Uh oh!

carlescufi left a comment

Choose a reason for hiding this comment

Uh oh!

marc-hb commented Feb 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marc-hb commented Feb 15, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

nashif commented Jan 21, 2019 •

edited

Loading

zephyrbot commented Jan 21, 2019 •

edited

Loading

codecov-io commented Jan 21, 2019 •

edited

Loading

dbkinder commented Jan 22, 2019 •

edited

Loading

marc-hb commented Feb 8, 2019 •

edited

Loading