Skip to content

Commit a19cdc7

Browse files
committed
Add the rustdoc readme
1 parent 2ed15c2 commit a19cdc7

File tree

2 files changed

+178
-0
lines changed

2 files changed

+178
-0
lines changed

src/SUMMARY.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
- [Walkthrough: a typical contribution](./walkthrough.md)
1111
- [High-level overview of the compiler source](./high-level-overview.md)
1212
- [The Rustc Driver](./rustc-driver.md)
13+
- [Rustdoc](./rustdoc.md)
1314
- [Queries: demand-driven compilation](./query.md)
1415
- [Incremental compilation](./incremental-compilation.md)
1516
- [The parser](./the-parser.md)

src/rustdoc.md

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
# The walking tour of rustdoc
2+
3+
Rustdoc actually uses the rustc internals directly. It lives in-tree with the compiler and standard
4+
library. This chapter is about how it works. (A new implementation is also [under way], though).
5+
6+
[under way]: https://github.com/steveklabnik/rustdoc
7+
8+
Rustdoc is implemented entirely within the crate `librustdoc`. After partially compiling a crate to
9+
get its AST (technically the HIR map) from rustc, librustdoc performs two major steps past that to
10+
render a set of documentation:
11+
12+
* "Clean" the AST into a form that's more suited to creating documentation (and slightly more
13+
resistant to churn in the compiler).
14+
* Use this cleaned AST to render a crate's documentation, one page at a time.
15+
16+
Naturally, there's more than just this, and those descriptions simplify out lots of details, but
17+
that's the high-level overview.
18+
19+
(Side note: this is a library crate! The `rustdoc` binary is crated using the project in
20+
`src/tools/rustdoc`. Note that literally all that does is call the `main()` that's in this crate's
21+
`lib.rs`, though.)
22+
23+
## Cheat sheet
24+
25+
* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on
26+
other projects.
27+
* Add `src/libtest` to be able to use `rustdoc --test`.
28+
* If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1` previously, then
29+
after the previous build command, `cargo +local doc` will Just Work.
30+
* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the standard library docs.
31+
* The completed docs will be available in `build/$TARGET/doc/std`, though the bundle is meant to
32+
be used as though you would copy out the `doc` folder to a web server, since that's where the
33+
CSS/JS and landing page are.
34+
* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`. It's in a bunch of
35+
`fmt::Display` implementations and supplementary functions.
36+
* The types that got `Display` impls above are defined in `clean/mod.rs`, right next to the custom
37+
`Clean` trait used to process them out of the rustc HIR.
38+
* The bits specific to using rustdoc as a test harness are in `test.rs`.
39+
* The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting
40+
doctests from a given block of Markdown.
41+
* The tests on rustdoc *output* are located in `src/test/rustdoc`, where they're handled by the test
42+
runner of rustbuild and the supplementary script `src/etc/htmldocck.py`.
43+
* Tests on search index generation are located in `src/test/rustdoc-js`, as a series of JavaScript
44+
files that encode queries on the standard library search index and expected results.
45+
46+
## From crate to clean
47+
48+
In `core.rs` are two central items: the `DocContext` struct, and the `run_core` function. The latter
49+
is where rustdoc calls out to rustc to compile a crate to the point where rustdoc can take over. The
50+
former is a state container used when crawling through a crate to gather its documentation.
51+
52+
The main process of crate crawling is done in `clean/mod.rs` through several implementations of the
53+
`Clean` trait defined within. This is a conversion trait, which defines one method:
54+
55+
```rust
56+
pub trait Clean<T> {
57+
fn clean(&self, cx: &DocContext) -> T;
58+
}
59+
```
60+
61+
`clean/mod.rs` also defines the types for the "cleaned" AST used later on to render documentation
62+
pages. Each usually accompanies an implementation of `Clean` that takes some AST or HIR type from
63+
rustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated
64+
items may have some extra processing in its `Clean` implementation, but for the most part these
65+
impls are straightforward conversions. The "entry point" to this module is the `impl Clean<Crate>
66+
for visit_ast::RustdocVisitor`, which is called by `run_core` above.
67+
68+
You see, I actually lied a little earlier: There's another AST transformation that happens before
69+
the events in `clean/mod.rs`. In `visit_ast.rs` is the type `RustdocVisitor`, which *actually*
70+
crawls a `hir::Crate` to get the first intermediate representation, defined in `doctree.rs`. This
71+
pass is mainly to get a few intermediate wrappers around the HIR types and to process visibility
72+
and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and `#[doc(hidden)]` are
73+
processed, as well as the logic for whether a `pub use` should get the full page or a "Reexport"
74+
line in the module page.
75+
76+
The other major thing that happens in `clean/mod.rs` is the collection of doc comments and
77+
`#[doc=""]` attributes into a separate field of the Attributes struct, present on anything that gets
78+
hand-written documentation. This makes it easier to collect this documentation later in the process.
79+
80+
The primary output of this process is a clean::Crate with a tree of Items which describe the
81+
publicly-documentable items in the target crate.
82+
83+
### Hot potato
84+
85+
Before moving on to the next major step, a few important "passes" occur over the documentation.
86+
These do things like combine the separate "attributes" into a single string and strip leading
87+
whitespace to make the document easier on the markdown parser, or drop items that are not public or
88+
deliberately hidden with `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one
89+
file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping
90+
private/hidden items can be bypassed by passing `--document-private-items` to rustdoc.
91+
92+
(Strictly speaking, you can fine-tune the passes run and even add your own, but [we're trying to
93+
deprecate that][44136]. If you need finer-grain control over these passes, please let us know!)
94+
95+
[44136]: https://github.com/rust-lang/rust/issues/44136
96+
97+
## From clean to crate
98+
99+
This is where the "second phase" in rustdoc begins. This phase primarily lives in the `html/`
100+
folder, and it all starts with `run()` in `html/render.rs`. This code is responsible for setting up
101+
the `Context`, `SharedContext`, and `Cache` which are used during rendering, copying out the static
102+
files which live in every rendered set of documentation (things like the fonts, CSS, and JavaScript
103+
that live in `html/static/`), creating the search index, and printing out the source code rendering,
104+
before beginning the process of rendering all the documentation for the crate.
105+
106+
Several functions implemented directly on `Context` take the `clean::Crate` and set up some state
107+
between rendering items or recursing on a module's child items. From here the "page rendering"
108+
begins, via an enormous `write!()` call in `html/layout.rs`. The parts that actually generate HTML
109+
from the items and documentation occurs within a series of `std::fmt::Display` implementations and
110+
functions that pass around a `&mut std::fmt::Formatter`. The top-level implementation that writes
111+
out the page body is the `impl<'a> fmt::Display for Item<'a>` in `html/render.rs`, which switches
112+
out to one of several `item_*` functions based on the kind of `Item` being rendered.
113+
114+
Depending on what kind of rendering code you're looking for, you'll probably find it either in
115+
`html/render.rs` for major items like "what sections should I print for a struct page" or
116+
`html/format.rs` for smaller component pieces like "how should I print a where clause as part of
117+
some other item".
118+
119+
Whenever rustdoc comes across an item that should print hand-written documentation alongside, it
120+
calls out to `html/markdown.rs` which interfaces with the Markdown parser. This is exposed as a
121+
series of types that wrap a string of Markdown, and implement `fmt::Display` to emit HTML text. It
122+
takes special care to enable certain features like footnotes and tables and add syntax highlighting
123+
to Rust code blocks (via `html/highlight.rs`) before running the Markdown parser. There's also a
124+
function in here (`find_testable_code`) that specifically scans for Rust code blocks so the
125+
test-runner code can find all the doctests in the crate.
126+
127+
### From soup to nuts
128+
129+
(alternate title: ["An unbroken thread that stretches from those first `Cell`s to us"][video])
130+
131+
[video]: https://www.youtube.com/watch?v=hOLAGYmUQV0
132+
133+
It's important to note that the AST cleaning can ask the compiler for information (crucially,
134+
`DocContext` contains a `TyCtxt`), but page rendering cannot. The `clean::Crate` created within
135+
`run_core` is passed outside the compiler context before being handed to `html::render::run`. This
136+
means that a lot of the "supplementary data" that isn't immediately available inside an item's
137+
definition, like which trait is the `Deref` trait used by the language, needs to be collected during
138+
cleaning, stored in the `DocContext`, and passed along to the `SharedContext` during HTML rendering.
139+
This manifests as a bunch of shared state, context variables, and `RefCell`s.
140+
141+
Also of note is that some items that come from "asking the compiler" don't go directly into the
142+
`DocContext` - for example, when loading items from a foreign crate, rustdoc will ask about trait
143+
implementations and generate new `Item`s for the impls based on that information. This goes directly
144+
into the returned `Crate` rather than roundabout through the `DocContext`. This way, these
145+
implementations can be collected alongside the others, right before rendering the HTML.
146+
147+
## Other tricks up its sleeve
148+
149+
All this describes the process for generating HTML documentation from a Rust crate, but there are
150+
couple other major modes that rustdoc runs in. It can also be run on a standalone Markdown file, or
151+
it can run doctests on Rust code or standalone Markdown files. For the former, it shortcuts straight
152+
to `html/markdown.rs`, optionally including a mode which inserts a Table of Contents to the output
153+
HTML.
154+
155+
For the latter, rustdoc runs a similar partial-compilation to get relevant documentation in
156+
`test.rs`, but instead of going through the full clean and render process, it runs a much simpler
157+
crate walk to grab *just* the hand-written documentation. Combined with the aforementioned
158+
"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of tests to run before
159+
handing them off to the libtest test runner. One notable location in `test.rs` is the function
160+
`make_test`, which is where hand-written doctests get transformed into something that can be
161+
executed.
162+
163+
## Dotting i's and crossing t's
164+
165+
So that's rustdoc's code in a nutshell, but there's more things in the repo that deal with it. Since
166+
we have the full `compiletest` suite at hand, there's a set of tests in `src/test/rustdoc` that make
167+
sure the final HTML is what we expect in various situations. These tests also use a supplementary
168+
script, `src/etc/htmldocck.py`, that allows it to look through the final HTML using XPath notation
169+
to get a precise look at the output. The full description of all the commands available to rustdoc
170+
tests is in `htmldocck.py`.
171+
172+
In addition, there are separate tests for the search index and rustdoc's ability to query it. The
173+
files in `src/test/rustdoc-js` each contain a different search query and the expected results,
174+
broken out by search tab. These files are processed by a script in `src/tools/rustdoc-js` and the
175+
Node.js runtime. These tests don't have as thorough of a writeup, but a broad example that features
176+
results in all tabs can be found in `basic.js`. The basic idea is that you match a given `QUERY`
177+
with a set of `EXPECTED` results, complete with the full item path of each item.

0 commit comments

Comments
 (0)