Filesize of generated static html documents is pretty heavy

[platzhersh]

Search Terms

filesize, static, generated, output, html

Problem

We are currently using Typedoc in a project to generate our code documentation for Gitlab Pages. Basically we are pretty happy with the tool, but we just realised that the filesize of the generated output is pretty heavy. We went over the default filesize limit of 100MB pretty soon.
Currently we have around 650 files being generated, with a filesize of around 120MB (more or less 170kb per generated file).

Suggested Solution

I was wondering if others have problems with this too and if there are plans to optimize the static output.
A minimize option would be nice.

[Gerrit0]

TypeDoc's output is fairly well optimized already, I took another look, and could make it somewhat better by changing how icons are cached (TypeDoc's docs went from 17->14mb, with 200 pages)

I also updated --pretty false to not print lines after block elements when rendering HTML (though you may still have newlines from marked output), pretty printing is on by default.

With 650 files, I suspect a large part of your generated doc size is repeating the navigation tree on each page. The primary navigation tree doesn't change much on a page-by-page basis (just which element is marked active), so one possible implementation change could be to push that into a JSON file which is parsed into the navigation on page load. While this would reduce the file size, it unfortunately means that navigating the site without JavaScript enabled would become effectively impossible, which I consider to be an important feature.

[platzhersh]

With 650 files, I suspect a large part of your generated doc size is repeating the navigation tree on each page.

Correct, that's exactly the case. I also get why you are hesitant to consider externalising the navigation into a json. Maybe this could be an optional setting for the build command? So per default it would generate as of now, with pure static html, but optionally you could enable the dynamic navigation with the JSON file?

[benhenny]

I'm also running into this problem while trying to upgrade the version of typedoc we're using from 0.17.8 to to 0.24.7. Our project is large, with 3500 files. The size of the generated html went from 60MB to 3GB and the render time went from 20 seconds to 5 minutes.

To confirm that it is the navigation causing the larger size I temporarily commented out the default theme's sidebar rendering. That brought the size back down to 80MB.

I believe the theme in 0.17.8 only generated the navigation entries for the current page, and not the entire tree. In addition to @platzhersh's suggestion (which would also work for me), could another approach be to have an option to generate a minimal navigation menu for each page? That would continue to work without Javascript, while also keeping the file size manageable for large projects.

[krisanselmo]

Same issue here after moving toward 0.24.7 !

Before 0.23.28

After 0.24.7

[Gerrit0]

Yikes, turning off the global navigation at least definitely needs to be an option... probably externalizing navigation too.

[Gerrit0]

0.24.8 includes a navigation.fullTree option (defaults to false) which addresses the biggest issue here. I really wanted to change this to render the partial tree statically, then the full tree dynamically using JS... but doing this properly is more work than I have time for today, so it will probably be delayed until 0.25, I'm going to leave this issue open to track doing that.

[platzhersh]

@Gerrit0 nice, I will give that a try asap. What does navigation.fullTree effectively do?

[typhonrt]

Glad to find a mention of navigation.fullTree as things went missing from expectations when updating from 0.24.7. What I see as the difference is that namespaces won't render / produce contained content in the navigation sidebar, etc. Setting fullTree to true gets things back to previous behavior.

Some interesting general news though related to the original post. I've been working for a bit on creating a theme though it's more like an augmentation of the default / theme where further processing occurs to the default HTML output vs replacing anything on the JSX side of things. I have implemented a left side navigation solution that dynamically creates a Svelte component on the fly during doc gen right at the beginning then promptly turns off navigation static generation. This reduces the disk space required by upwards of 90% for large projects, but equally impressive also reduces the doc generation time by 80%! :D I do very much believe the direction of dynamically generating the left hand navigation is a big win for small to large projects.

One of the big reasons to do this is precisely what is mentioned in the original issue. The default left side navigation simply linearly increases for every page potentially rendered. I am working on creating API docs for the built-in TS libs + WebGPU / WebCodec / WebXR. The DOM library is ~2400 pages. The default theme requires 1.4GB of disk space; each HTML page is at least ~550KB with the majority of the static navigation HTML taking the brunt of that size. The default search index is ~17MB. With my default theme augmentation it becomes 140MB on disk and a 1.5MB search index and takes ~50 seconds on my box vs 4 1/2 minutes.

Of course this requires Javascript enabled. I have also made several other general usability improvements with this "theme augmentation" though it will require a recent browser; IE I am using container queries to improve styles. All of this done hopefully in a way that can maintain compatibility w/ the default theme as it evolves.

I'll likely have this default theme augmentation package out in ~1 week. I'd be glad to coordinate perhaps w/ some early outside testing for anyone interested soon.

[Gerrit0]

@Gerrit0 nice, I will give that a try asap. What does navigation.fullTree effectively do?

The default only renders the navigation tree necessary to get to the current page, setting fullTree restores the behavior prior to 0.24.8.

The default search index is ~17MB. With my default theme augmentation it becomes 140MB on disk and a 1.5MB search index

This confuses me, did you replace the search function? The search index builder shouldn't care what the navigation looks like...

[typhonrt]

The default search index is ~17MB. With my default theme augmentation it becomes 140MB on disk and a 1.5MB search

This confuses me, did you replace the search function? The search index builder shouldn't care what the navigation looks like...

I did replace the default search index creation and short circuit the initSearch function in main.js. I still use lunr, but save the search index w/ MessagePack (msgpackr) and compress w/ fflate; both of which work great in Node & the browser. This gets a sizable decrease of the search index. I also have an additional option that can turn off the main search index creation / display entirely. The search UI is replaced w/ a Svelte implementation using the same data as the default theme just stored compactly.

I'm going for maximum size reduction to reduce the costs of any hosting of the built-in TS libs docs I'll be putting up soon.

Just a few more days and you can check out the results on some hosted docs. I kind of call it a "theme augmentation" vs replacement as I'm modifying the default theme output as it is generated. Totally down to upstream anything that makes sense and fits the more strict design goals of the default theme.

[typhonrt]

@platzhersh el al
I just released my "Default Modern Theme" that has the 90% disk space utilization reduction and a bit more. I'd be glad to get your feedback regarding your or others use cases for large project documentation generation with it.

My general estimate is that the ~120MB total size you initially posted about above will likely end up around ~11-12MB with the DMT; well within the Gitlab Pages limit.

I made a general discussion thread here: #2335