Build system for 2019

[isagalaev]

Our build system has outlived its time and needs to be replaced.

Current principles

• The library has to work in the browser as well as on the server under node.js. To achieve this the code is kept devoid of any module structure and the build system is responsible for wrapping and concatenating files into either a browser-ready script or a node.js compatible module.

• We don't simply host a library on npm and use a browserifying solution because our custom build system has custom compression code which is more efficient than generic solutions. It used to matter when we tried to keep the default build size under 30 KB.

• Browsers seriously suck at requesting many resources at once over HTTP/1.1, which is the reason to have a single packed file with the core and a subset of languages.

• There is a "common" subset of languages which is built for every release and hosted on a CDN. It is a subset because the full set weighs 500+ KB and most of the languages is a long tail that most users don't need.

Pain points

• Contributors don't like to mess with weird custom systems. This discourages contributions.
• Users don't understand why their "uncommon" language doesn't work. Requiring to read README for basic usage is bad UX.
• Build system is implemented using an old unsupported library (if I remember correctly).

Proposals

• Host the library in the most common and obvious way in today's JavaScript world (I'm not the one to offer specifics).
• Provide languages as separate files, so users can initialize the library with an explicit list of languages which will be downloaded dynamically. We had that in pre-historic time, but these days it should actually be feasible with HTTP/2. This should fix the second pain point, as users will know what they're requesting.
• Drop the custom compression scheme.
• (optional) It might make sense to severely limit the set of languages supported within the main repository, with others moving outside for community support, hosted somewhere else. For that the API between the library and languages needs to (finally) be stabilized.

[marcoscaceres]

It might make sense to severely limit the set of languages supported within the main repository, with others moving outside for community support, hosted somewhere else. For that the API between the library and languages needs to (finally) be stabilized.

New languages should move to npm, and not maintained by us. We need to look at how Babel plugins work, for instance. People should not be blocked on us and should not depend on us for their own languages.

[joshgoebel]

I tested just now the custom build with and without compression. I still minified it... I just skipped compress log, replace2,3,4, etc...

The file sizes (pre gzip) were 743kb vs 792kb... (that's a full build AFAIK, no limits specified)

And after gzip:
251kb vs 254kb (3kb difference for the custom compression)

Isn't the assumption now that everyone is gziping everything? Could all the custom compression stuff simply be removed and we switch to ES6 modules... so you get the node/npm setup for free... then use something light like rollup to build the modules into a web friendly format (with a tiny bit of help from a build wrapper).

That same wrapper could be tweaked to make dropping in external repos and just "linking" them easy when building a web version of the script...

Thoughts?