Discuss: 3rd party language packages spec

[joshgoebel]

Starting a discussion for an official 3rd party grammar format/spec (directory structure, how modules are exported, etc). I've been working on adding support to the build process for seamlessly building 3rd party grammars. The idea being that you just check them out:

mkdir extra
cd extra
git clone git@github.com:highlightjs/highlightjs-robots-txt.git
git clone git@github.com:highlightjs/highlightjs-tsql.git

And then building "just works":

node ./tools/build.js -t browser :common tsql robots-txt

And high-level grammar testing just works (detect tests, markup tests, etc) in context of the whole test suite. Just run tests like normal. The tests for extra languages are bundled with the extra languages, but magically "just work".

In looking at the existing 3rd party stuff it seems we have two major types of layouts. I think perhaps we could support both.

The slightly flatter npm package:

A more "bundled"/traditional layout:

One alternative to supporting both is to always prefer/require the bundled layout. I prefer this layout slightly as it makes it easier to add multiple languages to a single "package/repo". For example you might bundle Python, Python REPL... bundled layout gives you an easy way to do that (keeping the same directly structure we use internally). Also if we ever again decided to pull a 3rd party into the core library it'd be trivial to do with the bundled layout as it'd just essentially be copying files from one repo to another.

As to how NPM would work with bundled layout I'd suggest an index.js that simply required the individual modules and exported a hash of them by name:

# index.js
const python = require("./src/languages/python")
const pythonREPL = require("./src/languages/python_repl")
module.exports = { python, pythonREPL }

Thoughts?

[joshgoebel]

Also an anti-pattern (in my opinion) I'm seeing is:

module.exports = function(hljs) {
  hljs.registerLanguage('robots-txt', hljsDefineRobotsTxt);
};

module.exports.definer = hljsDefineRobotsTxt;

Resulting in custom registration code that hides the language name/key:

var hljsDefineRobotsTxt = require('highlightjs-robots-txt');
hljsDefineRobotsTxt(hljs);

---

I'm not sure what the big advantage here is and I'd prefer to instead see:

module.exports = function(hljs) {
  return {
    case_insensitive: true,
    contains: [
    ...

or (same thing really):

function tsql(hljs) { ... }

module.exports = tsl

And README examples like:

var tsqlGrammar = require('highlightjs-robots-tsql');
hljs.registerLanguage('tsql',tsqlGrammar);

IE, what is exported is the grammar function itself, nothing else. Prefer to let people do the registration themselves. This will allow a simpler build process and also I like that it keeps the naming explicit rather than implicit.

[joshgoebel]

From another thread about testing 3rd party languages in isolation.

Feel free to prototype some sort of prospective shared .travis.yml in https://github.com/highlightjs/highlightjs-shexc.

That might wait until later (or require someone else to step up). My focus right now is to make things "just work" in the context of a working highlight.js checkout. (Since you need that to build the actual library anyways, etc). And of course that's immediately most useful to me, as a maintainer of the library - everything in one place.

So checkout highlight.js and then vendored language projects, run full highlight.js test suite. No 3rd step.

---

The other side of things (as you suggest) would be to have your language stand-alone... then you'd pull in highlight.js as a dependency via npm/yarn... and you might have some sort of scaffold to run your custom tests plus "standardized" detect/markup test... of course currently none of the testing stuff ships with our npm library at all since the library itself is just a build product of the actual source.

Open to discussion this element of things here, but it might be outside the scope of what I'm currently working on.

[joshgoebel]

Another question is how to distribute "web"/CDN versions of languages. Currently I was imagining the source files would all be CommonJS (module.exports)... so they'd work with Node out of the box, but require building for distributable/web usage. I'd be game for ES6 modules, but Node.js isn't 100% there yet, so CommonJS is probably simpler for now.

I think some people may want an easy way to just get the CDN ready version... of course our build process could handle this and then drop CDN ready "binaries" in the dist directory of 3rd party languages (that they could then check-in/distribute)... Short of that I think we're looking at something like UMD? But that seems pretty ugly and complex for this... no?

Ok, actually Solidity does it with just a custom function name (which of course pollutes the global namespace also), but I think this is a bit messy:

// in the external solidity.js file
function hljsDefineSolidity(hljs) {
...

// and then to register after you loaded the JS
hljs.registerLanguage('solidity', window.hljsDefineSolidity);