Provide access to a parse tree as an alternative to HTML output

[derhuerst]

I've just had a quick look at hightlight.js and didn't any discussion on this. So forgive me if this is not the right place or has already been discussed.

I'd like to propose to make highlight.js more generally usable, or at least pull out the core highlighting logic to make it independent from the output format. Just like the famous Pygments has formatters, the HTML generation would be done by a separate module/package.

highlight.js seems to be the de factor standard from syntax highlighting, even among porject not related to HTML/browsers. See the slap editor and hicat for examples: Both use it and later on manually walk/manipulate the HTML generated by highlight.js.

In the world of lightweight do-one-thing NPM packages, it would make sense to split highlight.js into two parts, one generating an treeish object representation (possibly streaming), and the second generating HTML from it.

[isagalaev]

This idea is hanging in the air for a while. It's never been a priority as it lacked real use-cases, but now as you mentioned those two it makes very real sense. Another problem is that in one or two attempts at doing this it was done as a part of bigger (and controversial) refactorings of the whole core library, so it haven't got merged. I will probably get to it myself eventually, but if you'd fancy giving it a try, I could give some guidance.

[derhuerst]

I will probably get to it myself eventually, but if you'd fancy giving it a try, I could give some guidance.

I'm having difficulties in understanding the inner workings of highlight.js, both because there is not a lot of documentation/comments and because the code is written in a really imperative jump-around way.

Also, it seems like HTML generation is really baked into the core of highlight.js.

[isagalaev]

Yes, it was deliberately done so in the past to squeeze the code size as much as possible. Currently this "optimization target" is much less desirable and that's why I'm all for refactoring this.

I'll do a short writeup about the architecture of the core to make it more comprehensible.

[derhuerst]

I'll do a short writeup about the architecture of the core to make it more comprehensible.

This sounds great, especially since there seems to be no render-agnostic snytax highlighter in JavaScript land right now.

[isagalaev]

I'll do a short writeup about the architecture of the core to make it more comprehensible.

A status update on this… Before writing anything I decided to first do a pretty significant upgrade to the parser that I was planning to do since a long time ago anyway but a few bugs came up lately bumped the importance of this, and incidentally I've finally came up with an actual plan on how to implement it. And there'd be no point in documenting the current core as it will change immediately after that.

(If interested, the change in question is one I described in this blog post, under the "Complex modes" section.)

[dbkaplun]

Author of slap text editor here. I was going to open a new ticket for this but looks like @derhuerst phrased it very well.

highlight.js has many other uses than highlighting in the browser. It would be great if parsing was decoupled from HTML generation. See also slap-editor/editor-widget#134. We are considering porting our current highlighting implementation using highlight.js, in combination with HTML manipulation with cheerio, to another option that would not require any HTML manipulation such as lowlight (which itself depends on highlight.js but does not seem to perform any HTML manipulation).

Glad to help out in any way if this feature request moves forward!

[okonet]

I'm using https://github.com/wooorm/lowlight and writing my RTF formatter ATM so I'd be interested in support for AST out of the box. I think this would allow adding different formatters as pygment does.

[joshgoebel]

Just like the famous Pygments has formatters, the HTML generation would be done by a separate module/package.

I think our HTML stuff has some unique benefits, but if there were two pieces people could always choice to use the one we bundled or replace it with a different once.

I've been digging into the parser a lot, so I'll keep this in mind.

@derhuerst Any idea what that parsed but not HTML format might look like?