Discuss: Custom token callbacks, or mode value property?

[foo123]

Hello, i'm playing around with highlight.js (9.2.0) and want to integrate with a -grammar add-on (following previous work on syntax-highlighting, for example here) which enables to syntax-highlight code by defining a grammar specification for the language (e.g in BNF form).

i have already made some integration code (to be uploaded here), but so far, in order to use the grammar parser and integrate with hljs core highlighter some mode boilerplate code (for example multiple modes inside contains and dummy ;lexemesRe, beginRe, endRe functions). While it would be easier (and more flexible) if there was some property that allowed a callback or even a static value with the lexeme (i,e token) to be directly available from the mode itself (and passed directly to highlighter to be wrapped in <span>[token value]</span> for highlight).

To be more explicit, consider the fragment from hljs highlight method below:

// ..
    var mode_buffer = '';
    var relevance = 0;
    try {
      var match, count, index = 0;
      while (true) {
        top.terminators.lastIndex = index;
        // it seems the only way is to override mode.terminators.exec
       // in order to hook here with the token
       // maybe accepting an already parsed value/callback would make all this more flexibe
      // e.g if ( mode.value ) count = processLexeme(value.substr(index, match.index - index), mode.value());
      // or sth like that
        match = top.terminators.exec(value);
        if (!match)
          break;
        count = processLexeme(value.substr(index, match.index - index), match[0]);
        index = match.index + count;
      }
      processLexeme(value.substr(index));
      for(current = top; current.parent; current = current.parent) { // close dangling modes
        if (current.className) {
          result += '</span>';
        }
      }
      return {
        relevance: relevance,
        value: result,
        language: name,
        top: top
      };
    } catch (e) {

// ..

Nikos

[foo123]

Anyway i've managed to integrate the grammar add-on with highlight.js
and going to update a repo shortly with the code for anyone interested (see screenshot below for xml parsing including highlighting errors, if defined, as unstyled code)

Not sure if and how an option for a mode to return a ready token to be highlighted directly has any merit or ease of implementation (judging from the hljs code, it would probably need changes in more than one place)

Plus i'm only able to make it work, using the un-packed highlight.js, it breaks when i use the packed highlight.pack.js file from the download package (without any languages, just the highlighter).
Can you help me resolve this one?

[foo123]

The code is on this repository https://github.com/foo123/highlightjs-grammar, for anyone interested

[joshgoebel]

Wouldn't you have more luck hooking into the keywords pipeline? And just dynamically adding keywords? I'm not sure I 100% follow what you're trying to do here. A more general (or perhaps more specific) example would be great.

If you want more than just keywords though (which are highlighted in the "gaps") then yeah you get into the end expression compiling and having to recompile those expressions on the fly while parsing, etc... and it seems from the second snap that you want to selectively NOT highlight tags based on whether they've been seen before or not...

Not a lot of fun.

Really don't rules/modes need some type of callback system really to do this sort of thing well? I haven't looked at any of your code, just read this issue. I think that would work much better than CHANGING the rules on the fly... just let them continue to match and then dynamically decide what you want to do about it.

[foo123]

The whole point of the gramnmar addon is that a user can specify a grammar for a language and then highlight based on that grammar. This is a very general approach. Have implemented the add-on for various syntax-highlighter available including highlight.js. The problem (issue is quite old and havent worked on it since then) is that I found no way of adding the parsed tokens into the flow of the highlight them in order to be highlighted. So I made sth of a little hack. I dont know if a new version exists that eases the burden of this. It is working as is and you can see the online interactive example available.

The problem is that highlight.js assumes from the start that some regexes will be used and is hardcoded into that mindframe disallowing more general parsers (that dont use simply a set of regular expressions) to be used (like -grammar addon does). So only way I found (for the version I am refereing) was to make a hack and override the regular expressions, while I think some modular system of taking a general callback (as an option, regexes can still be used, but if callback passed use the callback to retrieve the token) would be more general and modular.

[joshgoebel]

The problem is that highlight.js assumes from the start that some regexes will be used and is hardcoded into that mindframe disallowing more general parsers (that dont use simply a set of regular expressions) to be used (like -grammar addon does)

Well because Hightlight JS is an integrated unit. It's a parser-highlighter. It's not two separate pieces. You can't just highlight an arbitrary stream of things.

I'm interested in the grammars perhaps being more complex with callbacks and such things. It's be helpful for a very specific example of what you're trying to do, how you hacked it, and how you imagined it'd work in a perfect world.

You mention tokens but I'm not sure what tokens you mean.. an actual concrete example might be helpful.

[joshgoebel]

Does your project already have it's own parser that spits out tokens or an AST that you use with other things? IE if we exposed JUST the highlighter (and not the parser) would that be helpful?

[foo123]

Yes what I mean (like the code example posted on my first comments) is that highlight.js works only with a language that defines a set of regular expressions and then parses them on its own and highlights them. There is no way a custom parser (whatever that may be, a full-blown parser, other regexes, not following same rationale as highlight.js assumes) can be used and hjs take the result and highlight it. So initially I suggested for some way the parser can be made autonomous and maybe ovewritten as well. For example by allowing a custom callback which takes the current state as argument and returns the next token (string + class). That would be more modular and allow for the addon to work transparently instead of me making this ugly hack. The only way to understand is to check the code of the addon which makes the hack and decide for the best way to make it modular.

https://github.com/foo123/highlightjs-grammar/blob/master/src/main.js#L58

You see in the code of the addon, in order to make the integration and use the addon's parser I had to make custom functions that simulate regular expressions and pass this hacky object as the language definition in hjs (as that is what hjs assumes by default and no way of changing that). That is my point. If you can think of any way, for example a callback, can be used to de-couple the parsing from highlighting and make it more general and modular. The default way can still be used (eg if a language defines a set of regular expressions) but allow for overwriting of tokens by some extra parameter and if present delegate parsing to that method (a token is a unit of code that is highlighted by itself, eg an identifier). Hope this is clear.

[joshgoebel]

Well, I don't think it's a high priority but one could imagine in the future separate the two pieces... so you'd have the highlighter and the parser... so in your case you'd parse code HOWEVER you wanted and then you'd pass an AST and ask for it to be highlighted.

It sounds like you'd rather just take over the parsing process entirely.

Once you have an AST though is it really that hard to just generate the markup yourself? It seems that is actually the easy part... and you could just leverage our CSS themes for the "look"...

[foo123]

I have already solved the issue with a hack. It would be good (like some other highlighters do) to have some process which could be overwriten in some modular way and not be hardcoded.

My initial intention was to create the add-ons for as many syntax-highlighters I was aware of. The benefit is that people create only a single grammar and can use it throught all highlighters and editors of their choice without any modification (except the styling names, the rest remains the same). So that was the intention and that is why I added support for hjs. Doing so I noticed this issue and suggested some workaround.

I leave it up to you if you want to close this issue.
Cheers!

[joshgoebel]

Also I'm not sure you really answered me... don't you really just want an AST from the parser? Or are you doing things the parser isn't technically capable of? In that case it sounds like you'd want your own parser and JUST our highlighting engine. I'm not sure I completely understand the desire to turn our parser into a more general purpose parser.

[joshgoebel]

I don't think I understand your goals. Lets start with the stated end goal:

The benefit is that people create only a single grammar and can use it throught all highlighters and editors of their choice without any modification

Ok... but what you'd quickly find is that the parsers are NOT all equal at all. Our approach is VERY different than Prism.js approach, for example. Because they're a bit more minimalistic in what they choose to parse, for one - which results in a very different design of the parser- and limits what they can do vs what we can do, for example.

So I'd think it would be difficult to impossible to achieve "write once run anywhere" simply because all the parsers you work with are so very different (or not even full blown parsers at all, in the sense of the type of parsing toolchain you'd find in a full blown compiler, for example). Highlight.js and Prism.js are really very, very advanced tokenizers.

So I truly don't understand why you wouldn't pivot to writing your own parser that did EVERYTHING you wanted, and then just hook to the highlighters at the very end - or perhaps none of them are easily designed to do that? BUT...

Speaking for Highlight.js the highlighting part is trivial, vs the parsing (and grammars). If we were ever to split the parser/highlighter the highlighting (parse tree -> HTML) code is likely 20-30 lines... you're just looping over a nested tree and turning it into linear HTML.

So if your goal was to have the "Highlight.js look" or be compatible with our themes, all you need is your own parser and a tiny shim to convert to HTML. If you have your own grammar files you don't really want our parser as far as I can see.

So I guess I'm wondering why you didn't go that route as it would seem to be MUCH simpler than what you're trying to do.

[foo123]

All the *-grammar addons have a full-blown parser of their own (in fact the exact same parser engine, only the integration code changes). The integration with existing editors and highlighters is what is done. So the parser of the addon is used and is integrated in some way into the highlighter or editor. The user simply defines a grammar and can be used throughout all highlighters and editors of their choice for that language as is. This is the intention and how it works.

Of course I can create my own syntax highlighting framework (and use my ready made grammars and parsers) but that is another question. The issue here is integration with existing highlighters as an addon.

I am aware that some highlighters are narrowly implemented in that parsing and highlighting is a single process, strictly coupled and hardcoded usualy with regular expressions (which solve the parsing problem only partialy at best).

Consider the grammar addon integration as an addon for your own framework. It is only in that purpose that is made as an addon which integrates with your own framework as well as others so people can choose freely and use it with your own framework. A kind of kudos, if you like, for this framework. Nothing more, nothing less.

I only made a suggestion that maybe the procedure can de-couple the parser from the highlighter so that other parsers (for example the parser of the grammar addon) can be plugged in in some modular way (while still allowing for the default parser and behaviour to work as is).

I have already resolved this issue with a hack. I wish I did not have to hack it, but it doesnt matter. it works (packaged version does not work though, only unpackaged version of the framework). So if you like to dismiss this issue, it is fine with me.