[plugin] Idea: Support YAML front matter in Markdown files

[gregives]

It's common to include YAML front matter at the top of a Markdown file, for example when using Jekyll. Currently, highlight.js parses the last line of YAML as a second-level heading because of the --- three dashes below it. Although YAML front matter isn't actually part of any Markdown specification, would it be possible to add this to the Markdown definition?

GitHub highlights this correctly:

---
title: Example Title
date: 2020-02-05
---

Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Highlight.js highlights this incorrectly:

[joshgoebel]

It's common to include YAML front matter at the top of a Markdown file

But that doesn't make the content Markdown. We also can't differential between the top of a file and the middle of a file, so contextual things like this are a bit outside our grasp.

GitHub highlights this correctly:

Does it, really? All it does it color the date, no matter how it's included, YAML or just inline text... I don't think it has anything to do with the YAML.

---

I think this would be better solved with a plugin designed specifically for YAML + Markdown. Take a look at: https://github.com/highlightjs/highlight.js/blob/master/docs/plugin-api.rst

You'd detect if the file was YAML + markdown BEFORE highlighting, then split it yourself, highlight both chunks separate then paste them back together and return that as the result. I think that would be possible with the new plugin support. All the glue might not be there, but if you wanted to work on a plug-in I'd look into the glue.

I'd guess 20-30 lines of Javascript maybe.

[gregives]

But that doesn't make the content Markdown. We also can't differential between the top of a file and the middle of a file, so contextual things like this are a bit outside our grasp.

Okay, I thought that might be the case.

Does it, really? All it does it color the date, no matter how it's included, YAML or just inline text... I don't think it has anything to do with the YAML.

I suppose I meant that GitHub doesn't seem to highlight it incorrectly, for that small example I posted.

---

I wasn't aware there was plugin support for highlight.js! I will take a look at writing a plugin for this. I've just had a quick look at the docs and noticed that before:highlightBlock doesn't have a return value: how would I return the result as you mention?

[joshgoebel]

I suppose I meant that GitHub doesn't seem to highlight it incorrectly, for that small example I posted.

Now that would be a better question. That might be fixable. Does a header require a blank line before it? We could check for that but then the problem becomes I don't think JS has \A and \Z so there is now way to say "blank line OR start of file"... since many times the very first line is a header. So I think you can have it one way or the other, but not both.

@allejo Does that sound right to you?

Thoughts?

I've just had a quick look at the docs and noticed that before:highlightBlock doesn't have a return value: how would I return the result as you mention?

Ok, so you can't coop the process completely currently, but you can still do it. You need to hook before and after. You can effect changes with after simply by modifying the object it was passed. Before just needs to store the original text in the object state so it can parse it when after is called.

[allejo]

Now that would be a better question. That might be fixable. Does a header require a blank line before it? We could check for that but then the problem becomes I don't think JS has \A and \Z so there is now way to say "blank line OR start of file"... since many times the very first line is a header. So I think you can have it one way or the other, but not both.

@allejo Does that sound right to you?

Thoughts?

My first thought would be to introduce front matter as a new language definition instead of trying to introduce complexity to the markdown syntax by detecting headers or the start of files. My thought would be to allow modifiers/combinations to languages. e.g.

• frontmatter--md
• frontmatter--html
• frontmatter--jsx

Depending on the tool using the front matter files, the body of it can be in a number of languages which is why I would be hesitant to introduce it just to markdown.

[joshgoebel]

Thanks for helping me clarify why this rubbed me the wrong way. This (front matter) is really more of a "concept" than a "language" itself. It's a convention I've seen used with blogging software but I assume it's also used elsewhere since it's kind of a useful pattern.

Also front matter doesn't necessarily have to be limited to YAML either... really if we had configurable grammars you could do this with something like:

// you could alias it however you wanted, I just used "markdown" here to overwrite markdown, as per the original request - if say that was the ONLY context you were using HLJS
registerLanguage("markdown", frontmatter("yaml",{ content: "html" } )

As I said before though I don't think you could do this purely with a grammar now though - I think you could do it with a grammar + plugin though. The grammar I think would be really light and would lean on the plugin for doing the actual work (so it can run parse code, split the content, etc). Honestly I see grammar + plugin as a way to do all sorts of crazy complex behavior that you can't do inside grammars themselves.

Is this the best thing long-term, I dunno... It probably needs some thought so we can come up with a common pattern for 3rd party grammars who want to do this. Actually I think this might be a good way to START things and then eventually you simply fold the plugins into the grammar itself... so a grammar would have a beforeHightlight and afterHighlight key... and they would function the same as plugins - but grammar specific plugins.

That would require adding hooks for highlight also, but that's on the TODO list anyways.

[joshgoebel]

@gregives It'd be quite cool if you build a plugin for this. That'd probably motivate me to go in and actually add support for highlight also not just highlightBlock... Then I could walk you thru publishing it as a 3rd party plugin/grammar if you thought it'd be useful for other people and we could test out this whole idea. :-)

[joshgoebel]

Something like:

#2395

(very raw)

I need to think about how this works in light of autoHighlight (which would call your plugin a zillion times) - but perhaps that is a problem for plugin authors?

And also need to think about sublanguage and how this handles continuations... that's really why I didn't tackle it at first and went with the much simpler highlightBLock instead.

[gregives]

Thanks for the great conversation around this! So at the moment, are we thinking a plugin is the best way forward for now? I can take a look at a plugin this weekend if so.

Out of interest, how does highlighting work for other 'nested' languages? For example, does highlight.js use the JavaScript definition for highlighting JavaScript within some HTML? Or is it duplicated within the HTML definition?

[joshgoebel]

@gregives Do you see how you might go about this? Define a dummy grammar and then some flag you store there enables the plug-in...

So given the example earlier:

registerLanguage("markdown", frontmatter("yaml",{ content: "html" } )

Perhaps frontmatter would return something like:

return {
  name: "markdown with yaml front matter",
  frontmatterIs: "yaml",
  bodyLanguage: "markdown",
  usePlugin: "frontmatter",
  // and perhaps long-term
  "before:highlight" : function () {},
  "after:highlight" : function () {},
}

When your plugin runs then it's looking for the usePlugin == 'frontmatter' to know it should do it's own magic.

[joshgoebel]

So at the moment, are we thinking a plugin is the best way forward for now?

Well a plugin that could also be/provide/generate it's own grammar. :-) I think over time the line will blur, as mentioned above. We're playing with the future here. :-) I kind of showed you how you might go about it in the previous message.

Out of interest, how does highlighting work for other 'nested' languages? For example, does highlight.js use the JavaScript definition for highlighting JavaScript within some HTML? Or is it duplicated within the HTML definition?

Both, but often we do it right and use sublanguage which tells Highlight.js to switch modes in the middle of a file and process part of it with a whole other grammar. It's pretty complex and one of the few pieces I still don't completely understand in and out. It's also not perfect and doesn't work for all cases (see open issues).

And theoretically you could use sublanguage here IF you could properly detect the front matter at the beginning of the file... that'd probably be the "correct" way to do it... but per discussion above I don't think we'd do this by changing the markdown grammar, it would be another grammar entirely or an auto-generated grammar.

If JS supported \A this would be trivial to do with just the grammar stuff we already support.

[joshgoebel]

  // and perhaps long-term
  "before:highlight" : function () {},
  "after:highlight" : function () {},

Actually one could add this type functionality via plugin even, without even changing core. LOL.

[allejo]

I agree that this should be a plugin. My main reason is because this concept can be expanded to highlighting embedded any language without complicating or bloating a language grammar. Say we want to highlight code in markdown blocks just like GitHub does:

# Hello World

I'm a markdown document and want to show some embedded HTML:

```html
<div>
  <p data-testid="highlightjs">Hi from HTML!</p>
</div>
```

Unless this is already possible?

[joshgoebel]

Well, I actually think that's a whole different case - and not comparable to the original example, but I have some very mixed feelings on that specifically. I think GitHub is only confusing the issue because it's actually RENDERING markdown, not highlighting it.

To me highlighting markdown would mean that you'd have markup like:

<div class="hljs-string">
```html
&lt;div&gt;
  &lt;p data-testid="highlightjs"&gt;Hi from HTML!&lt;/p&gt;
&lt;/div&gt;
```
</div>

Within the context of highlighting markdown that is really just a multi-line string... now when the markdown is actually rendered that string might be highlighted as code... but I see that as a job of the renderer... and we are not a renderer, we're a highlighter.

This is also related to how some people expect us to handle markdown... some people expect us to actually have styling that makes bold parts bold and italic parts italic... but that's not our job. We're a highlighter, NOT a renderer.

Even thinking about it makes my head hurt a little. :) Maybe I'm thinking about it wrong though - I know my text editor syntax highlighting does colorize markdown code snippets...

Unless this is already possible?

You can't do it dynamically, but you could "guess" or trust the auto-detect... look at how XML handles <script>, for example:

        begin: '<script(?=\\s|>)', end: '>',
        keywords: {name: 'script'},
        contains: [TAG_INTERNALS],
        starts: {
          end: '\<\/script\>', returnEnd: true,
          subLanguage: ['actionscript', 'javascript', 'handlebars', 'xml']
        }

[joshgoebel]

Or see http, which throws the whole "body" to auto-detect and lets it figure it out:

{
  begin: '\\n\\n',
  starts: {subLanguage: [], endsWithParent: true}
}