Provide a way to populate field values from Markdown/MDX processing pipeline

[motleydev]

Feature:
I'd like to use an MDX plugin to create programmatic meta (variable declarations) that can then be exposed in the final object contentlayer provides.

Use case:
An example of here this would be useful is generating a TOC, creating a list of unique keywords, etc.

Work around:
Use computed fields (which don't expose the the raw AST)

Related issues: kentcdodds/mdx-bundler#169

Consider the following config structure

export default makeSource({
  contentDirPath: "content",
  documentTypes: [Posts, Things, People],
  mdx: {
    rehypePlugins: [rehypeSlug, rehypeAutolinkHeadings, searchMeta],
  },
});

Where searchMeta looks at paragraph nodes of mhast, grabs a list of unique words, and adds them to the metadata as searchMeta.

A markdown file with the structure of:

---
title: Hello World
slug: hello-world
---
Hello World! Please say Hello!

Would generate a final object of:

{
"title": "Hello World",
"slug": "hello-world",
"searchMeta": ["hello", "world", "please", "say"],
"code": "().....",
"_raw": "..."
}

For sake of complete, if not ugly code, here's a working example of the plugin that adds searchMeta to the data attribute of the vFile in the rehype plugin chain.

import { visit } from "unist-util-visit";

export default function searchMeta() {
  return (tree, file) => {
    visit(tree, { tagName: "p" }, (node) => {
      let words = node.children.reduce((collector, current) => {
        if (typeof current.value === "string") {
          let wordList = current.value
            .split(" ")
            .filter((word) => !word.includes(":"))
            .map((word) => word.toLowerCase().replace(/[^a-z0-9]/gi, ""))
            .filter((word) => word.length > 3);
          let newCollector = new Set([...wordList, ...collector]);
          return newCollector;
        } else {
          return collector;
        }
      }, new Set());

      file.data.searchMeta = [...words];
    });
  };
}

[timlrx]

As a temporary workaround, one could consider defining a computedField that parses the raw output from contentlayer. Here's an example of extracting the table of contents of a markdown file and making it available as a toc property in contentlayer

// Assume a remark plugin that stores the information in `vfile.data.toc`
export async function extractTocHeadings(markdown) {
  const vfile = await remark().use(remarkTocHeadings).process(markdown)
  return vfile.data.toc
} 

const computedFields: ComputedFields = {
  toc: { type: 'string', resolve: (doc) => extractTocHeadings(doc.body.raw) },
  ...
}

[schickling]

@motleydev would having access to the vFile.data property from within computedFields be a good solution to your described problem?

Something along those lines

const computedFields: ComputedFields = {
  toc: { type: 'string', resolve: (_doc, { vfile }) => vfile.data.toc },
  ...
}

[motleydev]

When do computed fields get executed? At run time or at compilation? At the end of the day, what I'm trying to get is the data added to static output.

[schickling]

computedFields are executed together with all other fields - therefore are part of your static output. (Just opened a docs issue to clarify this).

[motleydev]

in that case, that would probably work just fine! Would still be nice to do the work during the original transform process to not need to revisit each file, but for a static output process, that's probably shaving the yack a bit too close.

[motleydev]

The more I think about it, accessing vfile.data from computed fields would totally solve my use-case. It'd still be nice to be able to do all the work "in" the handler, but being able to do visit work during the initial parsing and then passing that along with the payload would be more than sufficient. What do you think a reasonable timeline on that would be?

[essential-randomness]

Any update on this? I'd be willing to try my hand at a PR to pass vfile as an additional argument of resolve in computedFields. I need the same thing!