Treesitter features for non existent (anonymous) grammar/static AST

[daniilrozanov]

Problem

Background here #35855

Sometimes plugin authors faced with a problem to create some kind of control panel or just pretty render some structured data. For neovim there are:

• neogit renders each its buffer to user can interact with git
• gh.nvim for you to see issues and PRs inside nvim's buffer
• some other plugins that essentially are UI clients for something, which use non modifiable buffer to pretty print things on it

In all these cases if author wants highlight some things or make them foldable, one needs to reimplement all basic treesitter's logic related to a tree bypass, creation of custom nodes, etc. For example, neogit has a custom UI implementation, and it takes a lot of work to implement the already existing features that provide the neovim api + treesitter.

Root of that problem is there is no way to manually create a TSTree and TSNode with its own type, position, etc. to use it as a language. This is because treesitter have no API for that. It allows to create a language using const TSLanguage *tree_sitter_some_language(void);, which is always generated by treesitter.

Expected behavior

Currently I see 3 solutions. I am ready to implement the feature in both neovim and treesitter if necessary (see below).

New types of Node, Tree, LanguageTree in Lua level only

This idea relies on what @vanaigr suggested. Create Node, Tree and LanguageTree lua objects the minimum required functionality to mimic treesitter's ones. Then extend existing neovim lua API to support these objects.

The pros are that it's relatively simple to do. Problem is what exactly is a "minimum required functionality"? If you look at TSNode's API, there is a lot of functions and I can't say for sure which ones are essential and which ones can be ignored. This also applies to TSLanguage. There is a lot of contexts where we use treesitter, so if we want to support all of that we need to reimplement a lot of existing treesitter's logic in lua.

Note: the purpose for which I need this feature is, as I wrote above, to create a UI in an unmodifiable buffer with highlight and folding like neogit does. That is all. Therefore, I personally know my "minimum required functionality". But this does not mean that only this functionality should be implemented, as the new API should be general and cover many cases. This is the place where I need advice.

Make treesitter support this case and integrate its new API to neovim core

This way needs treesitter's team approval. Point is to allow create special TSLanguage object that holds custom tree. It created that way so TSParser just translate custom tree to TSTree rather than actually parse anything. Here's about what needs to be added:

// NEW: Node type that can be created via lua
typedef struct TSManualNode {
TSNode node;
bool visibility;
bool named;
/*sometype*/  children;
// what else?
} TSManualNode;

// NEW: returns special language. it is allocated on the heap, whereas it is usually a static object.
const TSLanguage *tree_sitter_manual_language(TSManualNode* root);

// CHANGED (?): parser now just makes TSManualNode -> TSTree 
ts_parser_set_language(parser, tree_sitter_manual_language(root));

// Also adopt some functions to new behaviour
// And add some utility functions

This is my preferred way since we don't need to duplicate anything and we get all the existing features of treesitter for free. Even after some new treesitter's module arrive, we immediately receive it.

The problem is that first of all it's most difficult way (at least for me). Second, this feature may be a fifth wheel for treesitter, since there probably will be only one consumer of that (neovim). But maybe TS team will be ok, IDK.

Write a plugin 🫤

The most prosaic way. Neovim already does folding and highlighting. Treesitter just helps do it right for programming languages. Not for custom UI buffers. So if we make things strictly through guidelines, maybe this is the best choice, and this issue should be closed as not planned.

However, , if I write such a plugin, 90% of its code will be a duplication of treesitter functionality. And when the neovim treesitter api has new features, I'll have to duplicate them again. So maybe one should compromise he's principles:)

---

Anyway, sorry for that TLDR. What do you think about that? What is the best way?

[justinmk]

If this requires complex changes or hard-to-support APIs, it's going to be a slow process, since we need to wait and hear from people that actually want this. If the changes are simple or don't add a big maintenance cost, then we can move more quickly.

From your proposed solutions: "Make treesitter support this case" is the most preferred. "New types of Node, Tree, LanguageTree" sounds pretty horrible. But I could be wrong, waiting to hear from people that know this area deeply.