Disable automatic colon insertion before function body

[mjambon]

This is something I noticed while testing the fix for #93.

The following code is parsed correctly by tree-sitter-javascript parser but incorrectly by tree-sitter-typescript:

function f()
{}

tree-sitter-typescript sees a function_signature followed by a statement_block:

(program [0, 0] - [2, 0]
  (function_signature [0, 0] - [0, 12]
    name: (identifier [0, 9] - [0, 10])
    parameters: (formal_parameters [0, 10] - [0, 12]))
  (statement_block [1, 0] - [1, 2]))

However, the tree-sitter-javascript implementation works fine and gives us a single function_declaration:

(program [0, 0] - [2, 0]
  (function_declaration [0, 0] - [1, 2]
    name: (identifier [0, 9] - [0, 10])
    parameters: (formal_parameters [0, 10] - [0, 12])
    body: (statement_block [1, 0] - [1, 2])))

[maxbrunsfeld]

Yeah, this fix will be slightly more complex than the one for : tokens, because an automatic semicolon can sometimes appear before a {, but not in this specific state.

The external scanner's scan function gets passed an array of booleans that tell it which tokens are currently valid. So we'll have to figure out a way of differentiating between the two cases based on the validity of a certain token.

You can add any token to the externals array in order to have that token's validity passed to the external scanner as an additional boolean. We have already done something like this here.

[maxbrunsfeld]

One idea: maybe we should add : to that array, so that the external scanner will be able to tell if a type annotation (e.g. : T) is currently valid. Then the external scanner could avoid creating an automatic semicolon if a type annotation is valid in the current state.

Edit: Actually, that probably won't work, because in some states where we do want an automatic semicolon, a : might be a valid lookahead for other reasons, like "labelled statements".

[resolritter]

Adding more context to this which might already be obvious for the participants, but it was unclear to me.

Going by the Declaration Reference and this snippet in the TypeScript playground, function f() is only considered a signature declaration inside of namespace declarations (declare namespace ...). Outside of them, the statement needs to start with declare in order to be considered a declaration; this is true even for top-level signatures on declaration files (.d.ts).

Is it fair to state that this ambiguity would be solved if there's a check for: "the statement is inside of a namespace declaration's block scope"? Or are there more contexts where function f() is read as a signature without the declare keyword upfront?

If looking at the scope is enough, then I can see some state (like the tags state) working, only it'd be namespace_scope_depth (since namespaces can nest arbitrarily inside of namespaces). Checking if namespace_scope_depth == 0 would disambiguate between declaration and signature.