Tokenize punctuation

[MaximSokolov]

Fixes #103

int main() { printf("Hello, world!"); }
{}[]()
;,.

[winstliu]

language-javascript uses punctuation.terminator.statement for this.

[winstliu]

What do you think about the differences between this new matching and the one used by language-javascript/other languages?

[MaximSokolov]

1. meta is used to provide context, so punctuation for punctuation is ok.
2. bracket.{round|square|curly} is correct as the term "braces" is for {,}
3. I guess separator (special case of "delimiter") is more correct for commas. Not sure about delimiter for the "."

["a", "b", "c" ] // separator
["a", "b", "c",] // delimiter

1. I use punctuation.other since punctuation.separator/definition/delimiter would be unclear without information about what that punctuation mark separates/defines/delimits (e.g. punctuation.definition.array) Maybe drop other?

[winstliu]

bracket.{round|square|curly} is correct as the term "braces" is for {,}

This will unfortunately break standardization with other packages. Might not be that big of a deal though depending on how quickly the others are updated and how many themes are using those. @simurai?

I do agree with moving away from meta though.

[simurai]

From the core themes only Solarized seems to style .brace.

Then probably some forks and people's styles.less overrides. So not too big impact, but still a few.

[MaximSokolov]

Only JS, Coffescript, Less tokeize brackets as meta.brace (Search results for "meta.brace" in grammar files) Also isBracket function uses meta.brace (search results for "isBracket") However this function doesn't work properly, since most brackets are tokenized as punctuation.definition.<section>

[MaximSokolov]

Changes in 0d8b634:

before:

int fn(...) {...}
   |  ^   ^     |  - punctuation.section.parens.begin/end
   |  ^---^     |  - meta.parens
   ^------------^  - meta.function

{ fn(...) }
 |  ^   |    - punctuation.definition.parameters
 |  |   ^    - no scope
 |  ^---^    - no scope
 ^--^        - meta.function-call

after:

int fn(...) {...}
   |  ^   ^     - punctuation.definition.parameters.begin/end
   |  ^---^     - meta.parameters
   ^------^     - meta.function

{ fn(...) }
 |  ^   ^    - punctuation.definition.arguments.begin/end
 |  ^---^    - meta.arguments
 ^------^    - meta.function-call

Fixes #100

#define ABC XYZ(1)
struct MY_STRUCT {
/* code */
};

Fixes #31

#define EOFCHECK() \
   if(is.eof()) \
    throw std::runtime_error(((std::string) "Setting '") + name + "' not found");

Foo foo(getSomeString() + " (" + otherString + ')');

Fixes #75

#define a >b()
int c;
int d;

Fixes #80

{
    ::printf("Error: MongoDB Invalid Query %s!\n", query ? query->toString().c_str() : "{}");
}

Fixes #82

int connect (const char *address,
            void (*data_recevied)(const char *, const int len)
            void (*on_connect)(),
            void (*on_disconnect)());

Fixes #111

#define LOG1(sev) \
  LOG2(sev) << "#" << " smth "

// everything further is highlighted as string literal
int f(void) {
// smth
}

Fixes this: atom/language-gfm#133 (comment)

[winstliu]

While we're improving punctuation, let's remove the unnecessary backslashes on curly brackets as well.

[MaximSokolov]

Would this be better suited as a contentName? Same for below.

It can be. This allows to style parens inside arguments though:

.meta.arguments {
  & > .punctuation.definition {...}
  // .variable.argument {...}, etc
}
// instead of
.punctuation.definition.arguments {...}
//.variable.argument {...}

[winstliu]

Ok, let's keep it as is then.

[winstliu]

Missing .bracket.round on these two

[winstliu]

Regarding ;:
I just looked and there's absolutely no standardization as to what this is scoped as.
Javascript has it as punctuation.terminator.statement, Java and Go use punctuation.terminator, Ruby uses punctuation.terminator.expression, and Less uses punctuation.terminator.rule.

We should agree on a common scope for this before continuing.

[MaximSokolov]

I've fixed unmatched brackets issue by removing begin/end capturing of preprocessor rules.
Everything seems to be working. The only regression:

#ifdef 1 // #ifdef, #ifndef, etc.
#endif
#endif // would be matched as well

This also makes possible to tag unmatched brackets as illegal:

Regarding ;

I think punctuation.terminator.statement is correct for imperative programming languages. Not sure about using punctuation.terminator.statement for CSS, Less, etc

[winstliu]

I think that regression is fine for now. Languages don't have to also be full-blown linters (though it's definitely nice when we can tokenize something as invalid) :P.

[winstliu]

Don't fix this in this PR, but this capture is missing a .cpp at the end.

[winstliu]

Just making sure...these three will still be matched by function_params, right?