Parser support for ?[

[johnniwinther]

Parser support for null aware index access without '.':

e?[e] and e?[e] = e in addition to e?.[e] and e?.[e] = e

[eernstg]

Actually, @mit-mit agreed that we do not have to support the variant with the ..

@lrhn, @munificent, @leafpetersen, do you agree that we can simply stop supporting v?.[0] and such?

[lrhn]

There are a few uses. A grep shows:

../sdk_nnbd/lib/_http/websocket.dart:    String? part = requested?.parameters?.[_serverMaxWindowBits];
../sdk_nnbd/lib/core/uri.dart:    String? charsetName = parameters?.["charset"];

(And then there are all the uses in the tests (pkg/analyzer, pkg/_fe_analyzer_shared, tests/language/nnbd).

If we change all of those at the same time that we land the ?[ syntax, we should be fine.

[jensjoha]

As I understand these are also new and has to be supported:

  // Method calls.
  a?<B>(c);
  a?(b);

[eernstg]

As I understand these are also new and has to be supported

True. I was just making the point that there is no need to support the form that includes the period for the index operator, so we can eliminate the code that does this, or omit implementing it if there is no such code. I believe that f?.() was never implemented, and then we just won't do that. As a result, the forms with no period are supported, and it's as if the forms with a period never existed.

[munificent]

As I understand these are also new and has to be supported:

  // Method calls.
  a?<B>(c);
  a?(b);

I could be wrong, but I don't believe we have decided to add null-aware call operators. Instead, for this case (at least for now), we're OK with users doing a?.call<B>(c) and a?.call(b).

[jensjoha]

As I understand these are also new and has to be supported:

  // Method calls.
  a?<B>(c);
  a?(b);

I could be wrong, but I don't believe we have decided to add null-aware call operators. Instead, for this case (at least for now), we're OK with users doing a?.call<B>(c) and a?.call(b).

I'll hold off that change then.

Generally I also think that these changes makes the parser slower.
When seeing a ? (not .? or .?[) before this change, we immediately knew that it was a conditional expression and could parse away happily.
With the e.?[e] => e?[e] change we now have to skip the expression whenever seeing a ? followed by a [ to figure out whether it's a conditional or an index access.
If also adding the call change, when seeing a ? followed by < or ( we would have to try to parse and expression and an argument list at the same time in an attempt to figure or what case we're in.

[eernstg]

@munificent wrote:

we're OK with users doing a?.call<B>(c) and a?.call(b).

Right, checking again: the language team meeting document actually doesn't commit to the null-aware function calls. In any case, it's good that we get to assess the difficulties and parser performance issues in smaller bites. The ambiguity does introduce a need to have some kind of back-tracking, so we will surely be able to write a pathological program that causes a timeout or stack overflow during parsing, and that is something that we don't want to have too frequently.

[munificent]

Generally I also think that these changes makes the parser slower.
When seeing a ? (not .? or .?[) before this change, we immediately knew that it was a conditional expression and could parse away happily.

Yeah, it was expected that this increases parsing complexity. However, users felt very strongly that ?[ was the better syntax.

[eernstg]

Summary of a discussion taken elsewhere:

The grammar rule for null-aware index operator invocations (plus invocations of the corresponding setter) like var z = x?[y]; (respectively x?[y] = z;) specifies that the ? and the [ are separate tokens.

This means that it makes no difference if some whitespace is added between the '?' and the '['. In particular var z = x? /* Some Comment */ [y]; works just as well as var z = x?[y];.

It may seem tempting to say "let's parse ?[ as a single token, that allows us to disambiguate immediately": When there is no space between '?' and '[' it must be a null-aware index, when there is a space it must be a conditional expression.

However, we did not specify it in this manner, because this would imply that we use whitespace as a source of disambiguation. So you have two syntactically correct Dart snippets, P1 and P2, that only differ in one spot: You can produce P2 by taking P1 and adding one space. If P1 and P2 have a different meaning (actually: they would parse into different abstract syntax trees), then we would be using whitespace to distinguish two different pieces of code. We preferred to aim for a situation where adding a space makes no difference, or it turns the snippet into a syntax error, but it doesn't give the snippet a new meaning.

So that's the reason why those two characters are tokenized as ? [ and not as ?[.

It may still be a bad choice to write var z = x? /* A Really Long Comment */ [y];, but we can handle that via the linter or a similar mechanism, if it actually turns out to be a problem.

[lrhn]

For the record, I'm not opposed to making ?[ have to be consecutive characters when interpreted as a null-aware index, as long as x?[1]:2 is still parsed as a conditional expression even without the space.

I'm not sure it makes a big difference in practice. Parsing may be faster for correctly formatted conditional expressions where the ? and [ are separated, but it won't be any faster when seeing ?[ together, so there is no change to the worst-case performance. If you think it is actually an advantage, the we can consider it.

[bwilkerson]

It may seem tempting to say "let's parse ?[ as a single token, that allows us to disambiguate immediately": When there is no space between '?' and '[' it must be a null-aware index, when there is a space it must be a conditional expression.

For the record, that's not what I was proposing (apologies if that wasn't clear). I was proposing that ? [ is never a null-aware index but that ?[ could be either depending on lookahead. As Erik pointed out in the other thread, it's the same treatment that we give to >>, which can either be a single operator or two separate > tokens, such as in a type argument list.

Unlike the either-or scenario you outlined, I don't believe that my proposal uses whitespace to disambiguate. Given a valid conditional expression c?[1]:[2], you get the same valid expression by adding whitespace c? [1]:[2]. Given a valid null-aware indexing expression a?[b] you get invalid code by adding whitespace a? [b]. Whitespace never changes how the code is interpreted, only whether it's valid.

If you think it is actually an advantage, the we can consider it.

It seems inconsistent to me to allow a space in this operator when we don't allow spaces in any other operators (unless the operator requires non-whitespace tokens inside it, such as [] and ?:). I think that inconsistency would be confusing to users. The comment concern illustrates the point. I expect that nobody would be surprised or confused to see a?[ /* comment */ b], but that most users would be surprised and confused to see a? /* comment */ [b]. As a result, I think it would be better for users if we did not allow a space.

By the way, how are we handling an ambiguity like

var y = { x ? [1] : [2] };

Is that a set or a map? Or is that just illegal?

If it's illegal, I'll note that we could make it valid if we didn't allow spaces between the ? and [ for the null-aware index operator because then it could only be interpreted as a set containing a single value that is the result of a conditional operator. There is, of course, another way to disambiguate the expression (by adding type arguments to the collection), so it isn't necessary to allow another disambiguation technique, but it might be nice.

[lrhn]

OK, let's think about this at the language team meeting today. Seems reasonable to me.

For

  var y = { x ? [1] : [2] };

it is a set literal containing a conditional expression, and so is

  var y = { x?[1]:[2] };

Because x?[1]:[2] can be parsed as a conditional expression, it will not parse x?[ as a null-aware index operator.