Regex look-ahead expressions do not work properly (end matchers)

[joshgoebel]

This is an extension of:
#1696

Currently end-matchers still don't work with look-ahead. You can hack it using the |$ hack (and if your rules aren't too complex this actually will work really well), but it's not fully supported for the same reason that begin didn't use to work.

The end rules are still "replayed" after a match to figure out if it was a REAL match. There are several reasons it might not be, such as sameAsBegin and things that CHANGE the matching criteria while the parser runs... and the "master" consolidated regex is never recompiled to account for these changes. So you could have a "potential" match that actually doesn't match begin... this happens mostly commonly with things like HEREDOCs.

End rules are only replayed against the match text itself... which won't include the look-ahead bits as technically they aren't part of the match... so the parser will stop with a match, but then fail to "replay" it because the look-ahead now sees only the end of the match string, instead of the full text. This is why the |$ trick works... unless you have a very complex ruleset end rules usually aren't competing with other end rules (except parents)... so if the look-ahead matched the first time but now it only matches $, that's 99% of the time still the right match.

Begin (because of the much larger set of rules) had a much higher chance of doing the "wrong" thing.

Still, this is something we should look into fixing as time allows.

[joshgoebel]

From the other thread: #1349

Example:
{ begin: /\w+/, end: /(?=\s*)/, }

Seems a useful enough case for a test case, although word boundary would technically also work in this case...