toke.c: S_intuit_more: Add more commentary #23708
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
khwilliamson
force-pushed
the
intuit_more_commentary
branch
from
4c36ccf to
f0a5a44
Compare
khwilliamson
force-pushed
the
intuit_more_commentary
branch
2 times, most recently
from
d1fffd6 to
3407c5f
Compare
khwilliamson
force-pushed
the
intuit_more_commentary
branch
from
3407c5f to
c77f0b2
Compare
jkeenan
reviewed
Oct 2, 2025
| /* Examine each character in the construct */ | ||
| /* That this knows nothing of UTF-8 can lead to opposite results if the | ||
| * text is encoded in UTF-8 or not; another relic of the Unicode Bug. | ||
| * Suppose a string consistis of various un-repeated code points between |
There was a problem hiding this comment.
s/consistis/consists/ in above
| * (Some do it though as a mnemonic that is meaningful to | ||
| * them.) But generally, repeated characters makes things | ||
| * more likely to be a charclass. But we have here that | ||
| * this an identifier so likely a subscript. Its spelling |
There was a problem hiding this comment.
Sentence beginning with But lacks a verb, is confusing.
This function is described in its comments as 'terrifying', and by its original author, Larry Wall, as "truly awful". As a result, it has been mostly untouched since its introduction in 1993. That means it has not been updated as new language features have been added. As an example, it does not know about lexical variables, so the code it has for globals just doesn't work on the vast majority of modern day coding practices. Another example is it knows nothing of UTF-8, and as a result simply changing the input encoding from Latin1 to UTF-8 can result in its outcome being the opposite result. And it is buggy. An example of how hard this can be to get right is this fairly common use in our test suite: [$A-Z] That looks like a character class matching 27 characters. But wait, what if there exists a $A and a parameterless subroutine 'Z'. Then this could instead be an expression for a subcript. A few years ago, I set out to try to understand it. I added commentary and simplified some overly complicated expressions, but left its behavior unchanged. Now, I set out to make some changes, and found many more issues than I had earlier. This commit adds commentary about those. Hopefully this will lead to some discussion and a consensus on the way forward.
khwilliamson
force-pushed
the
intuit_more_commentary
branch
from
c77f0b2 to
6ef8978
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This function is described in its comments as 'terrifying', and by its original author, Larry Wall, as "truly awful". As a result, it has been mostly untouched since its introduction in 1993. That means it has not been updated as new language features have been added.
As an example, it does not know about lexical variables, so the code it has for globals just doesn't work on the vast majority of modern day coding practices.
Another example is it knows nothing of UTF-8, and as a result simply changing the input encoding from Latin1 to UTF-8 can result in its outcome being the opposite result.
And it is buggy.
A few years ago, I set out to try to understand it. I added commentary and simplified some overly complicated expressions, but left its behavior unchanged.
Now, I set out to make some changes, and found many more issues than I had earlier. This commit adds commentary about those. Hopefully this will lead to some discussion and a consensus on the way forward.