toke.c: Some tidying #23797
Merged
toke.c: Some tidying #23797
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change it from being an array length, to being a pointer to the ending position of the array. This makes this function consistent with most of the others in this file. It allows us to usually use C_ARRAY_END() to wrap the parameter, which simplifies some expressions,
Change it from being an array length, to being a pointer to the ending position of the array. This makes this function consistent with most of the others in this file. It allows us to usually use C_ARRAY_END() to wrap the parameter, which simplifies some expressions,
The length of a constant literal is STRLENs The end position of a C array is C_ARRAY_END The number of elements in a C array is C_ARRAY_LENGTH
'uni' commonly is short for Unicode. Here, it was short for 'unary', which I found highly confusing. This also changes the name of a formal parameter to also not be 'uni' when 'unary' is meant.
All calls to it set it to TRUE or FALSE
isSPACE only matches single-byte characters; no need to be concerned with UTF-8ness
By using the output of the first macro, and skipping ahead in the parse string, we avoid reparsing the same bytes.
It is a bit simpler
tonycoz
reviewed
Oct 7, 2025
tonycoz
approved these changes
Oct 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changes:
To use mnemonic macros instead of sizeof()
To declare the correct return type from strlen()
To change the confusing names of a static function and formal parameter
To change a a static function parameter that is boolean-valued to bool, not I32
To change two static functions parameters to point to string-end, not length, simplifying some expressions.
To change a complicated for loop that dealt with non-existent UTF-8ness to a simpler while
To change some other for loops to simpler while or do while ones.