Skip to content

gh-135676: Simplify docs on lexing names #140464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

gh-135676: Simplify docs on lexing names #140464

wants to merge 4 commits into from
+86 −58

Conversation

encukou
Copy link
Member

@encukou encukou commented Oct 22, 2025 •
edited by github-actions bot
Loading

This simplifies the Lexical Analysis section on Names (but keeps it technically correct) by putting all the info about non-ASCII characters in a separate (and very technical) section.

It uses a mental model where the parser doesn't handle Unicode complexity “immediately”, but:

  1. parses any non-ASCII character (outside strings/comments) as part of a name, since these can't (yet) be e.g. operators
  2. normalizes the name
  3. validates the name, using the id_start/id_continue sets (referred to in previous sections as “letter-like” and “number-like” characters, with a link to the details)

This also means we don't need xid_start/xid_continue to define the behaviour :)

📚 Documentation preview 📚: https://cpython-previews--140464.org.readthedocs.build/

@bedevere-app bedevere-app bot added docs Documentation in the Doc dir skip news labels Oct 22, 2025
@github-project-automation github-project-automation bot moved this to Todo in Docs PRs Oct 22, 2025
@bedevere-app bedevere-app bot mentioned this pull request Oct 22, 2025
Open
@StanFromIreland StanFromIreland linked an issue Oct 22, 2025 that may be closed by this pull request
Docs: note requirement to normalise unicode identifiers passed to globals() and locals() #86846
Open
willingc
willingc approved these changes Oct 22, 2025
View reviewed changes
Copy link
Contributor

@willingc willingc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Outstanding document @encukou. I had one small suggestion to be a bit more explicit on the normalization example with number.

Doc/reference/lexical_analysis.rst
This means that, for example, some typographic variants of characters are
converted to their "basic" form, for example::

>>> nᵘₘᵇₑʳ = 3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be helpful to add an explicit comment that the normalized form of nᵘₘᵇₑʳis number.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@willingc willingc willingc approved these changes

@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Assignees

No one assigned

Labels

awaiting merge docs Documentation in the Doc dir skip news

Projects

Docs PRs
Status: Todo

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Docs: note requirement to normalise unicode identifiers passed to globals() and locals()

2 participants

@encukou @willingc