[PEP 747] Recognize TypeForm[T] type and values (#9773)

[davidfstr]

(This PR replaces an earlier draft of the same feature: #18690 )

Feedback from @JukkaL integrated since the last PR, by commit title:

• Apply feedback: Change MAYBE_UNRECOGNIZED_STR_TYPEFORM from unaccompanied note to standalone error
• Apply feedback: Refactor extract save/restore of SemanticAnalyzer state to a new context manager
• Apply feedback: Suppress SyntaxWarnings when parsing strings as types at the most-targeted location
• Apply feedback: Add TypeForm profiling counters to SemanticAnalyzer and the --dump-build-stats option
  • Increase efficiency of quick rejection heuristic from 85.8% -> 99.6% in SemanticAnalyzer.try_parse_as_type_expression()
• Apply feedback: Recognize assignment to union of TypeForm with non-TypeForm
• Apply feedback: Alter primitives.pyi fixture rather than tuple.pyi and dict.pyi

Feedback NOT integrated, with rationale:

• ✖️ Add tests related to recursive types
  • Recursive cases are already well-covered by tests related to TypeType (is_type_form=False).
  • I did find an infinite recursion bug affecting garden-variety Type[...], which I can fix in a separate PR.
• ✖️ Define TypeForm(...) in value contexts as a regular function like Callable[[TypeForm[T]], TypeForm[T]] rather than as a special expression node (TypeFormExpr).
  • The special expression node allows mypy to print out better error messages when a user puts an invalid type expression inside TypeForm(...). See case 4 of testTypeFormExpression in check-typeform.test

There is one NOMERGE commit temporarily in this PR so that mypy_primer gives more insightful CI output:

• NOMERGE: mypy_primer: Enable --enable-incomplete-feature=TypeForm when checking open source code

There is one commit unrelated to the core function of this PR that could be split to a separate PR:

• Allow TypeAlias and PlaceholderNode to be stringified/printed

Closes #9773

---

(Most of the following description is copied from the original PR, except for the text in bold)

Implements the TypeForm PEP 747, as an opt-in feature enabled by the CLI flag --enable-incomplete-feature=TypeForm.

Implementation approach:

• The TypeForm[T] is represented as a type using the existing TypeType class, with an is_type_form=True constructor parameter. Type[C] continues to be represented using TypeType, but with is_type_form=False (the default).

• Recognizing a type expression literal such as int | str requires parsing an Expression as a type expression. Only the SemanticAnalyzer pass has the ability to parse arbitrary type expressions (including stringified annotations), using SemanticAnalyzer.expr_to_analyzed_type(). (I've extended the TypeChecker pass to parse all kinds of type expressions except stringified annotations, using the new TypeCheckerAsSemanticAnalyzer adapter.)

• Therefore during the SemanticAnalyzer pass, at certain syntactic locations (i.e. assignment r-values, callable arguments, returned expressions), the analyzer tries to parse the Expression it is looking at using try_parse_as_type_expression() - a new function - and stores the result (a Type) in {IndexExpr, OpExpr, StrExpr}.as_type - a new attribute.

• During the later TypeChecker pass, when looking at an Expression to determine its type, if the expression is in a type context that expects some kind of TypeForm[...] and the expression was successfully parsed as a type expression by the earlier SemanticAnalyzer pass (or can be parsed as a type expression immediately during the type checker pass), the expression will be given the type TypeForm[expr.as_type] rather than using the regular type inference rules for a value expression.

• Key relationships between TypeForm[T], Type[C], and object types are defined in the visitors powering is_subtype, join_types, and meet_types.

• The TypeForm(T) expression is recognized as a TypeFormExpr and has the return type TypeForm[T].

• The new test suite in check-typeform.test is a good reference to the expected behaviors for operations that interact with TypeForm in some way.

Controversial parts of this PR, in @davidfstr 's opinion:

• Type form literals containing stringified annotations are only recognized in certain syntactic locations (and not ALL possible locations). Namely they are recognized as (1) assignment r-values, (2) callable expression arguments, and (3) as returned expressions, but nowhere else. For example they aren't recognized in expressions like dict_with_typx_keys[int | str]. Attempting to use stringified annotations in other locations will emit a MAYBE_UNRECOGNIZED_STR_TYPEFORM error.

• The existing TypeType class is now used to represent BOTH the Type[T] and TypeForm[T] types, rather than introducing a distinct subclass of Type to represent the TypeForm[T] type. This was done to simplify logic that manipulates both Type[T] and TypeForm[T] values, since they are both manipulated in very similar ways.

• The "normalized" form of TypeForm[X | Y] - as returned by TypeType.make_normalized() - is just TypeForm[X | Y] rather than TypeForm[X] | TypeForm[Y], differing from the normalization behavior of Type[X | Y].

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

discord.py (https://github.com/Rapptz/discord.py)
- ...venv/lib/python3.13/site-packages/mypy/typeshed/stdlib/typing.pyi:1016: note: "update" of "TypedDict" defined here
+ ...venv/lib/python3.13/site-packages/mypy/typeshed/stdlib/typing.pyi:1017: note: "update" of "TypedDict" defined here

spark (https://github.com/apache/spark)
- python/pyspark/pandas/supported_api_gen.py:400: SyntaxWarning: invalid escape sequence '\_'
-   return func_str[:-1] + "\_"  # noqa: W605

[davidfstr]

I fixed all the "only in CI" check issues so this PR is now actually ready for review.

[davidfstr]

@JukkaL Did you want to look at this latest PR draft yourself, or recommend a review from any of the other mypy maintainers?

[JukkaL]

I can have a look. Sorry for the delay, this is a big PR and I need to block some time to review this properly.

[davidfstr]

Sometime between 2-6 weeks from now I'm expecting to become unavailable for a few months due to increased family commitments. If it's possible to get feedback on this PR before then, it will be able to be merged in a timely fashion and mypy will gain TypeForm support in 2025.