PEP 727: Review (#3316)

* Strip thread name

* Link to Python docs

* Reference PEP 484

* Fix heading levels

* Add missing sections (as a comment)

* Rename EAaOPV to Reference Implementation

* Code block formatting

* Remove leading elipses

* Tighten usage criteria

* Remove non-specification content

* Rewrite the Specification

* Simplify Examples and move DocInfo to the reference implementation

* Improve the Reference Implementation

* Syntax

Author: AA-Turner

pep-0727.rst

@@ -2,22 +2,22 @@ PEP: 727
 Title: Documentation Metadata in Typing
 Author: Sebastián Ramírez <[email protected]>
 Sponsor: Jelle Zijlstra <[email protected]>
-Discussions-To: https://discuss.python.org/t/pep-727-documentation-metadata-in-typing/32566
+Discussions-To: https://discuss.python.org/t/32566
 Status: Draft
 Type: Standards Track
 Topic: Typing
 Content-Type: text/x-rst
 Created: 28-Aug-2023
 Python-Version: 3.13
-Post-History: `30-Aug-2023 <https://discuss.python.org/t/pep-727-documentation-metadata-in-typing/32566>`__
+Post-History: `30-Aug-2023 <https://discuss.python.org/t/32566>`__
 
 
 Abstract
 ========
 
 This document proposes a way to complement docstrings to add additional documentation
-to Python symbols using type annotations with ``Annotated`` (in class attributes,
-function and method parameters, return values, and variables).
+to Python symbols using type annotations with :py:class:`~typing.Annotated`
+(in class attributes, function and method parameters, return values, and variables).
 
 
 Motivation
@@ -60,7 +60,7 @@ documentation in some other way (e.g. an API, a CLI, etc).
 Some of these previous formats tried to account for the lack of type annotations
 in older Python versions by including typing information in the docstrings,
 but now that information doesn't need to be in docstrings as there is now an official
-syntax for type annotations.
+:pep:`syntax for type annotations <484>`.
 
 
 Rationale
@@ -84,79 +84,56 @@ like to adopt it.
 Specification
 =============
 
+The main proposal is to introduce a new function, ``typing.doc()``,
+to be used when documenting Python objects.
+This function MUST only be used within :py:class:`~typing.Annotated` annotations.
+The function takes a single string argument, ``documentation``,
+and returns an instance of ``typing.DocInfo``,
+which stores the input string unchanged.
 
-``typing.doc``
---------------
+Any tool processing ``typing.DocInfo`` objects SHOULD interpret the string as
+a docstring, and therefore SHOULD normalize whitespace
+as if ``inspect.cleandoc()`` were used.
 
-The main proposal is to have a new function ``doc()`` in the ``typing`` module.
-Even though this is not strictly related to the type annotations, it's expected
-to go in ``Annotated`` type annotations, and to interact with type annotations.
+The string passed to ``typing.doc()`` SHOULD be of the form that would be a valid docstring.
+This means that `f-strings`__ and string operations SHOULD NOT be used.
+As this cannot be enforced by the Python runtime,
+tools SHOULD NOT rely on this behaviour,
+and SHOULD exit with an error if such a prohibited string is encountered.
 
-There's also the particular benefit that it could be implemented in the
-``typing_extensions`` package to have support for older versions of Python and
-early adopters of this proposal.
-
-This ``doc()`` function would receive one single parameter ``documentation`` with
-a documentation string.
-
-This string could be a multi-line string, in which case, when extracted by tools,
-should be interpreted cleaning up indentation as if using ``inspect.cleandoc()``,
-the same procedure used for docstrings.
-
-This string could probably contain markup, like Markdown or reST. As that could
-be highly debated, that decision is left for a future proposal, to focus here
-on the main functionality.
-
-This specification targets static analysis tools and editors, and as such, the
-value passed to ``doc()`` should allow static evaluation and analysis. If a
-developer passes as the value something that requires runtime execution
-(e.g. a function call) the behavior of static analysis tools is unspecified
-and they could omit it from their process and results. For static analysis
-tools to be conformant with this specification they need only to support
-statically accessible values.
-
-An example documenting the attributes of a class, or in this case, the keys
-of a ``TypedDict``, could look like this:
-
-.. code-block::
-
-    from typing import Annotated, TypedDict, NotRequired, doc
-
-
-    class User(TypedDict):
-        firstname: Annotated[str, doc("The user's first name")]
-        lastname: Annotated[str, doc("The user's last name")]
+__ https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals
 
+Examples
+--------
 
-An example documenting the parameters of a function could look like this:
+Class attributes may be documented:
 
-.. code-block::
+.. code:: python
 
-    from typing import Annotated, doc
+   from typing import Annotated, doc
 
+   class User:
+       first_name: Annotated[str, doc("The user's first name")]
+       last_name: Annotated[str, doc("The user's last name")]
 
-    def create_user(
-        lastname: Annotated[str, doc("The **last name** of the newly created user")],
-        firstname: Annotated[str | None, doc("The user's **first name**")] = None,
-    ) -> Annotated[User, doc("The created user after saving in the database")]:
-        """
-        Create a new user in the system, it needs the database connection to be already
-        initialized.
-        """
-        pass
+       ...
 
+As can function or method parameters:
 
-The return of the ``doc()`` function is an instance of a class that can be checked
-and used at runtime, defined similar to:
+.. code:: python
 
-.. code-block::
+   from typing import Annotated, doc
 
-    class DocInfo:
-        def __init__(self, documentation: str):
-            self.documentation = documentation
+   def create_user(
+       first_name: Annotated[str, doc("The user's first name")],
+       last_name: Annotated[str, doc("The user's last name")],
+       cursor: DatabaseConnection | None = None,
+   ) -> Annotated[User, doc("The created user after saving in the database")]:
+       """Create a new user in the system.
 
-...where the attribute ``documentation`` contains the same value string passed to
-the function ``doc()``.
+       It needs the database connection to be already initialized.
+       """
+       pass
 
 
 Additional Scenarios
@@ -171,52 +148,48 @@ but implementers are not required to support them.
 
 
 Type Alias
-----------
+''''''''''
 
 When creating a type alias, like:
 
-.. code-block::
+.. code:: python
 
-    Username = Annotated[str, doc("The name of a user in the system")]
+   Username = Annotated[str, doc("The name of a user in the system")]
 
 
-...the documentation would be considered to be carried by the parameter annotated
+The documentation would be considered to be carried by the parameter annotated
 with ``Username``.
 
 So, in a function like:
 
-.. code-block::
+.. code:: python
 
-    def hi(
-      to: Username,
-    ) -> None: ...
+   def hi(to: Username) -> None: ...
 
 
-...it would be equivalent to:
+It would be equivalent to:
 
-.. code-block::
+.. code:: python
 
-    def hi(
-      to: Annotated[str, doc("The name of a user in the system")],
-    ) -> None: ...
+   def hi(to: Annotated[str, doc("The name of a user in the system")]) -> None: ...
 
 Nevertheless, implementers would not be required to support type aliases outside
 of the final type annotation to be conformant with this specification, as it
 could require more complex dereferencing logic.
 
 
 Annotating Type Parameters
---------------------------
+''''''''''''''''''''''''''
 
 When annotating type parameters, as in:
 
-.. code-block::
+.. code:: python
 
-    def hi(
-      to: list[Annotated[str, doc("The name of a user in a list")]],
-    ) -> None: ...
+   def hi(
+       to: list[Annotated[str, doc("The name of a user in a list")]],
+   ) -> None: ...
 
-...the documentation in ``doc()`` would refer to what it is annotating, in this
+The documentation in ``doc()`` would refer to what it is annotating, in this
 case, each item in the list, not the list itself.
 
 There are currently no practical use cases for documenting type parameters,
@@ -225,17 +198,17 @@ conformant, but it's included for completeness.
 
 
 Annotating Unions
------------------
+'''''''''''''''''
 
 If used in one of the parameters of a union, as in:
 
-.. code-block::
+.. code:: python
 
-    def hi(
-      to: str | Annotated[list[str], doc("List of user names")],
-    ) -> None: ...
+   def hi(
+       to: str | Annotated[list[str], doc("List of user names")],
+   ) -> None: ...
 
-...again, the documentation in ``doc()`` would refer to what it is annotating,
+Again, the documentation in ``doc()`` would refer to what it is annotating,
 in this case, this documents the list itself, not its items.
 
 In particular, the documentation would not refer to a single string passed as a
@@ -247,22 +220,23 @@ included for completeness.
 
 
 Nested ``Annotated``
---------------------
+''''''''''''''''''''
 
 Continuing with the same idea above, if ``Annotated`` was used nested and used
 multiple times in the same parameter, ``doc()`` would refer to the type it
 is annotating.
 
 So, in an example like:
 
-.. code-block::
+.. code:: python
 
-    def hi(
-      to: Annotated[
-            Annotated[str, doc("A user name")] | Annotated[list, doc("A list of user names")],
-            doc("Who to say hi to"),
-      ],
-    ) -> None: ...
+   def hi(
+       to: Annotated[
+           Annotated[str, doc("A user name")]
+           | Annotated[list, doc("A list of user names")],
+           doc("Who to say hi to"),
+       ],
+   ) -> None: ...
 
 
 The documentation for the whole parameter ``to`` would be considered to be
@@ -281,16 +255,16 @@ of the parameter passed is of one type or another, but they are not required to
 
 
 Duplication
------------
+'''''''''''
 
 If ``doc()`` is used multiple times in a single ``Annotated``, it would be
 considered invalid usage from the developer, for example:
 
-.. code-block::
+.. code:: python
 
-    def hi(
-      to: Annotated[str, doc("A user name"), doc("The current user name")],
-    ) -> None: ...
+   def hi(
+       to: Annotated[str, doc("A user name"), doc("The current user name")],
+   ) -> None: ...
 
 
 Implementers can consider this invalid and are not required to support this to be
@@ -302,46 +276,68 @@ can opt to support one of the ``doc()`` declarations.
 In that case, the suggestion would be to support the last one, just because
 this would support overriding, for example, in:
 
-.. code-block::
+.. code:: python
 
-    User = Annotated[str, doc("A user name")]
+   User = Annotated[str, doc("A user name")]
 
-    CurrentUser = Annotated[User, doc("The current user name")]
+   CurrentUser = Annotated[User, doc("The current user name")]
 
 
 Internally, in Python, ``CurrentUser`` here is equivalent to:
 
-.. code-block::
+.. code:: python
 
-    CurrentUser = Annotated[str, doc("A user name"), doc("The current user name")]
+   CurrentUser = Annotated[str,
+                           doc("A user name"),
+                           doc("The current user name")]
 
 
 For an implementation that supports the last ``doc()`` appearance, the above
 example would be equivalent to:
 
-.. code-block::
+.. code:: python
+
+   def hi(to: Annotated[str, doc("The current user name")]) -> None: ...
+
+
+.. you need to fill these in:
+
+   Backwards Compatibility
+   =======================
+
+   [Describe potential impact and severity on pre-existing code.]
+
+
+   Security Implications
+   =====================
+
+   [How could a malicious user take advantage of this new feature?]
+
+
+   How to Teach This
+   =================
+
+   [How to teach users, new and experienced, how to apply the PEP to their work.]
 
-    def hi(
-      to: Annotated[str, doc("The current user name")],
-    ) -> None: ...
 
+Reference Implementation
+========================
 
-Early Adopters and Older Python Versions
-========================================
+``typing.doc`` and ``typing.DocInfo`` are implemented as follows:
 
-For older versions of Python and early adopters of this proposal, ``doc()`` and
-``DocInfo`` can be imported from the ``typing_extensions`` package.
+.. code:: python
 
-.. code-block::
+   def doc(documentation: str, /) -> DocInfo:
+       return DocInfo(documentation)
 
-    from typing import Annotated
+   class DocInfo:
+       def __init__(self, documentation: str, /):
+           self.documentation = documentation
 
-    from typing_extensions import doc
 
+These have been implemented in the `typing_extensions`__ package.
 
-    def hi(
-      to: Annotated[str, doc("The current user name")],
-    ) -> None: ...
+__ https://pypi.org/project/typing-extensions/
 
 
 Rejected Ideas
@@ -407,8 +403,8 @@ to be used by those that are willing to take the extra verbosity in exchange
 for the benefits.
 
 
-Doc is not Typing
------------------
+Documentation is not Typing
+---------------------------
 
 It could also be argued that documentation is not really part of typing, or that
 it should live in a different module. Or that this information should not be part