GH-95245: Document use of MANAGED flags instead of offsets. (GH-96044)

Author: markshannon

Doc/c-api/structures.rst

@@ -469,19 +469,21 @@ Accessing attributes of extension types
    .. _pymemberdef-offsets:
 
    Heap allocated types (created using :c:func:`PyType_FromSpec` or similar),
-   ``PyMemberDef`` may contain definitions for the special members
-   ``__dictoffset__``, ``__weaklistoffset__`` and ``__vectorcalloffset__``,
-   corresponding to
-   :c:member:`~PyTypeObject.tp_dictoffset`,
-   :c:member:`~PyTypeObject.tp_weaklistoffset` and
+   ``PyMemberDef`` may contain definitions for the special member
+   ``__vectorcalloffset__``, corresponding to
    :c:member:`~PyTypeObject.tp_vectorcall_offset` in type objects.
    These must be defined with ``T_PYSSIZET`` and ``READONLY``, for example::
 
       static PyMemberDef spam_type_members[] = {
-          {"__dictoffset__", T_PYSSIZET, offsetof(Spam_object, dict), READONLY},
+          {"__vectorcalloffset__", T_PYSSIZET, offsetof(Spam_object, vectorcall), READONLY},
           {NULL}  /* Sentinel */
       };
 
+   The legacy offsets :c:member:`~PyTypeObject.tp_dictoffset` and
+   :c:member:`~PyTypeObject.tp_weaklistoffset` are still supported, but extensions are
+   strongly encouraged to use ``Py_TPFLAGS_MANAGED_DICT`` and
+   ``Py_TPFLAGS_MANAGED_WEAKREF`` instead.
+
 
 .. c:function:: PyObject* PyMember_GetOne(const char *obj_addr, struct PyMemberDef *m)
 

Doc/c-api/type.rst

@@ -327,9 +327,9 @@ The following functions and structs are used to create
       * :c:member:`~PyTypeObject.tp_weaklist`
       * :c:member:`~PyTypeObject.tp_vectorcall`
       * :c:member:`~PyTypeObject.tp_weaklistoffset`
-        (see :ref:`PyMemberDef <pymemberdef-offsets>`)
+        (use :const:`Py_TPFLAGS_MANAGED_WEAKREF` instead)
       * :c:member:`~PyTypeObject.tp_dictoffset`
-        (see :ref:`PyMemberDef <pymemberdef-offsets>`)
+        (use :const:`Py_TPFLAGS_MANAGED_DICT` instead)
       * :c:member:`~PyTypeObject.tp_vectorcall_offset`
         (see :ref:`PyMemberDef <pymemberdef-offsets>`)
 

Doc/c-api/typeobj.rst

@@ -96,7 +96,7 @@ Quick Reference
    |                                                |                                   | __gt__,           |   |   |   |   |
    |                                                |                                   | __ge__            |   |   |   |   |
    +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
-   | :c:member:`~PyTypeObject.tp_weaklistoffset`    | :c:type:`Py_ssize_t`              |                   |   | X |   | ? |
+   | (:c:member:`~PyTypeObject.tp_weaklistoffset`)  | :c:type:`Py_ssize_t`              |                   |   | X |   | ? |
    +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
    | :c:member:`~PyTypeObject.tp_iter`              | :c:type:`getiterfunc`             | __iter__          |   |   |   | X |
    +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
@@ -117,7 +117,7 @@ Quick Reference
    | :c:member:`~PyTypeObject.tp_descr_set`         | :c:type:`descrsetfunc`            | __set__,          |   |   |   | X |
    |                                                |                                   | __delete__        |   |   |   |   |
    +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
-   | :c:member:`~PyTypeObject.tp_dictoffset`        | :c:type:`Py_ssize_t`              |                   |   | X |   | ? |
+   | (:c:member:`~PyTypeObject.tp_dictoffset`)      | :c:type:`Py_ssize_t`              |                   |   | X |   | ? |
    +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
    | :c:member:`~PyTypeObject.tp_init`              | :c:type:`initproc`                | __init__          | X | X |   | X |
    +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
@@ -1018,7 +1018,6 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    :const:`Py_TPFLAGS_HAVE_GC` flag bit is clear in the subtype and the
    :c:member:`~PyTypeObject.tp_traverse` and :c:member:`~PyTypeObject.tp_clear` fields in the subtype exist and have
    ``NULL`` values.
-
    .. XXX are most flag bits *really* inherited individually?
 
    **Default:**
@@ -1135,6 +1134,33 @@ and :c:type:`PyType_Type` effectively act as defaults.)
       :const:`Py_TPFLAGS_IMMUTABLETYPE` flag set.  For extension types, it is
       inherited whenever :c:member:`~PyTypeObject.tp_descr_get` is inherited.
 
+    .. data:: Py_TPFLAGS_MANAGED_DICT
+
+       This bit indicates that instances of the class have a ``__dict___``
+       attribute, and that the space for the dictionary is managed by the VM.
+
+       If this flag is set, :const:`Py_TPFLAGS_HAVE_GC` should also be set.
+
+       .. versionadded:: 3.12
+
+      **Inheritance:**
+
+      This flag is inherited unless the
+      :c:member:`~PyTypeObject.tp_dictoffset` field is set in a superclass.
+
+
+    .. data:: Py_TPFLAGS_MANAGED_WEAKREF
+
+       This bit indicates that instances of the class should be weakly
+       referenceable.
+
+       .. versionadded:: 3.12
+
+      **Inheritance:**
+
+      This flag is inherited unless the
+      :c:member:`~PyTypeObject.tp_weaklistoffset` field is set in a superclass.
+
 
    .. XXX Document more flags here?
 
@@ -1487,6 +1513,9 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
 .. c:member:: Py_ssize_t PyTypeObject.tp_weaklistoffset
 
+   While this field is still supported, :const:`Py_TPFLAGS_MANAGED_WEAKREF`
+   should be used instead, if at all possible.
+
    If the instances of this type are weakly referenceable, this field is greater
    than zero and contains the offset in the instance structure of the weak
    reference list head (ignoring the GC header, if present); this offset is used by
@@ -1497,26 +1526,22 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    Do not confuse this field with :c:member:`~PyTypeObject.tp_weaklist`; that is the list head for
    weak references to the type object itself.
 
+   It is an error to set both the :const:`Py_TPFLAGS_MANAGED_WEAKREF` bit and
+   :c:member:`~PyTypeObject.tp_weaklist`.
+
    **Inheritance:**
 
    This field is inherited by subtypes, but see the rules listed below. A subtype
    may override this offset; this means that the subtype uses a different weak
    reference list head than the base type.  Since the list head is always found via
    :c:member:`~PyTypeObject.tp_weaklistoffset`, this should not be a problem.
 
-   When a type defined by a class statement has no :attr:`~object.__slots__` declaration,
-   and none of its base types are weakly referenceable, the type is made weakly
-   referenceable by adding a weak reference list head slot to the instance layout
-   and setting the :c:member:`~PyTypeObject.tp_weaklistoffset` of that slot's offset.
-
-   When a type's :attr:`__slots__` declaration contains a slot named
-   :attr:`__weakref__`, that slot becomes the weak reference list head for
-   instances of the type, and the slot's offset is stored in the type's
-   :c:member:`~PyTypeObject.tp_weaklistoffset`.
+   **Default:**
 
-   When a type's :attr:`__slots__` declaration does not contain a slot named
-   :attr:`__weakref__`, the type inherits its :c:member:`~PyTypeObject.tp_weaklistoffset` from its
-   base type.
+   If the :const:`Py_TPFLAGS_MANAGED_WEAKREF` bit is set in the
+   :c:member:`~PyTypeObject.tp_dict` field, then
+   :c:member:`~PyTypeObject.tp_weaklistoffset` will be set to a negative value,
+   to indicate that it is unsafe to use this field.
 
 
 .. c:member:: getiterfunc PyTypeObject.tp_iter
@@ -1695,6 +1720,9 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
 .. c:member:: Py_ssize_t PyTypeObject.tp_dictoffset
 
+   While this field is still supported, :const:`Py_TPFLAGS_MANAGED_DICT` should be
+   used instead, if at all possible.
+
    If the instances of this type have a dictionary containing instance variables,
    this field is non-zero and contains the offset in the instances of the type of
    the instance variable dictionary; this offset is used by
@@ -1703,48 +1731,34 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    Do not confuse this field with :c:member:`~PyTypeObject.tp_dict`; that is the dictionary for
    attributes of the type object itself.
 
-   If the value of this field is greater than zero, it specifies the offset from
-   the start of the instance structure.  If the value is less than zero, it
-   specifies the offset from the *end* of the instance structure.  A negative
-   offset is more expensive to use, and should only be used when the instance
-   structure contains a variable-length part.  This is used for example to add an
-   instance variable dictionary to subtypes of :class:`str` or :class:`tuple`. Note
-   that the :c:member:`~PyTypeObject.tp_basicsize` field should account for the dictionary added to
-   the end in that case, even though the dictionary is not included in the basic
-   object layout.  On a system with a pointer size of 4 bytes,
-   :c:member:`~PyTypeObject.tp_dictoffset` should be set to ``-4`` to indicate that the dictionary is
-   at the very end of the structure.
+   The value specifies the offset of the dictionary from the start of the instance structure.
 
    The :c:member:`~PyTypeObject.tp_dictoffset` should be regarded as write-only.
    To get the pointer to the dictionary call :c:func:`PyObject_GenericGetDict`.
    Calling :c:func:`PyObject_GenericGetDict` may need to allocate memory for the
    dictionary, so it is may be more efficient to call :c:func:`PyObject_GetAttr`
    when accessing an attribute on the object.
 
-   **Inheritance:**
-
-   This field is inherited by subtypes, but see the rules listed below. A subtype
-   may override this offset; this means that the subtype instances store the
-   dictionary at a difference offset than the base type.  Since the dictionary is
-   always found via :c:member:`~PyTypeObject.tp_dictoffset`, this should not be a problem.
+   It is an error to set both the :const:`Py_TPFLAGS_MANAGED_WEAKREF` bit and
+   :c:member:`~PyTypeObject.tp_dictoffset`.
 
-   When a type defined by a class statement has no :attr:`~object.__slots__` declaration,
-   and none of its base types has an instance variable dictionary, a dictionary
-   slot is added to the instance layout and the :c:member:`~PyTypeObject.tp_dictoffset` is set to
-   that slot's offset.
-
-   When a type defined by a class statement has a :attr:`__slots__` declaration,
-   the type inherits its :c:member:`~PyTypeObject.tp_dictoffset` from its base type.
+   **Inheritance:**
 
-   (Adding a slot named :attr:`~object.__dict__` to the :attr:`__slots__` declaration does
-   not have the expected effect, it just causes confusion.  Maybe this should be
-   added as a feature just like :attr:`__weakref__` though.)
+   This field is inherited by subtypes. A subtype should not override this offset;
+   doing so could be unsafe, if C code tries to access the dictionary at the
+   previous offset.
+   To properly support inheritance, use :const:`Py_TPFLAGS_MANAGED_DICT`.
 
    **Default:**
 
    This slot has no default.  For :ref:`static types <static-types>`, if the
    field is ``NULL`` then no :attr:`__dict__` gets created for instances.
 
+   If the :const:`Py_TPFLAGS_MANAGED_DICT` bit is set in the
+   :c:member:`~PyTypeObject.tp_dict` field, then
+   :c:member:`~PyTypeObject.tp_dictoffset` will be set to ``-1``, to indicate
+   that it is unsafe to use this field.
+
 
 .. c:member:: initproc PyTypeObject.tp_init
 
@@ -2663,18 +2677,16 @@ A type that supports weakrefs, instance dicts, and hashing::
    typedef struct {
        PyObject_HEAD
        const char *data;
-       PyObject *inst_dict;
-       PyObject *weakreflist;
    } MyObject;
 
    static PyTypeObject MyObject_Type = {
        PyVarObject_HEAD_INIT(NULL, 0)
        .tp_name = "mymod.MyObject",
        .tp_basicsize = sizeof(MyObject),
        .tp_doc = PyDoc_STR("My objects"),
-       .tp_weaklistoffset = offsetof(MyObject, weakreflist),
-       .tp_dictoffset = offsetof(MyObject, inst_dict),
-       .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC,
+       .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE |
+            Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_MANAGED_DICT |
+            Py_TPFLAGS_MANAGED_WEAKREF,
        .tp_new = myobj_new,
        .tp_traverse = (traverseproc)myobj_traverse,
        .tp_clear = (inquiry)myobj_clear,

Doc/extending/newtypes.rst

@@ -570,43 +570,28 @@ performance-critical objects (such as numbers).
 .. seealso::
    Documentation for the :mod:`weakref` module.
 
-For an object to be weakly referencable, the extension type must do two things:
+For an object to be weakly referencable, the extension type must set the
+``Py_TPFLAGS_MANAGED_WEAKREF`` bit of the :c:member:`~PyTypeObject.tp_flags`
+field. The legacy :c:member:`~PyTypeObject.tp_weaklistoffset` field should
+be left as zero.
 
-#. Include a :c:type:`PyObject\*` field in the C object structure dedicated to
-   the weak reference mechanism.  The object's constructor should leave it
-   ``NULL`` (which is automatic when using the default
-   :c:member:`~PyTypeObject.tp_alloc`).
-
-#. Set the :c:member:`~PyTypeObject.tp_weaklistoffset` type member
-   to the offset of the aforementioned field in the C object structure,
-   so that the interpreter knows how to access and modify that field.
-
-Concretely, here is how a trivial object structure would be augmented
-with the required field::
-
-   typedef struct {
-       PyObject_HEAD
-       PyObject *weakreflist;  /* List of weak references */
-   } TrivialObject;
-
-And the corresponding member in the statically declared type object::
+Concretely, here is how the statically declared type object would look::
 
    static PyTypeObject TrivialType = {
        PyVarObject_HEAD_INIT(NULL, 0)
        /* ... other members omitted for brevity ... */
-       .tp_weaklistoffset = offsetof(TrivialObject, weakreflist),
+       .tp_flags = Py_TPFLAGS_MANAGED_WEAKREF | ...,
    };
 
+
 The only further addition is that ``tp_dealloc`` needs to clear any weak
-references (by calling :c:func:`PyObject_ClearWeakRefs`) if the field is
-non-``NULL``::
+references (by calling :c:func:`PyObject_ClearWeakRefs`)::
 
    static void
    Trivial_dealloc(TrivialObject *self)
    {
        /* Clear weakrefs first before calling any destructors */
-       if (self->weakreflist != NULL)
-           PyObject_ClearWeakRefs((PyObject *) self);
+       PyObject_ClearWeakRefs((PyObject *) self);
        /* ... remainder of destruction code omitted for brevity ... */
        Py_TYPE(self)->tp_free((PyObject *) self);
    }