From 72ad764ceddea29980e5bbd45dddfa27c2952ff8 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Mon, 16 Oct 2023 15:48:00 +0100 Subject: [PATCH 01/48] first draft as copied from the capi-workgroup repo --- peps/pep-9999.rst | 632 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 632 insertions(+) create mode 100644 peps/pep-9999.rst diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst new file mode 100644 index 00000000000..bfd9b7bbaf3 --- /dev/null +++ b/peps/pep-9999.rst @@ -0,0 +1,632 @@ +PEP: 9999 +Title: An Evaluation of Python's Public C API +Author: Erlend E. Aasland , Domenico Andreoli <>, Stefan Behnel , Carl Friedrich Bolz-Tereick <>, Simon Cross <>, Steve Dower , Tim Felgentreff <>, David Hewitt <>, Shantanu Jain , Wenzel Jakob <>, Irit Katriel , Marc-Andre Lemburg , Dong-hee Na <>, Karl Nelson <>, Ronald Oussoren , Matti Picus <>, Antoine Pitrou , Benedikt Reinartz <>, Guido van Rossum , Neil Schemenauer , Mark Shannon , Stepan Sindelar <>, Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka <>, Petr Viktorin , Carol Willing , William Woodruff , David Woods <>, Jelle Zijlstra +Status: Draft +Type: Informational +Content-Type: text/x-rst +Created: 16-Oct-2023 + + +Abstract +======== + +This **informational** PEP describes our shared view of the public C API. The +document defines: + +* purposes of the C API +* stakeholders and their particular use cases and requirements +* strengths of the C API +* problems of the C API categorized into nine areas of weakness + +This document does not propose solutions to any of the identified problems. By +creating a shared list of C API issues, this document will help to guide +continuing discussion about change proposals and to identify evaluation +criteria. + + +Introduction +============ + +Python's C API was not designed for the purposes it currently fulfils. +It evolved from what was initially the internal API between the C code +of the interpreter and the Python language and libraries. In its first +incarnation, it was exposed to make it possible to embed Python into C/C++ +applications and to write extension modules in C/C++. +These capabilities were instrumental to the growth of Python's ecosystem. +Over the decades, the C API grew to provide different tiers of stability, +conventions changed, and new usage patterns have emerged, such as bindings +to languages other than C/C++. In addition, CPython is no longer the only +implementation of the C API, and some of the design decisions made when +it was, are difficult for alternative implementations to work with +[`Issue 64 `__]. +Finally, lessons were learned and mistakes in both the design and the +implementation of the C API were identified. + +Evolving the C API is hard due to the combination of backwards +compatibility constraints and its inherent complexity, both +technical and social. Different types of users bring different, +sometimes conflicting, requirements. The tradeoff between stability +and progress is an ongoing, highly contentious topic of discussion +when suggestions are made for incremental improvements. +Several proposals have been put forward for improvement, redesign +or replacement of the C API, each representing a deep analysis of +the problems. At the 2023 Language Summit, three back-to-back +sessions were devoted to different aspects of the C API. There is +general agreement that a new design can remedy the problems that +the C API has accumulated over the last 30 years, while at the same +time updating it for use cases that it was not originally designed for. + +However, there was also a sense at the Language Summit that we are +trying to discuss solutions without a clear common understanding +of the problems that we are trying to solve. It was decided that +we need to agree on the current problems with the C API, before +we are able to evaluate any of the proposed solutions. We +therefore created the +[`capi-workgroup `__] +repository on GitHub in order to collect everyone's ideas on that +question. + +Over 60 different issues were created on that repository, each +describing a problem with the C API. They were categorized and +a number of recurring themes were identified. The sections below +mostly correspond to these themes, and each contains a combined +description of the issues raised in that category, along with +links to the individual issues. In addition, we included a section +that aims to identify the different stakeholders of the C API, +and the particular requirements that each of them has. + + +C API Stakeholders +================== + +As mentioned in the introduction, the C API was originally +created as the internal interface between CPython's +interpreter and the Python layer. It was later exposed as +a way for third party developers to extend and embed Python +programs. Over the years, new types of stakeholders emerged, +with different requirements and areas of focus. This section +describes this complex state of affairs in terms of the +actions that different stakeholders need to perform through +the C API. + +Universal Actions for All Stakeholders +-------------------------------------- + +There are actions which are generic, and required by +all types of API users: + +* Define functions and call them +* Create classes and instantiate them +* Create instances of builtin and user defined types + and perform operations on them +* Introspect objects, including types, instances, and functions. +* Raise and handle exceptions +* Import modules +* Manage threads +* Manage sub-interpreters +* Handle and send signals + +The following sections look at the unique requirements of various stakeholders. + +Extension writers +----------------- + +Extension writers are the traditional users of the C API. Their requirements +are the universal actions listed above. + +Authors of Embedded Python Applications +--------------------------------------- + +Applications with an embedded Python interpreter. Examples are +`Blender `__ and +`OBS `__. + +They need to be able to: + +* Configure the interpreter (import paths, inittab, sys.argv, ...) +* Interact with the execution model and program lifetime, including + clean interpreter shutdown and restart +* Represent complex data models in a way Python can use without + having to create deep copies. +* Provide and import frozen modules. +* Run multiple independent interpreters (in particular, when embedded + in a library that wants to avoid global effects). + +Alternative Python Implementations +---------------------------------- + +Alternative implementations of Python (such as +`PyPy `__, +`GraalPy `__, +`IronPython `__, +`RustPython `__, +`MicroPython `__, +and `Jython `__), may take +very different approaches for the implementation of +different subsystems. They need: + +* The API to be abstract and hide implementation details. +* A specification of the API, ideally with a test suite + that ensures compatibility. +* It would be nice to have an ABI that can be shared + across Python implementations. + +Alternative APIs +---------------- + +There are several projects that implement alternatives to the +C API, which offer extension users advantanges over programming +directly with the C API. These APIs are implemented with the +C API, and in some cases by using cpython internals. +Some examples are +`Cython `__, +`HPy `__ and +`pythoncapi-compat `__. +CPython's DSL for parsing function arguments, the +`Argument Clinic `__, +can also be seen as belonging to this category of stakeholders. + +Such systems need minimal building blocks for accessing CPython +efficiently. They don't necessarily need an ergonomic API, because +they typically generate code that is not intended to be read +by humans. But they do need it to be comprehensive enough so that +they don't need to access internals, while offering them stability, +and without sacrificing performance. + +An alternative is to have a fast API tier with less error checking +and lower stability guarantees. Then the developers and users of +these tools can choose whether to generate code that uses the +faster or the safer and more stable version of the API. + +Binding generators +------------------ + +Libraries that create bindings between Python and other object models, +paradigms or languages, such as +`pybind11 `__ for C++11, +`PyO3 `__ for Rust, +`PySide `__ for Qt, +`PyGObject `__ for GTK, +`Pygolo `__ for Go, +`PyJNIus `__ for Java, or +`SWIG `__ for C/C++. + +They need to: + +* Create custom objects (e.g. function/module objects + and traceback entries) that match the behavior of equivalent + Python code as closely as possible. +* Dynamically create objects which are static in traditional + C extensions (e.g. classes/modules), and need CPython to manage + their state and lifetime. +* Adapt foreign objects (strings, GC'd containers), with low overhead. +* Adapt external mechanisms, execution models and guarantees to the + Python way (green threads/continuations, one-writer-or-multiple-readers + semantics, virtual multiple inheritance, 1-based indexing, super-long + inheritance chains, goroutines, channels, ...) + +Strengths of the C API +====================== + +While the bulk of this document is devoted to problems with the +C API that we would like to see fixed in any new design, it is +also important to point out the strengths of the C API, and to +make sure that they are preserved. + +As mentioned in the introduction, the C API enabled the +development and growth of the Python ecosystem over the last +three decades, while evolving to support use cases that it was +not originally designed for. This track record in itself is +indication of how effective and valuable it has been. + +A number of specific strengths were mentioned in the +capi-workgroup discussions. Heap types were identified +as much safer and easier to use than static types +[`Issue 4 `__]. + +API functions that take a C string literal for lookups based +on a Python string are very convenient +[`Issue 30 `__]. + +The Limited API and stable ABI hide implementation details and +make it easier to evolve Python +[`Issue 30 `__]. + +C API problems +============== + +The remainder of this document summarizes and categorizes the problems that were reported on +the [`capi-workgroup `__] repository. The issues are grouped into nine categories: + +- API Evolution and Maintenance +- API Specification and Abstraction +- Object Reference Management +- Type Definition and Object Creation +- Error Handling +- API Tiers and Stability Guarantees +- Use of the C Language +- Implementation Flaws +- Missing Functionality + + +API Evolution and Maintenance +----------------------------- + +The difficulty of making changes in the C API is central to this report. It is +implicit in many of the issues we discuss here, particularly when we need to +decide whether an incremental bugfix can resolve the issue, or whether it can +only be addressed as part of an API redesign +[`Issue 44 `__]. The +benefit of each incremental change is often viewed as too small to justify the +disruption. Over time, this implies that every mistake we make in an API's +design or implementation remains with us indefinitely. + +We can take two views on this issue. One is that this is a problem and the +solution needs to be baked into any new C API we design, in the form of a +process for incremental API evolution. The other possible approach is that +this is not a problem to be solved, but rather a feature of any API. In this +view, API evolution should not be incremental, but rather through large +redesigns, each of which learns from the mistakes of the past and is not +shackled by backwards compatibility requirements. A compromise approach +is somewhere between these two extremes, fixing issues which are easy +or important enough to tackle incrementally, and leaving others alone. + +The problem we have in CPython is that we don't have an agreed, official +approach to API evolution. Different members of the core team are pulling in +different directions and this is an ongoing source of disagreements. +Any new C API needs to come with a clear decision about the model +that its maintenance will follow, as well as the technical and +organizational processes by which this will work. + +If the model does include provisions for incremental evolution of the API, +it will include processes for managing the impact of the change on users +[`Issue 60 `__], +perhaps through introducing an external backwards compatibility module +[`Issue 62 `__], +or a new API tier of "blessed" functions +[`Issue 55 `__]. + + +API Specification and Abstraction +--------------------------------- + +The C API does not have a formal specification, it is described +semi-formally in the documentation and exposed through C header +files. This creates a number of problems. + +Bindings for languages other than C/C++ must parse C code +[`Issue 7 `__]. +Some C language features are hard to handle in this way, because +they produce compiler-dependent output (such as enums) or require +a C preprocessor/compiler rather than just a parser (such as macros) +[`Issue 35 `__]. + +Furthermore, C header files tend to expose more than what is intended +to be part of the public API +[`Issue 34 `__]. +In particular, implementation details such as the fields of C structs +can be exposed +[`Issue 22 `__ +and :pep:`620`]. +This can make API evolution very difficult, in particular when it +occurs in the stable ABI as in the case of ``ob_refcnt`` and ``ob_type``, +which are accessed via the reference counting macros +[`Issue 45 `__]. + +A deeper issue was identified in relation to the way that reference +counting is exposed. The way that C extensions are required to +manage references with calls to ``Py_INCREF`` and ``Py_DECREF`` is +specific to CPython's memory model, and is hard for alternative +Python implementations to emulate. +[`Issue 12 `__]. + +Another set of problems arises from the fact that a ``PyObject*`` is +exposed in the C API as an actual pointer rather than a handle. The +address of an object serves as its ID and is used for comparison, +and this complicates matters for alternative Python implementations +that move objects during GC +[`Issue 37 `__]. + +A separate issue is that object references are opaque to the runtime, +discoverable only through calls to ``tp_traverse``/``tp_clear``, +which have their own purposes. If there was a way for the runtime to +know the structure of the object graph, and keep up with changes in it, +this would make it possible for alternative implementations to implement +different memory management schemes +[`Issue 33 `__]. + +Object Reference Management +--------------------------- + +There are C API functions that return borrowed references, and +functions that steal references to arguments, but there isn't a +naming convention that makes this obvious, so this is error prone +[`Issue 8 `__ +and `Issue 52 `__]. +The terminology used to describe these situations in the documentation +can also be improved +[`Issue 11 `__]. + +A more radical change is necessary in the case of functions that +return borrowed references (such as ``PyList_GetItem``) +[`Issue 5 `__ and +`Issue 21 `__] +or pointers to parts of the internal structure of an object +(such as ``PyBytes_AsString``) +[`Issue 57 `__]. +In both cases, the reference/pointer is valid for as long as the +owning object is alive, but this time is hard to reason about. Such +functions should not exist in the API without a mechanism that can +make them safe. + +For containers, the API is currently missing bulk operations on the +references of contained objects. This is particularly important for +a stable ABI where ``INCREF`` and ``DECREF`` cannot be macros, making +bulk operations expensive when implemented as a sequence of function +calls +[`Issue 15 `__]. + +Type Definition and Object Creation +----------------------------------- + +The C API has functions that make it possible to create incomplete +or inconsistent Python objects, such as ``PyTuple_New`` and +``PyUnicode_New``. This causes problem when the object is tracked +by GC or its ``tp_traverse``/``tp_clear`` functions are called. +Such functions should be removed from the C API. Related functions, +such as ``PyTuple_SetItem`` which is used to modify a partially +initialized tuple, should also be removed (tuples are immutable +once fully initialized) +[`Issue 56 `__]. + +A few issues were identified with type definition APIs. For legacy +reasons, there is often a significant amount of code duplication +between ``tp_new`` and ``tp_vectorcall`` +[`Issue 24 `__]. +The type slot function should be called indirectly, so that their +signatures can change to include context information +[`Issue 13 `__]. +Several aspects of the type definition and creation process are not +well defined, such as which stage of the process is responsible for +initializing and clearing different fields of the type object +[`Issue 49 `__]. + +Error Handling +-------------- + +Error handling in the C API is based on the error indicator which is stored +on the thread state (in global scope). The design intention was that each +API function returns a value indicating whether an error has occurred (by +convention, ``-1`` or ``NULL``). When the program knows that an error +occurred, it can fetch the exception object which is stored in the +error indicator. A number of problems were identified which are related +to error handling, pointing at APIs which are too easy to use incorrectly. + +There are functions that do not report all errors that occur while they +execute. For example, ``PyDict_GetItem`` clears any errors that occur +when it calls the key's hash function, or while performing a lookup +in the dictionary +[`Issue 51 `__]. + +Python code never executes with an in-flight exception (by definition), +and by the same token C API functions should never be called with the error +indicator set. This is currently not checked in most C API functions, and +there are places in the interpreter where error handling code calls a C API +function while an exception is set. For example, see the call to +``PyUnicode_FromString`` in the error handler of ``_PyErr_WriteUnraisableMsg`` +[`Issue 2 `__]. + + +There are functions that do not return a value, so a caller is forced to +query the error indicator in order to identify whether an error has occurred. +An example is ``PyBuffer_Release`` +[`Issue 20 `__]. +There are other functions which do have a return value, but this return value +does not unambiguously indicate whether an error has occurred. For example, +``PyLong_AsLong`` returns ``-1`` in case of error, or when the value of the +argument is indeed ``-1`` +[`Issue 1 `__]. +In both cases, the API is error prone because it is possible that the +error indicator was already set before the function was called, and the +error is incorrectly attributed. The fact that the error was not detected +before the call is a bug in the calling code, but the behaviour of the +program in this case doesn't make it easy to identify and debug the +problem. + +There are functions that take a ``PyObject*`` argument, with special meaning +when it is ``NULL``. For example, if ``PyObject_SetAttr`` receives ``NULL`` as +the value to set, this means that the attribute should be cleared. This is error +prone because it could be that ``NULL`` indicates an error in the construction +of the value, and the program failed to check for this error. The program will +misinterpret the ``NULL`` to mean something different than error +[`Issue 47 `__]. + + +API Tiers and Stability Guarantees +---------------------------------- + +The different API tiers provide different tradeoffs of stability vs +API evolution, and sometimes performance. + +The stable ABI was identified as an area that needs to be looked into. At +the moment it is incomplete and not widely adopted. At the same time, its +existence is making it hard to make changes to some implementation +details, because it exposes struct fields such as ``ob_refcnt``, +``ob_type`` and ``ob_size``. There was some discussion about whether +the stable ABI is worth keeping. Arguments on both sides can be +found in [`Issue 4 `__] +and [`Issue 9 `__]. + +Alternatively, it was suggested that in order to be able to evolve +the stable ABI, we need a mechanism to support multiple versions of +it in the same Python binary. It was pointed out that versioning +individual functions within a single ABI version is not enough +because it may be necessary to evolve, together, a group of functions +that interoperate with each other +[`Issue 39 `__]. + +The limited API was introduced in 3.2 as a blessed subset of the C API +which is recommended for users who would like to restrict themselves +to high quality APIs which are not likely to change often. The +``Py_LIMITED_API`` flag allows users to restrict their program to older +versions of the limited API, but we now need the opposite option, to +exclude older versions. This would make it possible to evolve the +limited API by replacing flawed elements in it +[`Issue 54 `__]. +More generally, in a redesign we should revisit the way that API +tiers are specified and consider designing a method that will unify the +way we currently select between the different tiers +[`Issue 59 `__]. + +API elements whose names begin with an underscore are considered +private, essentially an API tier with no stability guarantees. +However, this was only clarified recently, in :pep:`689`. It is +not clear what the change policy should be with respect to such +API elements that predate PEP 689 +[`Issue 58 `__]. + +There are API functions which have an unsafe (but fast) version as well as +a safe version which performs error checking (for example, +``PyTuple_GET_ITEM`` vs ``PyTuple_GetItem``). It may help to +be able to group them into their own tiers - the "unsafe API" tier and +the "safe API" tier +[`Issue 61 `__]. + +Use of the C Language +--------------------- + +A number of issues were raised with respect to the way that CPython +uses the C language. First there is the issue of which C dialect +we use, and how we test our compatibility with it +[`Issue 42 `__]. + +Usage of ``const`` in the API is currently sparse, but it is not +clear whether this is something that we should consider changing +[`Issue 38 `__]. + +We currently use the C types ``long`` and ``int``, where ``stdint`` +and ``int32_t`` would have been better choices +[`Issue 27 `__]. + +We are using C language features which are hard for other languages +to interact with +[`Issue 35 `__]. + +There are API functions that take a ``PyObject*`` arg which must be +of a more specific type (such as ``PyTuple_Size``, which fails if +its arg is not a ``PyTupleObject*``). It is an open question whether this +is a good pattern to have, or whether the API should expect the +more specific type +[`Issue 31 `__]. + +There are functions in the API that take concrete types, such as +``PyDict_GetItemString`` which performs a dictionary lookup for a key +specified as a c string rather than ``PyObject*``. At the same time, +for ``PyDict_ContainsString`` it is not considered appropriate to +add a concrete type alternative. The principle around this should +be documented in the guidelines +[`Issue 23 `__]. + +Implementation Flaws +-------------------- + +Below is a list of localized implementation flaws. Most of these can +probably be fixed incrementally, if we choose to do so. They should, +in any case, be avoided in any new API design. + +There are functions that don't follow the convention of +returning ``0`` for success and ``-1`` for failure. For +example, ``PyArg_ParseTuple`` returns 0 for success and +non-zero for failure +[`Issue 25 `__]. + +The macros ``Py_CLEAR`` and ``Py_SETREF`` access their arg more than +once, so if the arg is an expression with side effects, they are +duplicated +[`Issue 3 `__]. + +The meaning of ``Py_SIZE`` depends on the type and is not always +reliable +[`Issue 10 `__]. + +**Naming** + +``PyLong`` and ``PyUnicode`` use names which don't match the python +types they represent (int/str). This can be fixed in a new API +[`Issue 14 `__]. + +There are identifiers in the API which are lacking a ``Py``/``_Py`` +prefix +[`Issue 46 `__]. + +Some API function do not have the same behaviour as their Python +equivalents. The behaviour of ``PyIter_Next`` is different from +``tp_iternext``. +[`Issue 29 `__]. +The behaviour of ``PySet_Contains`` is different from ``set.__contains__`` +[`Issue 6 `__]. + +The fact that ``PyArg_ParseTupleAndKeywords`` takes a non-const +char* array as argument makes it more difficult to use. +[`Issue 28 `__]. + +Python.h does not expose the whole API. Some headers (like marshal.h) +are not included from Python.h. +[`Issue 43 `__]. + + +Missing Functionality +--------------------- + +This section consists of a list of feature requests, i.e., functionality +that was identified as missing in the current C API. + +Debug Mode +~~~~~~~~~~ + +A debug mode that can be activated without recompilation and which +activates various checks that can help detect various types of errors. +[`Issue 36 `__]. + +Introspection +~~~~~~~~~~~~~ + +There aren't currently reliable introspection capabilities for objects +defined in C in the same way as there are for Python objects. +[`Issue 32 `__]. + +Efficient type checking for heap types, similar to what ``Py*_Check`` +can do for a static type. +[`Issue 17 `__]. + +Improved Interaction with Other Languages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Interfacing with other GC based languages, and integrating their +GC with Python's GC. +[`Issue 19 `__]. + +Inject foreign stack frames to the traceback. +[`Issue 18 `__]. + +Concrete strings that can be used in other languages +[`Issue 16 `__]. + +References +========== + +1. `Python/C API Reference Manual `__ +2. `2023 Language Summit Blog Post: Three Talks on the C API `__ +3. `capi-workgroup on GitHub `__ +4. `Irit's Discord post with Core Sprint 2023 slides about C API workgroup `__ +5. `Petr's Discord post with Core Sprint 2023 slides `__ +6. `Steve's Discord post with Core Sprint 2023 slides for Things to Learn from HPy `__ +7. `Victor's slides of Core Sprint 2023 Python C API talk: `__ +8. `The Python's stability promise — Cristián Maureira-Fredes, PySide maintainer `__ +9. `Report on the issues PySide had 5 years ago when switching to the stable ABI: `__ + + +Copyright +========= + +This document has been placed in the public domain. From 7feea4d2c93033480ef597e96659cbbae09b5723 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Mon, 16 Oct 2023 17:17:06 +0100 Subject: [PATCH 02/48] added an email --- peps/pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index bfd9b7bbaf3..bcca6862fbf 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -1,6 +1,6 @@ PEP: 9999 Title: An Evaluation of Python's Public C API -Author: Erlend E. Aasland , Domenico Andreoli <>, Stefan Behnel , Carl Friedrich Bolz-Tereick <>, Simon Cross <>, Steve Dower , Tim Felgentreff <>, David Hewitt <>, Shantanu Jain , Wenzel Jakob <>, Irit Katriel , Marc-Andre Lemburg , Dong-hee Na <>, Karl Nelson <>, Ronald Oussoren , Matti Picus <>, Antoine Pitrou , Benedikt Reinartz <>, Guido van Rossum , Neil Schemenauer , Mark Shannon , Stepan Sindelar <>, Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka <>, Petr Viktorin , Carol Willing , William Woodruff , David Woods <>, Jelle Zijlstra +Author: Erlend E. Aasland , Domenico Andreoli <>, Stefan Behnel , Carl Friedrich Bolz-Tereick , Simon Cross <>, Steve Dower , Tim Felgentreff <>, David Hewitt <>, Shantanu Jain , Wenzel Jakob <>, Irit Katriel , Marc-Andre Lemburg , Dong-hee Na <>, Karl Nelson <>, Ronald Oussoren , Matti Picus <>, Antoine Pitrou , Benedikt Reinartz <>, Guido van Rossum , Neil Schemenauer , Mark Shannon , Stepan Sindelar <>, Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka <>, Petr Viktorin , Carol Willing , William Woodruff , David Woods <>, Jelle Zijlstra Status: Draft Type: Informational Content-Type: text/x-rst From 32f0f8e86aeff9829a36f922983210a7db2107a5 Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Mon, 16 Oct 2023 21:09:09 +0100 Subject: [PATCH 03/48] padlock Co-authored-by: Hugo van Kemenade --- peps/pep-9999.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index bcca6862fbf..3b2af444836 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -618,9 +618,9 @@ References 1. `Python/C API Reference Manual `__ 2. `2023 Language Summit Blog Post: Three Talks on the C API `__ 3. `capi-workgroup on GitHub `__ -4. `Irit's Discord post with Core Sprint 2023 slides about C API workgroup `__ -5. `Petr's Discord post with Core Sprint 2023 slides `__ -6. `Steve's Discord post with Core Sprint 2023 slides for Things to Learn from HPy `__ +4. 🔒 `Irit's Discord post with Core Sprint 2023 slides about C API workgroup `__ +5. 🔒 `Petr's Discord post with Core Sprint 2023 slides `__ +6. 🔒 `Steve's Discord post with Core Sprint 2023 slides for Things to Learn from HPy `__ 7. `Victor's slides of Core Sprint 2023 Python C API talk: `__ 8. `The Python's stability promise — Cristián Maureira-Fredes, PySide maintainer `__ 9. `Report on the issues PySide had 5 years ago when switching to the stable ABI: `__ From 7f47ebe87e2b59bbabe5f6ff2fe46351583eb422 Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Mon, 16 Oct 2023 21:12:41 +0100 Subject: [PATCH 04/48] Apply suggestions from code review Co-authored-by: Hugo van Kemenade --- peps/pep-9999.rst | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 3b2af444836..9c7a48ea0e1 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -58,17 +58,17 @@ time updating it for use cases that it was not originally designed for. However, there was also a sense at the Language Summit that we are trying to discuss solutions without a clear common understanding -of the problems that we are trying to solve. It was decided that +of the problems that we are trying to solve. We decided that we need to agree on the current problems with the C API, before we are able to evaluate any of the proposed solutions. We therefore created the -[`capi-workgroup `__] +`capi-workgroup `__ repository on GitHub in order to collect everyone's ideas on that question. Over 60 different issues were created on that repository, each -describing a problem with the C API. They were categorized and -a number of recurring themes were identified. The sections below +describing a problem with the C API. We categorized them and +identified a number of recurring themes. The sections below mostly correspond to these themes, and each contains a combined description of the issues raised in that category, along with links to the individual issues. In addition, we included a section @@ -82,7 +82,7 @@ C API Stakeholders As mentioned in the introduction, the C API was originally created as the internal interface between CPython's interpreter and the Python layer. It was later exposed as -a way for third party developers to extend and embed Python +a way for third-party developers to extend and embed Python programs. Over the years, new types of stakeholders emerged, with different requirements and areas of focus. This section describes this complex state of affairs in terms of the @@ -97,7 +97,7 @@ all types of API users: * Define functions and call them * Create classes and instantiate them -* Create instances of builtin and user defined types +* Create instances of builtin and user-defined types and perform operations on them * Introspect objects, including types, instances, and functions. * Raise and handle exceptions @@ -108,7 +108,7 @@ all types of API users: The following sections look at the unique requirements of various stakeholders. -Extension writers +Extension Writers ----------------- Extension writers are the traditional users of the C API. Their requirements @@ -123,9 +123,9 @@ Applications with an embedded Python interpreter. Examples are They need to be able to: -* Configure the interpreter (import paths, inittab, sys.argv, ...) +* Configure the interpreter (import paths, inittab, ``sys.argv``, etc.). * Interact with the execution model and program lifetime, including - clean interpreter shutdown and restart + clean interpreter shutdown and restart. * Represent complex data models in a way Python can use without having to create deep copies. * Provide and import frozen modules. @@ -178,7 +178,7 @@ and lower stability guarantees. Then the developers and users of these tools can choose whether to generate code that uses the faster or the safer and more stable version of the API. -Binding generators +Binding Generators ------------------ Libraries that create bindings between Python and other object models, @@ -203,7 +203,7 @@ They need to: * Adapt external mechanisms, execution models and guarantees to the Python way (green threads/continuations, one-writer-or-multiple-readers semantics, virtual multiple inheritance, 1-based indexing, super-long - inheritance chains, goroutines, channels, ...) + inheritance chains, goroutines, channels, etc.). Strengths of the C API ====================== @@ -236,7 +236,8 @@ C API problems ============== The remainder of this document summarizes and categorizes the problems that were reported on -the [`capi-workgroup `__] repository. The issues are grouped into nine categories: +the `capi-workgroup `__ repository. +The issues are grouped into nine categories: - API Evolution and Maintenance - API Specification and Abstraction @@ -313,7 +314,7 @@ occurs in the stable ABI as in the case of ``ob_refcnt`` and ``ob_type``, which are accessed via the reference counting macros [`Issue 45 `__]. -A deeper issue was identified in relation to the way that reference +We identified a deeper issue in relation to the way that reference counting is exposed. The way that C extensions are required to manage references with calls to ``Py_INCREF`` and ``Py_DECREF`` is specific to CPython's memory model, and is hard for alternative @@ -379,7 +380,7 @@ initialized tuple, should also be removed (tuples are immutable once fully initialized) [`Issue 56 `__]. -A few issues were identified with type definition APIs. For legacy +We identified a few issues with type definition APIs. For legacy reasons, there is often a significant amount of code duplication between ``tp_new`` and ``tp_vectorcall`` [`Issue 24 `__]. @@ -399,7 +400,7 @@ on the thread state (in global scope). The design intention was that each API function returns a value indicating whether an error has occurred (by convention, ``-1`` or ``NULL``). When the program knows that an error occurred, it can fetch the exception object which is stored in the -error indicator. A number of problems were identified which are related +error indicator. We identified a number of problems which are related to error handling, pointing at APIs which are too easy to use incorrectly. There are functions that do not report all errors that occur while they @@ -551,8 +552,8 @@ reliable **Naming** -``PyLong`` and ``PyUnicode`` use names which don't match the python -types they represent (int/str). This can be fixed in a new API +``PyLong`` and ``PyUnicode`` use names which don't match the Python +types they represent (``int``/``str``). This can be fixed in a new API [`Issue 14 `__]. There are identifiers in the API which are lacking a ``Py``/``_Py`` From 981088e781581ccabb3f48c75746780c7e995b04 Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Mon, 16 Oct 2023 21:14:55 +0100 Subject: [PATCH 05/48] Apply suggestions from code review Co-authored-by: Hugo van Kemenade --- peps/pep-9999.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 9c7a48ea0e1..2727dae538e 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -99,7 +99,7 @@ all types of API users: * Create classes and instantiate them * Create instances of builtin and user-defined types and perform operations on them -* Introspect objects, including types, instances, and functions. +* Introspect objects, including types, instances, and functions * Raise and handle exceptions * Import modules * Manage threads @@ -568,11 +568,11 @@ The behaviour of ``PySet_Contains`` is different from ``set.__contains__`` [`Issue 6 `__]. The fact that ``PyArg_ParseTupleAndKeywords`` takes a non-const -char* array as argument makes it more difficult to use. +``char*`` array as argument makes it more difficult to use [`Issue 28 `__]. -Python.h does not expose the whole API. Some headers (like marshal.h) -are not included from Python.h. +``Python.h`` does not expose the whole API. Some headers (like ``marshal.h``) +are not included from ``Python.h``. [`Issue 43 `__]. @@ -586,28 +586,28 @@ Debug Mode ~~~~~~~~~~ A debug mode that can be activated without recompilation and which -activates various checks that can help detect various types of errors. +activates various checks that can help detect various types of errors [`Issue 36 `__]. Introspection ~~~~~~~~~~~~~ There aren't currently reliable introspection capabilities for objects -defined in C in the same way as there are for Python objects. +defined in C in the same way as there are for Python objects [`Issue 32 `__]. Efficient type checking for heap types, similar to what ``Py*_Check`` -can do for a static type. +can do for a static type [`Issue 17 `__]. Improved Interaction with Other Languages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interfacing with other GC based languages, and integrating their -GC with Python's GC. +GC with Python's GC [`Issue 19 `__]. -Inject foreign stack frames to the traceback. +Inject foreign stack frames to the traceback [`Issue 18 `__]. Concrete strings that can be used in other languages From 1f4302eda2144670aa13c7182305e79216b6f996 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Mon, 16 Oct 2023 21:19:38 +0100 Subject: [PATCH 06/48] copyright. Update a couple of author emails --- peps/pep-9999.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 2727dae538e..55be5602c7d 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -1,6 +1,6 @@ PEP: 9999 Title: An Evaluation of Python's Public C API -Author: Erlend E. Aasland , Domenico Andreoli <>, Stefan Behnel , Carl Friedrich Bolz-Tereick , Simon Cross <>, Steve Dower , Tim Felgentreff <>, David Hewitt <>, Shantanu Jain , Wenzel Jakob <>, Irit Katriel , Marc-Andre Lemburg , Dong-hee Na <>, Karl Nelson <>, Ronald Oussoren , Matti Picus <>, Antoine Pitrou , Benedikt Reinartz <>, Guido van Rossum , Neil Schemenauer , Mark Shannon , Stepan Sindelar <>, Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka <>, Petr Viktorin , Carol Willing , William Woodruff , David Woods <>, Jelle Zijlstra +Author: Erlend E. Aasland , Domenico Andreoli <>, Stefan Behnel , Carl Friedrich Bolz-Tereick , Simon Cross <>, Steve Dower , Tim Felgentreff , David Hewitt <>, Shantanu Jain , Wenzel Jakob <>, Irit Katriel , Marc-Andre Lemburg , Dong-hee Na <>, Karl Nelson <>, Ronald Oussoren , Matti Picus <>, Antoine Pitrou , Benedikt Reinartz <>, Guido van Rossum , Neil Schemenauer , Mark Shannon , Stepan Sindelar , Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka <>, Petr Viktorin , Carol Willing , William Woodruff , David Woods <>, Jelle Zijlstra Status: Draft Type: Informational Content-Type: text/x-rst @@ -630,4 +630,5 @@ References Copyright ========= -This document has been placed in the public domain. +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From 7c5922440c266b98658c4e55aa3af46e68e76257 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Mon, 16 Oct 2023 21:22:12 +0100 Subject: [PATCH 07/48] pep number --- peps/{pep-9999.rst => pep-0733.rst} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename peps/{pep-9999.rst => pep-0733.rst} (100%) diff --git a/peps/pep-9999.rst b/peps/pep-0733.rst similarity index 100% rename from peps/pep-9999.rst rename to peps/pep-0733.rst From 348cb6de0e6660c0d6230d3aac29f0b611e2c04f Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Mon, 16 Oct 2023 21:23:06 +0100 Subject: [PATCH 08/48] update number --- peps/pep-0733.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 55be5602c7d..3d8c45c468b 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -1,4 +1,4 @@ -PEP: 9999 +PEP: 733 Title: An Evaluation of Python's Public C API Author: Erlend E. Aasland , Domenico Andreoli <>, Stefan Behnel , Carl Friedrich Bolz-Tereick , Simon Cross <>, Steve Dower , Tim Felgentreff , David Hewitt <>, Shantanu Jain , Wenzel Jakob <>, Irit Katriel , Marc-Andre Lemburg , Dong-hee Na <>, Karl Nelson <>, Ronald Oussoren , Matti Picus <>, Antoine Pitrou , Benedikt Reinartz <>, Guido van Rossum , Neil Schemenauer , Mark Shannon , Stepan Sindelar , Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka <>, Petr Viktorin , Carol Willing , William Woodruff , David Woods <>, Jelle Zijlstra Status: Draft From 2da5752b6b01be93c71445f7b65b1102f6ad301c Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Mon, 16 Oct 2023 21:28:50 +0100 Subject: [PATCH 09/48] add codeowners --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 25e077ef219..e5115b2d0eb 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -609,6 +609,7 @@ peps/pep-0727.rst @JelleZijlstra peps/pep-0729.rst @JelleZijlstra @hauntsaninja peps/pep-0730.rst @ned-deily peps/pep-0731.rst @gvanrossum @encukou @vstinner @zooba @iritkatriel +peps/pep-0733.rst @gvanrossum @encukou @vstinner @zooba @iritkatriel # ... # peps/pep-0754.rst # ... From 1510019b828513f5404f6372a1b08fbc99076603 Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Mon, 16 Oct 2023 23:28:16 +0100 Subject: [PATCH 10/48] reword re int types Co-authored-by: Antoine Pitrou --- peps/pep-0733.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 3d8c45c468b..522c90f0b60 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -505,8 +505,8 @@ Usage of ``const`` in the API is currently sparse, but it is not clear whether this is something that we should consider changing [`Issue 38 `__]. -We currently use the C types ``long`` and ``int``, where ``stdint`` -and ``int32_t`` would have been better choices +We currently use the C types ``long`` and ``int``, where fixed-width integers +such as ``int32_t`` and ``int64_t`` would have been better choices [`Issue 27 `__]. We are using C language features which are hard for other languages From f5b749e84b2902ef0532037ed3e60b904b7acb28 Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Mon, 16 Oct 2023 23:35:36 +0100 Subject: [PATCH 11/48] add a name --- peps/pep-0733.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 522c90f0b60..2c60208d61b 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -1,6 +1,6 @@ PEP: 733 Title: An Evaluation of Python's Public C API -Author: Erlend E. Aasland , Domenico Andreoli <>, Stefan Behnel , Carl Friedrich Bolz-Tereick , Simon Cross <>, Steve Dower , Tim Felgentreff , David Hewitt <>, Shantanu Jain , Wenzel Jakob <>, Irit Katriel , Marc-Andre Lemburg , Dong-hee Na <>, Karl Nelson <>, Ronald Oussoren , Matti Picus <>, Antoine Pitrou , Benedikt Reinartz <>, Guido van Rossum , Neil Schemenauer , Mark Shannon , Stepan Sindelar , Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka <>, Petr Viktorin , Carol Willing , William Woodruff , David Woods <>, Jelle Zijlstra +Author: Erlend E. Aasland , Domenico Andreoli <>, Stefan Behnel , Carl Friedrich Bolz-Tereick , Simon Cross <>, Steve Dower , Tim Felgentreff , David Hewitt <>, Shantanu Jain , Wenzel Jakob <>, Irit Katriel , Marc-Andre Lemburg , Donghee Na , Karl Nelson <>, Ronald Oussoren , Matti Picus <>, Antoine Pitrou , Benedikt Reinartz <>, Guido van Rossum , Neil Schemenauer , Mark Shannon , Stepan Sindelar , Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka <>, Petr Viktorin , Carol Willing , William Woodruff , David Woods <>, Jelle Zijlstra Status: Draft Type: Informational Content-Type: text/x-rst From c5ced72882eca6e91648aa568e127e85cb4be10b Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Tue, 17 Oct 2023 12:06:07 +0100 Subject: [PATCH 12/48] Apply suggestions from code review Co-authored-by: Jelle Zijlstra --- peps/pep-0733.rst | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 2c60208d61b..57e23ef9cbe 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -157,7 +157,7 @@ Alternative APIs There are several projects that implement alternatives to the C API, which offer extension users advantanges over programming directly with the C API. These APIs are implemented with the -C API, and in some cases by using cpython internals. +C API, and in some cases by using CPython internals. Some examples are `Cython `__, `HPy `__ and @@ -237,17 +237,7 @@ C API problems The remainder of this document summarizes and categorizes the problems that were reported on the `capi-workgroup `__ repository. -The issues are grouped into nine categories: - -- API Evolution and Maintenance -- API Specification and Abstraction -- Object Reference Management -- Type Definition and Object Creation -- Error Handling -- API Tiers and Stability Guarantees -- Use of the C Language -- Implementation Flaws -- Missing Functionality +The issues are grouped into several categories. API Evolution and Maintenance @@ -510,7 +500,7 @@ such as ``int32_t`` and ``int64_t`` would have been better choices [`Issue 27 `__]. We are using C language features which are hard for other languages -to interact with +to interact with, such as macros, variadic arguments, enums, and bitfields [`Issue 35 `__]. There are API functions that take a ``PyObject*`` arg which must be @@ -522,7 +512,7 @@ more specific type There are functions in the API that take concrete types, such as ``PyDict_GetItemString`` which performs a dictionary lookup for a key -specified as a c string rather than ``PyObject*``. At the same time, +specified as a C string rather than ``PyObject*``. At the same time, for ``PyDict_ContainsString`` it is not considered appropriate to add a concrete type alternative. The principle around this should be documented in the guidelines From 8ebcf5e25166aa6c7dffcbb1eef571469ef2bf29 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Tue, 17 Oct 2023 15:24:23 +0100 Subject: [PATCH 13/48] Address review comments from Jelle and David --- peps/pep-0733.rst | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 57e23ef9cbe..f50565c6704 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -191,6 +191,10 @@ paradigms or languages, such as `PyJNIus `__ for Java, or `SWIG `__ for C/C++. +Some of the alternative APIs mentioned in the previous section create +alternative object models and therefore they can also be seen as +belonging to this category. + They need to: * Create custom objects (e.g. function/module objects @@ -540,16 +544,6 @@ The meaning of ``Py_SIZE`` depends on the type and is not always reliable [`Issue 10 `__]. -**Naming** - -``PyLong`` and ``PyUnicode`` use names which don't match the Python -types they represent (``int``/``str``). This can be fixed in a new API -[`Issue 14 `__]. - -There are identifiers in the API which are lacking a ``Py``/``_Py`` -prefix -[`Issue 46 `__]. - Some API function do not have the same behaviour as their Python equivalents. The behaviour of ``PyIter_Next`` is different from ``tp_iternext``. @@ -565,6 +559,15 @@ The fact that ``PyArg_ParseTupleAndKeywords`` takes a non-const are not included from ``Python.h``. [`Issue 43 `__]. +**Naming** + +``PyLong`` and ``PyUnicode`` use names which don't match the Python +types they represent (``int``/``str``). This can be fixed in a new API +[`Issue 14 `__]. + +There are identifiers in the API which are lacking a ``Py``/``_Py`` +prefix +[`Issue 46 `__]. Missing Functionality --------------------- From c11a4e23cef2ab8c8e6e845583d2f42f6d672544 Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Tue, 17 Oct 2023 15:25:27 +0100 Subject: [PATCH 14/48] Apply suggestions from code review Co-authored-by: Victor Stinner --- peps/pep-0733.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index f50565c6704..f0f394f37d2 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -615,9 +615,9 @@ References 4. 🔒 `Irit's Discord post with Core Sprint 2023 slides about C API workgroup `__ 5. 🔒 `Petr's Discord post with Core Sprint 2023 slides `__ 6. 🔒 `Steve's Discord post with Core Sprint 2023 slides for Things to Learn from HPy `__ -7. `Victor's slides of Core Sprint 2023 Python C API talk: `__ +7. `Victor's slides of Core Sprint 2023 Python C API talk `__ 8. `The Python's stability promise — Cristián Maureira-Fredes, PySide maintainer `__ -9. `Report on the issues PySide had 5 years ago when switching to the stable ABI: `__ +9. `Report on the issues PySide had 5 years ago when switching to the stable ABI `__ Copyright From d4bb55e6cf88a9c3953662343e7405da28e9f606 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 17 Oct 2023 17:34:25 +0200 Subject: [PATCH 15/48] Update peps/pep-0733.rst --- peps/pep-0733.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index f0f394f37d2..78e69f43fad 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -1,6 +1,6 @@ PEP: 733 Title: An Evaluation of Python's Public C API -Author: Erlend E. Aasland , Domenico Andreoli <>, Stefan Behnel , Carl Friedrich Bolz-Tereick , Simon Cross <>, Steve Dower , Tim Felgentreff , David Hewitt <>, Shantanu Jain , Wenzel Jakob <>, Irit Katriel , Marc-Andre Lemburg , Donghee Na , Karl Nelson <>, Ronald Oussoren , Matti Picus <>, Antoine Pitrou , Benedikt Reinartz <>, Guido van Rossum , Neil Schemenauer , Mark Shannon , Stepan Sindelar , Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka <>, Petr Viktorin , Carol Willing , William Woodruff , David Woods <>, Jelle Zijlstra +Author: Erlend E. Aasland , Domenico Andreoli , Stefan Behnel , Carl Friedrich Bolz-Tereick , Simon Cross , Steve Dower , Tim Felgentreff , David Hewitt <1939362+davidhewitt@users.noreply.github.com>, Shantanu Jain , Wenzel Jakob , Irit Katriel , Marc-Andre Lemburg , Donghee Na , Karl Nelson , Ronald Oussoren , Antoine Pitrou , Guido van Rossum , Neil Schemenauer , Mark Shannon , Stepan Sindelar , Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka , Petr Viktorin , Carol Willing , William Woodruff , David Woods , Jelle Zijlstra Status: Draft Type: Informational Content-Type: text/x-rst From 2ae8846b4772a0c6774088f8f5c9ab8755d388bd Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 17 Oct 2023 17:35:44 +0200 Subject: [PATCH 16/48] Update peps/pep-0733.rst --- peps/pep-0733.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 78e69f43fad..30b0bbcd16b 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -188,8 +188,9 @@ paradigms or languages, such as `PySide `__ for Qt, `PyGObject `__ for GTK, `Pygolo `__ for Go, -`PyJNIus `__ for Java, or -`SWIG `__ for C/C++. +`PyJNIus `__ for Java, +`SWIG `__ for C/C++, or +`Python.NET `__ for .NET (C#). Some of the alternative APIs mentioned in the previous section create alternative object models and therefore they can also be seen as From b64c0f4556a57047848bc3aceed704f768eb3346 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 17 Oct 2023 17:40:38 +0200 Subject: [PATCH 17/48] Update peps/pep-0733.rst Co-authored-by: Benedikt Reinartz --- peps/pep-0733.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 30b0bbcd16b..4cc7c917d07 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -505,7 +505,8 @@ such as ``int32_t`` and ``int64_t`` would have been better choices [`Issue 27 `__]. We are using C language features which are hard for other languages -to interact with, such as macros, variadic arguments, enums, and bitfields +to interact with, such as macros, variadic arguments, enums, bitfields, +and non-function symbols [`Issue 35 `__]. There are API functions that take a ``PyObject*`` arg which must be From 332437a097891e291e299d5552d5caad741f23f9 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 18 Oct 2023 11:45:15 +0200 Subject: [PATCH 18/48] Update peps/pep-0733.rst Co-authored-by: Petr Viktorin --- peps/pep-0733.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 4cc7c917d07..2a63bfd0698 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -615,7 +615,7 @@ References 2. `2023 Language Summit Blog Post: Three Talks on the C API `__ 3. `capi-workgroup on GitHub `__ 4. 🔒 `Irit's Discord post with Core Sprint 2023 slides about C API workgroup `__ -5. 🔒 `Petr's Discord post with Core Sprint 2023 slides `__ +5. `Petr's Core Sprint 2023 slides `__ 6. 🔒 `Steve's Discord post with Core Sprint 2023 slides for Things to Learn from HPy `__ 7. `Victor's slides of Core Sprint 2023 Python C API talk `__ 8. `The Python's stability promise — Cristián Maureira-Fredes, PySide maintainer `__ From ccd9909f81fba6d7b5434e80e6faa87906d445d7 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 18 Oct 2023 12:26:36 +0200 Subject: [PATCH 19/48] Update peps/pep-0733.rst --- peps/pep-0733.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 2a63bfd0698..a40f2619369 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -614,7 +614,7 @@ References 1. `Python/C API Reference Manual `__ 2. `2023 Language Summit Blog Post: Three Talks on the C API `__ 3. `capi-workgroup on GitHub `__ -4. 🔒 `Irit's Discord post with Core Sprint 2023 slides about C API workgroup `__ +4. `Irit's Core Sprint 2023 slides about C API workgroup `__ 5. `Petr's Core Sprint 2023 slides `__ 6. 🔒 `Steve's Discord post with Core Sprint 2023 slides for Things to Learn from HPy `__ 7. `Victor's slides of Core Sprint 2023 Python C API talk `__ From ad4020daa22ed772026e265d770bcf57d723670a Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Wed, 18 Oct 2023 11:50:57 +0100 Subject: [PATCH 20/48] Apply suggestions from code review Co-authored-by: Terry Jan Reedy Co-authored-by: Steve Dower --- peps/pep-0733.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index a40f2619369..1d514b5ae97 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -27,7 +27,7 @@ criteria. Introduction ============ -Python's C API was not designed for the purposes it currently fulfils. +Python's C API was not designed for the purposes it currently fulfills. It evolved from what was initially the internal API between the C code of the interpreter and the Python language and libraries. In its first incarnation, it was exposed to make it possible to embed Python into C/C++ @@ -405,8 +405,8 @@ in the dictionary [`Issue 51 `__]. Python code never executes with an in-flight exception (by definition), -and by the same token C API functions should never be called with the error -indicator set. This is currently not checked in most C API functions, and +and typically code using native functions should also be interrupted by +an error being raised. This is not checked in most C API functions, and there are places in the interpreter where error handling code calls a C API function while an exception is set. For example, see the call to ``PyUnicode_FromString`` in the error handler of ``_PyErr_WriteUnraisableMsg`` @@ -501,7 +501,7 @@ clear whether this is something that we should consider changing [`Issue 38 `__]. We currently use the C types ``long`` and ``int``, where fixed-width integers -such as ``int32_t`` and ``int64_t`` would have been better choices +such as ``int32_t`` and ``int64_t`` may now be better choices [`Issue 27 `__]. We are using C language features which are hard for other languages From b5cbad5b6d561c99124d03110e68b95077f73bea Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 18 Oct 2023 17:51:50 +0200 Subject: [PATCH 21/48] Update .github/CODEOWNERS --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index e5115b2d0eb..18d17047d7b 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -609,7 +609,7 @@ peps/pep-0727.rst @JelleZijlstra peps/pep-0729.rst @JelleZijlstra @hauntsaninja peps/pep-0730.rst @ned-deily peps/pep-0731.rst @gvanrossum @encukou @vstinner @zooba @iritkatriel -peps/pep-0733.rst @gvanrossum @encukou @vstinner @zooba @iritkatriel +peps/pep-0733.rst @encukou @vstinner @zooba @iritkatriel # ... # peps/pep-0754.rst # ... From 78601835863ca0093c1f2b98a5690637caee9e1b Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 18 Oct 2023 17:52:01 +0200 Subject: [PATCH 22/48] Update peps/pep-0733.rst --- peps/pep-0733.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 1d514b5ae97..3b6ed63cfaa 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -1,6 +1,6 @@ PEP: 733 Title: An Evaluation of Python's Public C API -Author: Erlend E. Aasland , Domenico Andreoli , Stefan Behnel , Carl Friedrich Bolz-Tereick , Simon Cross , Steve Dower , Tim Felgentreff , David Hewitt <1939362+davidhewitt@users.noreply.github.com>, Shantanu Jain , Wenzel Jakob , Irit Katriel , Marc-Andre Lemburg , Donghee Na , Karl Nelson , Ronald Oussoren , Antoine Pitrou , Guido van Rossum , Neil Schemenauer , Mark Shannon , Stepan Sindelar , Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka , Petr Viktorin , Carol Willing , William Woodruff , David Woods , Jelle Zijlstra +Author: Erlend E. Aasland , Domenico Andreoli , Stefan Behnel , Carl Friedrich Bolz-Tereick , Simon Cross , Steve Dower , Tim Felgentreff , David Hewitt <1939362+davidhewitt@users.noreply.github.com>, Shantanu Jain , Wenzel Jakob , Irit Katriel , Marc-Andre Lemburg , Donghee Na , Karl Nelson , Ronald Oussoren , Antoine Pitrou , Neil Schemenauer , Mark Shannon , Stepan Sindelar , Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka , Petr Viktorin , Carol Willing , William Woodruff , David Woods , Jelle Zijlstra Status: Draft Type: Informational Content-Type: text/x-rst From 29c938a9b78caf7b0e2c8092de1034f82cc9a077 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Wed, 18 Oct 2023 19:30:25 +0100 Subject: [PATCH 23/48] address some of the review comments --- peps/pep-0733.rst | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 3b6ed63cfaa..afde0ba84ef 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -158,8 +158,7 @@ There are several projects that implement alternatives to the C API, which offer extension users advantanges over programming directly with the C API. These APIs are implemented with the C API, and in some cases by using CPython internals. -Some examples are -`Cython `__, +Examples are `HPy `__ and `pythoncapi-compat `__. CPython's DSL for parsing function arguments, the @@ -170,19 +169,20 @@ Such systems need minimal building blocks for accessing CPython efficiently. They don't necessarily need an ergonomic API, because they typically generate code that is not intended to be read by humans. But they do need it to be comprehensive enough so that -they don't need to access internals, while offering them stability, -and without sacrificing performance. +they don't need to access internals, without sacrificing performance. + +These tools might also benefit from a choice between a more stable +and a more performant (possibly lower-level) API. Their users could +then decide whether they can afford to regenerate the code often or +trade some performance for more stability and less maintenance work. -An alternative is to have a fast API tier with less error checking -and lower stability guarantees. Then the developers and users of -these tools can choose whether to generate code that uses the -faster or the safer and more stable version of the API. Binding Generators ------------------ Libraries that create bindings between Python and other object models, paradigms or languages, such as +`Cython `__, `pybind11 `__ for C++11, `PyO3 `__ for Rust, `PySide `__ for Qt, @@ -563,7 +563,7 @@ are not included from ``Python.h``. **Naming** -``PyLong`` and ``PyUnicode`` use names which don't match the Python +``PyLong`` and ``PyUnicode`` use names which no longer match the Python types they represent (``int``/``str``). This can be fixed in a new API [`Issue 14 `__]. @@ -591,8 +591,7 @@ There aren't currently reliable introspection capabilities for objects defined in C in the same way as there are for Python objects [`Issue 32 `__]. -Efficient type checking for heap types, similar to what ``Py*_Check`` -can do for a static type +Efficient type checking for heap types [`Issue 17 `__]. Improved Interaction with Other Languages From 717c4244f8b3b4b6ac307f631e7823bd0b75c846 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Wed, 18 Oct 2023 12:42:10 -0700 Subject: [PATCH 24/48] Update peps/pep-0733.rst Co-authored-by: C.A.M. Gerlach --- peps/pep-0733.rst | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index afde0ba84ef..70a8a0be06c 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -1,6 +1,33 @@ PEP: 733 Title: An Evaluation of Python's Public C API -Author: Erlend E. Aasland , Domenico Andreoli , Stefan Behnel , Carl Friedrich Bolz-Tereick , Simon Cross , Steve Dower , Tim Felgentreff , David Hewitt <1939362+davidhewitt@users.noreply.github.com>, Shantanu Jain , Wenzel Jakob , Irit Katriel , Marc-Andre Lemburg , Donghee Na , Karl Nelson , Ronald Oussoren , Antoine Pitrou , Neil Schemenauer , Mark Shannon , Stepan Sindelar , Gregory P. Smith , Eric Snow , Victor Stinner , Serhiy Storchaka , Petr Viktorin , Carol Willing , William Woodruff , David Woods , Jelle Zijlstra +Author: Erlend E. Aasland , + Domenico Andreoli , + Stefan Behnel , + Carl Friedrich Bolz-Tereick , + Simon Cross , + Steve Dower , + Tim Felgentreff , + David Hewitt <1939362+davidhewitt@users.noreply.github.com>, + Shantanu Jain , + Wenzel Jakob , + Irit Katriel , + Marc-Andre Lemburg , + Donghee Na , + Karl Nelson , + Ronald Oussoren , + Antoine Pitrou , + Neil Schemenauer , + Mark Shannon , + Stepan Sindelar , + Gregory P. Smith , + Eric Snow , + Victor Stinner , + Serhiy Storchaka , + Petr Viktorin , + Carol Willing , + William Woodruff , + David Woods , + Jelle Zijlstra , Status: Draft Type: Informational Content-Type: text/x-rst From bfc627dc838eadcbbacebb1810f6703a8f71ca72 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Thu, 19 Oct 2023 11:12:53 +0100 Subject: [PATCH 25/48] address a couple more review comments --- peps/pep-0733.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 70a8a0be06c..1ae99092afe 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -123,7 +123,7 @@ There are actions which are generic, and required by all types of API users: * Define functions and call them -* Create classes and instantiate them +* Define new types * Create instances of builtin and user-defined types and perform operations on them * Introspect objects, including types, instances, and functions @@ -327,8 +327,8 @@ a C preprocessor/compiler rather than just a parser (such as macros) Furthermore, C header files tend to expose more than what is intended to be part of the public API [`Issue 34 `__]. -In particular, implementation details such as the fields of C structs -can be exposed +In particular, implementation details such as the precise memory +layouts of internal data structures can be exposed [`Issue 22 `__ and :pep:`620`]. This can make API evolution very difficult, in particular when it From 00c0dec13826edaf84ed378e50a0679c1a6e5db4 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Thu, 19 Oct 2023 22:25:35 +0100 Subject: [PATCH 26/48] remove obsolete sentence --- peps/pep-0733.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 1ae99092afe..55505c6a682 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -219,10 +219,6 @@ paradigms or languages, such as `SWIG `__ for C/C++, or `Python.NET `__ for .NET (C#). -Some of the alternative APIs mentioned in the previous section create -alternative object models and therefore they can also be seen as -belonging to this category. - They need to: * Create custom objects (e.g. function/module objects From d014748ad7a683bd91aab9e56760e7e47bc5caf4 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 20 Oct 2023 00:10:03 +0200 Subject: [PATCH 27/48] Update peps/pep-0733.rst --- peps/pep-0733.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 55505c6a682..73c67dd4604 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -189,7 +189,7 @@ Examples are `HPy `__ and `pythoncapi-compat `__. CPython's DSL for parsing function arguments, the -`Argument Clinic `__, +`Argument Clinic `__, can also be seen as belonging to this category of stakeholders. Such systems need minimal building blocks for accessing CPython From 303683a6e8a43afdb44bcb52e47bb8ecfac97e51 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 20 Oct 2023 00:14:50 +0200 Subject: [PATCH 28/48] Update peps/pep-0733.rst --- peps/pep-0733.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 73c67dd4604..69fe246c834 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -210,6 +210,7 @@ Binding Generators Libraries that create bindings between Python and other object models, paradigms or languages, such as `Cython `__, +`cffi `__, `pybind11 `__ for C++11, `PyO3 `__ for Rust, `PySide `__ for Qt, From 69483adad3bd591aa57c515ee58bdebcf6e517c6 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 20 Oct 2023 00:19:21 +0200 Subject: [PATCH 29/48] Update peps/pep-0733.rst --- peps/pep-0733.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 69fe246c834..efac0f55252 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -159,10 +159,11 @@ They need to be able to: * Run multiple independent interpreters (in particular, when embedded in a library that wants to avoid global effects). -Alternative Python Implementations ----------------------------------- +Python Implementations +---------------------- -Alternative implementations of Python (such as +Python implementations such as +`CPython `__, `PyPy `__, `GraalPy `__, `IronPython `__, From cbaa0a369337020835a835268d47eac6ab71fef8 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Fri, 20 Oct 2023 13:23:51 +0100 Subject: [PATCH 30/48] apply some of the review comments from Simon and Steve --- peps/pep-0733.rst | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index efac0f55252..7285cc2ec3f 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -229,7 +229,8 @@ They need to: * Dynamically create objects which are static in traditional C extensions (e.g. classes/modules), and need CPython to manage their state and lifetime. -* Adapt foreign objects (strings, GC'd containers), with low overhead. +* Dynamically adapt foreign objects (strings, GC'd containers), with + low overhead. * Adapt external mechanisms, execution models and guarantees to the Python way (green threads/continuations, one-writer-or-multiple-readers semantics, virtual multiple inheritance, 1-based indexing, super-long @@ -284,13 +285,15 @@ design or implementation remains with us indefinitely. We can take two views on this issue. One is that this is a problem and the solution needs to be baked into any new C API we design, in the form of a -process for incremental API evolution. The other possible approach is that -this is not a problem to be solved, but rather a feature of any API. In this +process for incremental API evolution, which includes deprecation and +removal of API elements. The other possible approach is that this is not +a problem to be solved, but rather a feature of any API. In this view, API evolution should not be incremental, but rather through large redesigns, each of which learns from the mistakes of the past and is not -shackled by backwards compatibility requirements. A compromise approach -is somewhere between these two extremes, fixing issues which are easy -or important enough to tackle incrementally, and leaving others alone. +shackled by backwards compatibility requirements (in the meantime, new +API elements may be added, but nothing can ever be removed). A compromise +approach is somewhere between these two extremes, fixing issues which are +easy or important enough to tackle incrementally, and leaving others alone. The problem we have in CPython is that we don't have an agreed, official approach to API evolution. Different members of the core team are pulling in From 4d849c58dd6bf66597ecd27247737db27a791ec5 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Fri, 20 Oct 2023 15:03:10 +0100 Subject: [PATCH 31/48] add sentence about stable ABI and limited API also not being clearly defined --- peps/pep-0733.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 7285cc2ec3f..50078a2a419 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -316,7 +316,8 @@ API Specification and Abstraction The C API does not have a formal specification, it is described semi-formally in the documentation and exposed through C header -files. This creates a number of problems. +files. This is also true for the limited API and the stable ABI. +This creates a number of problems. Bindings for languages other than C/C++ must parse C code [`Issue 7 `__]. From 05269621fa2b0111104649ccf647ff0314ccd928 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Fri, 20 Oct 2023 15:27:44 +0100 Subject: [PATCH 32/48] review comment from Ronald --- peps/pep-0733.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 50078a2a419..3f375f859e9 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -218,6 +218,7 @@ paradigms or languages, such as `PyGObject `__ for GTK, `Pygolo `__ for Go, `PyJNIus `__ for Java, +`PyObjC `__ for Objective-C, `SWIG `__ for C/C++, or `Python.NET `__ for .NET (C#). @@ -380,8 +381,8 @@ or pointers to parts of the internal structure of an object (such as ``PyBytes_AsString``) [`Issue 57 `__]. In both cases, the reference/pointer is valid for as long as the -owning object is alive, but this time is hard to reason about. Such -functions should not exist in the API without a mechanism that can +owning object holds the reference, but this time is hard to reason about. +Such functions should not exist in the API without a mechanism that can make them safe. For containers, the API is currently missing bulk operations on the From bd51aa312d91afe763955682e63f70860d48cd56 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Fri, 20 Oct 2023 18:35:02 +0200 Subject: [PATCH 33/48] Update peps/pep-0733.rst --- peps/pep-0733.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 3f375f859e9..fb9e876bd74 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -1,6 +1,6 @@ PEP: 733 Title: An Evaluation of Python's Public C API -Author: Erlend E. Aasland , +Author: Erlend Egeberg Aasland , Domenico Andreoli , Stefan Behnel , Carl Friedrich Bolz-Tereick , From e54c36e2d72294e9a97b93410f213deb23a9d81d Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Mon, 23 Oct 2023 15:22:20 +0100 Subject: [PATCH 34/48] apply a couple more review comments --- peps/pep-0733.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index fb9e876bd74..00a2b63fe56 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -116,8 +116,8 @@ describes this complex state of affairs in terms of the actions that different stakeholders need to perform through the C API. -Universal Actions for All Stakeholders --------------------------------------- +Common Actions for All Stakeholders +----------------------------------- There are actions which are generic, and required by all types of API users: @@ -129,9 +129,8 @@ all types of API users: * Introspect objects, including types, instances, and functions * Raise and handle exceptions * Import modules -* Manage threads * Manage sub-interpreters -* Handle and send signals +* Access to Python's OS interface The following sections look at the unique requirements of various stakeholders. @@ -139,7 +138,7 @@ Extension Writers ----------------- Extension writers are the traditional users of the C API. Their requirements -are the universal actions listed above. +are the common actions listed above. Authors of Embedded Python Applications --------------------------------------- From a0029cbf550980902898a07aa97b6c3186469863 Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Thu, 26 Oct 2023 18:59:52 +0100 Subject: [PATCH 35/48] Update peps/pep-0733.rst Co-authored-by: Guido van Rossum --- peps/pep-0733.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 00a2b63fe56..5f934b93079 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -23,6 +23,7 @@ Author: Erlend Egeberg Aasland , Eric Snow , Victor Stinner , Serhiy Storchaka , + Guido van Rossum , Petr Viktorin , Carol Willing , William Woodruff , From 9a724472fa9912881a2eb9cc4ac8fcbe6625b63d Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Thu, 26 Oct 2023 19:13:07 +0100 Subject: [PATCH 36/48] Petr's compromise --- peps/pep-0733.rst | 52 ++++++++++++++++++++++++----------------------- 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 5f934b93079..20cc4c78dac 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -179,40 +179,25 @@ different subsystems. They need: * It would be nice to have an ABI that can be shared across Python implementations. -Alternative APIs ----------------- +Alternative APIs and Binding Generators +--------------------------------------- There are several projects that implement alternatives to the C API, which offer extension users advantanges over programming directly with the C API. These APIs are implemented with the C API, and in some cases by using CPython internals. -Examples are -`HPy `__ and -`pythoncapi-compat `__. -CPython's DSL for parsing function arguments, the -`Argument Clinic `__, -can also be seen as belonging to this category of stakeholders. - -Such systems need minimal building blocks for accessing CPython -efficiently. They don't necessarily need an ergonomic API, because -they typically generate code that is not intended to be read -by humans. But they do need it to be comprehensive enough so that -they don't need to access internals, without sacrificing performance. - -These tools might also benefit from a choice between a more stable -and a more performant (possibly lower-level) API. Their users could -then decide whether they can afford to regenerate the code often or -trade some performance for more stability and less maintenance work. +There are also libraries that create bindings between Python and +other object models, paradigms or languages. -Binding Generators ------------------- +There is overlap between these categories: binding generators +usually provide alternative APIs, and vice versa. -Libraries that create bindings between Python and other object models, -paradigms or languages, such as +Examples are `Cython `__, `cffi `__, -`pybind11 `__ for C++11, +`pybind11 `__ and +`nanobind `__ for C++, `PyO3 `__ for Rust, `PySide `__ for Qt, `PyGObject `__ for GTK, @@ -221,8 +206,19 @@ paradigms or languages, such as `PyObjC `__ for Objective-C, `SWIG `__ for C/C++, or `Python.NET `__ for .NET (C#). +`HPy `__ and +`pythoncapi-compat `__. +CPython's DSL for parsing function arguments, the +`Argument Clinic `__, +can also be seen as belonging to this category of stakeholders. + +Alternative APIs need minimal building blocks for accessing CPython +efficiently. They don't necessarily need an ergonomic API, because +they typically generate code that is not intended to be read +by humans. But they do need it to be comprehensive enough so that +they can avoid accessing internals, without sacrificing performance. -They need to: +Binding generators often need to: * Create custom objects (e.g. function/module objects and traceback entries) that match the behavior of equivalent @@ -237,6 +233,12 @@ They need to: semantics, virtual multiple inheritance, 1-based indexing, super-long inheritance chains, goroutines, channels, etc.). +These tools might also benefit from a choice between a more stable +and a more performant (possibly lower-level) API. Their users could +then decide whether they can afford to regenerate the code often or +trade some performance for more stability and less maintenance work. + + Strengths of the C API ====================== From d95b882a0922959e30393afeeb16d218caa3acfa Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Fri, 27 Oct 2023 13:10:09 +0100 Subject: [PATCH 37/48] reviews from Greg and Mark --- peps/pep-0733.rst | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 20cc4c78dac..0d7dbb34330 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -130,7 +130,6 @@ all types of API users: * Introspect objects, including types, instances, and functions * Raise and handle exceptions * Import modules -* Manage sub-interpreters * Access to Python's OS interface The following sections look at the unique requirements of various stakeholders. @@ -156,8 +155,8 @@ They need to be able to: * Represent complex data models in a way Python can use without having to create deep copies. * Provide and import frozen modules. -* Run multiple independent interpreters (in particular, when embedded - in a library that wants to avoid global effects). +* Run and manage multiple independent interpreters (in particular, when + embedded in a library that wants to avoid global effects). Python Implementations ---------------------- @@ -229,12 +228,13 @@ Binding generators often need to: * Dynamically adapt foreign objects (strings, GC'd containers), with low overhead. * Adapt external mechanisms, execution models and guarantees to the - Python way (green threads/continuations, one-writer-or-multiple-readers - semantics, virtual multiple inheritance, 1-based indexing, super-long - inheritance chains, goroutines, channels, etc.). + Python way (stackful coroutines, continuations, + one-writer-or-multiple-readers semantics, virtual multiple inheritance, + 1-based indexing, super-long inheritance chains, goroutines, channels, + etc.). These tools might also benefit from a choice between a more stable -and a more performant (possibly lower-level) API. Their users could +and a faster (possibly lower-level) API. Their users could then decide whether they can afford to regenerate the code often or trade some performance for more stability and less maintenance work. @@ -262,8 +262,8 @@ API functions that take a C string literal for lookups based on a Python string are very convenient [`Issue 30 `__]. -The Limited API and stable ABI hide implementation details and -make it easier to evolve Python +The limited API demonstrates that an API which hides implementation +details makes it easier to evolve Python [`Issue 30 `__]. C API problems @@ -367,7 +367,7 @@ Object Reference Management --------------------------- There are C API functions that return borrowed references, and -functions that steal references to arguments, but there isn't a +functions that decref arguments, but there isn't a naming convention that makes this obvious, so this is error prone [`Issue 8 `__ and `Issue 52 `__]. @@ -375,8 +375,8 @@ The terminology used to describe these situations in the documentation can also be improved [`Issue 11 `__]. -A more radical change is necessary in the case of functions that -return borrowed references (such as ``PyList_GetItem``) +A more radical change is necessary in the case of a function that +returns a ``PyObject*`` that it did not incref (such as ``PyList_GetItem``) [`Issue 5 `__ and `Issue 21 `__] or pointers to parts of the internal structure of an object @@ -525,7 +525,8 @@ Use of the C Language A number of issues were raised with respect to the way that CPython uses the C language. First there is the issue of which C dialect -we use, and how we test our compatibility with it +we use, and how we test our compatibility with it, as well as API +header compatibility with C++ dialects [`Issue 42 `__]. Usage of ``const`` in the API is currently sparse, but it is not From 9cd58b9cc62ccbfac82e1e12c93731e3157e7cfb Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Fri, 27 Oct 2023 17:39:19 +0100 Subject: [PATCH 38/48] reword reference management paragraph --- peps/pep-0733.rst | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 0d7dbb34330..74f1aa64183 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -366,17 +366,26 @@ different memory management schemes Object Reference Management --------------------------- -There are C API functions that return borrowed references, and -functions that decref arguments, but there isn't a -naming convention that makes this obvious, so this is error prone +There does not exist a consistent naming convention for functions +which makes their reference semantics obvious, and this leads to +error prone C API functions, where they do not follow the typical +behaviour. When a C API function returns a ``PyObject*``, the +caller typically gains ownership of a reference to the object. +However, there are exceptions where a function returns a +"borrowed" reference, which the caller can access but does not own +a reference to. Similarly, functions typically do not change the +ownership of references to their arguments, but there are +exceptions where a function "steals" a reference, i.e., the +ownership of the reference is permanently transferred from the +caller to the callee by the call [`Issue 8 `__ and `Issue 52 `__]. The terminology used to describe these situations in the documentation can also be improved [`Issue 11 `__]. -A more radical change is necessary in the case of a function that -returns a ``PyObject*`` that it did not incref (such as ``PyList_GetItem``) +A more radical change is necessary in the case of functions that +return "borrowed" references (such as ``PyList_GetItem``) [`Issue 5 `__ and `Issue 21 `__] or pointers to parts of the internal structure of an object From b6d7da9aa2caa9ee7d138dbc12bee2fb62b8d403 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Fri, 27 Oct 2023 22:17:21 +0100 Subject: [PATCH 39/48] mention Pythran and Mypyc --- peps/pep-0733.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 74f1aa64183..7d74339634b 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -203,9 +203,11 @@ Examples are `Pygolo `__ for Go, `PyJNIus `__ for Java, `PyObjC `__ for Objective-C, -`SWIG `__ for C/C++, or -`Python.NET `__ for .NET (C#). -`HPy `__ and +`SWIG `__ for C/C++, +`Python.NET `__ for .NET (C#), +`HPy `__, +`Mypyc `__, +`Pythran `__ and `pythoncapi-compat `__. CPython's DSL for parsing function arguments, the `Argument Clinic `__, From 5bb87df0827825f8ea2c686eeb074b6e888880d6 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Mon, 30 Oct 2023 18:20:57 +0000 Subject: [PATCH 40/48] MAL's review --- peps/pep-0733.rst | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 7d74339634b..b3415b29973 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -55,20 +55,23 @@ criteria. Introduction ============ -Python's C API was not designed for the purposes it currently fulfills. -It evolved from what was initially the internal API between the C code -of the interpreter and the Python language and libraries. In its first -incarnation, it was exposed to make it possible to embed Python into C/C++ -applications and to write extension modules in C/C++. +Python's C API was not designed for the different purposes it currently +fulfills. It evolved from what was initially the internal API between +the C code of the interpreter and the Python language and libraries. +In its first incarnation, it was exposed to make it possible to embed +Python into C/C++ applications and to write extension modules in C/C++. These capabilities were instrumental to the growth of Python's ecosystem. Over the decades, the C API grew to provide different tiers of stability, conventions changed, and new usage patterns have emerged, such as bindings -to languages other than C/C++. In addition, CPython is no longer the only -implementation of the C API, and some of the design decisions made when -it was, are difficult for alternative implementations to work with +to languages other than C/C++. However, this growth was not supported by +clearly documented guidelines, resulting in inconsitent approaches to +API design in different subsystems of CPython. In addition, CPython is +no longer the only implementation of Python, and some of the design +decisions made when it was, are difficult for alternative implementations +to work with [`Issue 64 `__]. -Finally, lessons were learned and mistakes in both the design and the -implementation of the C API were identified. +In the meantime, lessons were learned and mistakes in both the design +and the implementation of the C API were identified. Evolving the C API is hard due to the combination of backwards compatibility constraints and its inherent complexity, both @@ -81,8 +84,9 @@ or replacement of the C API, each representing a deep analysis of the problems. At the 2023 Language Summit, three back-to-back sessions were devoted to different aspects of the C API. There is general agreement that a new design can remedy the problems that -the C API has accumulated over the last 30 years, while at the same -time updating it for use cases that it was not originally designed for. +the C API has accumulated over the last 30 years, while at the +same time updating it for use cases that it was not originally +designed for. However, there was also a sense at the Language Summit that we are trying to discuss solutions without a clear common understanding @@ -138,7 +142,11 @@ Extension Writers ----------------- Extension writers are the traditional users of the C API. Their requirements -are the common actions listed above. +are the common actions listed above. They also commonly need to: + +* Create new modules +* Efficiently interface between modules at the C level + Authors of Embedded Python Applications --------------------------------------- @@ -149,7 +157,8 @@ Applications with an embedded Python interpreter. Examples are They need to be able to: -* Configure the interpreter (import paths, inittab, ``sys.argv``, etc.). +* Configure the interpreter (import paths, inittab, ``sys.argv``, memory + allocator, etc.). * Interact with the execution model and program lifetime, including clean interpreter shutdown and restart. * Represent complex data models in a way Python can use without From abfd8a1b918de00b6d582a04ffa1cf8905b732ce Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Mon, 30 Oct 2023 18:31:34 +0000 Subject: [PATCH 41/48] more of MAL's review --- peps/pep-0733.rst | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index b3415b29973..3e992c86383 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -419,12 +419,11 @@ Type Definition and Object Creation The C API has functions that make it possible to create incomplete or inconsistent Python objects, such as ``PyTuple_New`` and -``PyUnicode_New``. This causes problem when the object is tracked +``PyUnicode_New``. This causes problems when the object is tracked by GC or its ``tp_traverse``/``tp_clear`` functions are called. -Such functions should be removed from the C API. Related functions, -such as ``PyTuple_SetItem`` which is used to modify a partially -initialized tuple, should also be removed (tuples are immutable -once fully initialized) +A related issue is with functions such as ``PyTuple_SetItem`` +which is used to modify a partially initialized tuple (tuples +are immutable once fully initialized) [`Issue 56 `__]. We identified a few issues with type definition APIs. For legacy @@ -617,7 +616,7 @@ are not included from ``Python.h``. **Naming** ``PyLong`` and ``PyUnicode`` use names which no longer match the Python -types they represent (``int``/``str``). This can be fixed in a new API +types they represent (``int``/``str``). This could be fixed in a new API [`Issue 14 `__]. There are identifiers in the API which are lacking a ``Py``/``_Py`` From 97bb32abfc92889198b2f9ec9869e4e989c618a2 Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Mon, 30 Oct 2023 18:36:31 +0000 Subject: [PATCH 42/48] mention nogil and jit --- peps/pep-0733.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 3e992c86383..afc97a52193 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -63,12 +63,14 @@ Python into C/C++ applications and to write extension modules in C/C++. These capabilities were instrumental to the growth of Python's ecosystem. Over the decades, the C API grew to provide different tiers of stability, conventions changed, and new usage patterns have emerged, such as bindings -to languages other than C/C++. However, this growth was not supported by -clearly documented guidelines, resulting in inconsitent approaches to -API design in different subsystems of CPython. In addition, CPython is -no longer the only implementation of Python, and some of the design -decisions made when it was, are difficult for alternative implementations -to work with +to languages other than C/C++. In the next few years, new developments +are expected to further test the C API, such as the removal of the GIL +and the development of a JIT compiler. However, this growth was not +supported by clearly documented guidelines, resulting in inconsitent +approaches to API design in different subsystems of CPython. In addition, +CPython is no longer the only implementation of Python, and some of the +design decisions made when it was, are difficult for alternative +implementations to work with [`Issue 64 `__]. In the meantime, lessons were learned and mistakes in both the design and the implementation of the C API were identified. From 3a7ad9ecee97401008d55b332566a5ba52a5f229 Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Mon, 30 Oct 2023 18:43:04 +0000 Subject: [PATCH 43/48] Update peps/pep-0733.rst Co-authored-by: Jelle Zijlstra --- peps/pep-0733.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index afc97a52193..5e3b4826dcf 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -212,6 +212,7 @@ Examples are `PySide `__ for Qt, `PyGObject `__ for GTK, `Pygolo `__ for Go, +`JPype `__ and `PyJNIus `__ for Java, `PyObjC `__ for Objective-C, `SWIG `__ for C/C++, From 5c431ebe2f54ec8814e638c4b92d47c5dc554d0a Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Mon, 30 Oct 2023 18:44:44 +0000 Subject: [PATCH 44/48] PyJNIus is for Android --- peps/pep-0733.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 5e3b4826dcf..d6fcb2a3390 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -212,8 +212,8 @@ Examples are `PySide `__ for Qt, `PyGObject `__ for GTK, `Pygolo `__ for Go, -`JPype `__ and -`PyJNIus `__ for Java, +`JPype `__ for Java, +`PyJNIus `__ for Android, `PyObjC `__ for Objective-C, `SWIG `__ for C/C++, `Python.NET `__ for .NET (C#), From b17866ac235a87d89e373f2180881f97a57d291d Mon Sep 17 00:00:00 2001 From: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Tue, 31 Oct 2023 14:55:07 +0000 Subject: [PATCH 45/48] HPy reference Co-authored-by: Stepan Sindelar --- peps/pep-0733.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index d6fcb2a3390..97fe2c19b33 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -670,7 +670,7 @@ References 3. `capi-workgroup on GitHub `__ 4. `Irit's Core Sprint 2023 slides about C API workgroup `__ 5. `Petr's Core Sprint 2023 slides `__ -6. 🔒 `Steve's Discord post with Core Sprint 2023 slides for Things to Learn from HPy `__ +6. `HPy team's Core Sprint 2023 slides for Things to Learn from HPy `__ 7. `Victor's slides of Core Sprint 2023 Python C API talk `__ 8. `The Python's stability promise — Cristián Maureira-Fredes, PySide maintainer `__ 9. `Report on the issues PySide had 5 years ago when switching to the stable ABI `__ From 05dd7d2605a0ebec36ffc6111469c6492b83f09c Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Tue, 31 Oct 2023 19:18:41 +0000 Subject: [PATCH 46/48] split sentence --- peps/pep-0733.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 97fe2c19b33..4d6e3a59b31 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -132,7 +132,7 @@ all types of API users: * Define functions and call them * Define new types * Create instances of builtin and user-defined types - and perform operations on them +* Perform operations on object instances * Introspect objects, including types, instances, and functions * Raise and handle exceptions * Import modules From 20408e1303eb58eda46568c8dcbfd5bb3eff5f5e Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Tue, 31 Oct 2023 19:19:29 +0000 Subject: [PATCH 47/48] remove one author --- peps/pep-0733.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 4d6e3a59b31..9994c581b03 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -22,7 +22,6 @@ Author: Erlend Egeberg Aasland , Gregory P. Smith , Eric Snow , Victor Stinner , - Serhiy Storchaka , Guido van Rossum , Petr Viktorin , Carol Willing , From d3e45fda21fe678da2ddb2c6db52033a4e76134c Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Tue, 31 Oct 2023 23:22:23 +0000 Subject: [PATCH 48/48] reworded paragraph on specification --- peps/pep-0733.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 9994c581b03..f052f375ad0 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -330,10 +330,13 @@ or a new API tier of "blessed" functions API Specification and Abstraction --------------------------------- -The C API does not have a formal specification, it is described -semi-formally in the documentation and exposed through C header -files. This is also true for the limited API and the stable ABI. -This creates a number of problems. +The C API does not have a formal specification, it is currently defined +as whatever the reference implementation (CPython) contains in a +particular version. The documentation acts as an incomplete description, +which is not sufficient for verifying the correctness of either the full +API, the limited API, or the stable ABI. As a result, the C API may +change significantly between releases without needing a more visible +specification update, and this leads to a number of problems. Bindings for languages other than C/C++ must parse C code [`Issue 7 `__].