From 1ff96cce9da9fc061d2999d88540a5353f5d1571 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Thu, 1 Aug 2024 17:18:55 +1000 Subject: [PATCH 1/9] gh-70870: Clarify dual usage of 'free variable' --- Doc/c-api/code.rst | 8 +++++--- Doc/glossary.rst | 13 +++++++++++++ Doc/library/dis.rst | 14 +++++++------- Doc/library/functions.rst | 7 ++++--- Doc/library/symtable.rst | 5 +++-- Doc/library/types.rst | 2 +- Doc/reference/datamodel.rst | 13 +++++++++---- Doc/reference/executionmodel.rst | 4 +++- .../2024-08-01-17-18-21.gh-issue-70870.fZnBM9.rst | 3 +++ 9 files changed, 48 insertions(+), 21 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2024-08-01-17-18-21.gh-issue-70870.fZnBM9.rst diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst index 968c472219c643..f42a4fa57fce86 100644 --- a/Doc/c-api/code.rst +++ b/Doc/c-api/code.rst @@ -32,11 +32,12 @@ bound into a function. .. c:function:: Py_ssize_t PyCode_GetNumFree(PyCodeObject *co) - Return the number of free variables in a code object. + Return the number of :attr:`free variables ` in a code object. .. c:function:: int PyUnstable_Code_GetFirstFree(PyCodeObject *co) - Return the position of the first free variable in a code object. + Return the position of the first :attr:`free variable ` + in a code object. .. versionchanged:: 3.13 @@ -144,7 +145,8 @@ bound into a function. Equivalent to the Python code ``getattr(co, 'co_freevars')``. Returns a new reference to a :c:type:`PyTupleObject` containing the names of - the free variables. On error, ``NULL`` is returned and an exception is raised. + the free :term:`closure variables `. On error, ``NULL`` is returned + and an exception is raised. .. versionadded:: 3.11 diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 281dde30dc78ed..ca9c50a12b3225 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -226,6 +226,12 @@ Glossary A variable defined in a class and intended to be modified only at class level (i.e., not in an instance of the class). + closure variable + A variable referenced from a :term:`nested scope` that is defined in an outer scope + rather than being resolved at runtime from the globals or builtin namespaces. + May be explicitly defined with the :keyword:`nonlocal` keyword to allow write access, + or implicitly defined if the variable is only read without being written to. + complex number An extension of the familiar real number system in which all numbers are expressed as a sum of a real part and an imaginary part. Imaginary @@ -444,6 +450,13 @@ Glossary the :term:`global interpreter lock` which allows only one thread to execute Python bytecode at a time. See :pep:`703`. + free variable + Formally, as defined in the :ref:`language execution model `, a free + variable is any variable used in a function which is not a local variable of that + function. + Pragmatically, due to the name of the :attr:`codeobject.co_freevars` attribute, + the term is also frequently used as a synonym for :term:`closure variable`. + function A series of statements which returns some value to a caller. It can also be passed zero or more :term:`arguments ` which may be used in diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 26b13c87181000..e19f73eee1188f 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -1397,7 +1397,7 @@ iterations of the loop. slot ``i`` of the "fast locals" storage in this mapping. If the name is not found there, loads it from the cell contained in slot ``i``, similar to :opcode:`LOAD_DEREF`. This is used for loading - free variables in class bodies (which previously used + :term:`closure variables ` in class bodies (which previously used :opcode:`!LOAD_CLASSDEREF`) and in :ref:`annotation scopes ` within class bodies. @@ -1426,8 +1426,8 @@ iterations of the loop. .. opcode:: COPY_FREE_VARS (n) - Copies the ``n`` free variables from the closure into the frame. - Removes the need for special code on the caller's side when calling + Copies the ``n`` free :term:`closure variables ` from the closure into + the frame. Removes the need for special code on the caller's side when calling closures. .. versionadded:: 3.11 @@ -1894,10 +1894,10 @@ instructions: .. data:: hasfree - Sequence of bytecodes that access a free variable. 'free' in this - context refers to names in the current scope that are referenced by inner - scopes or names in outer scopes that are referenced from this scope. It does - *not* include references to global or builtin scopes. + Sequence of bytecodes that access a free :term:`closure variable`. + 'free' in this context refers to names in the current scope that are + referenced by inner scopes or names in outer scopes that are referenced + from this scope. It does *not* include references to global or builtin scopes. .. data:: hasname diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 2c649376efee70..772a0b5677f9c4 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -682,9 +682,10 @@ are always available. They are listed here in alphabetical order. ``__builtins__`` dictionary into *globals* before passing it to :func:`exec`. The *closure* argument specifies a closure--a tuple of cellvars. - It's only valid when the *object* is a code object containing free variables. - The length of the tuple must exactly match the number of free variables - referenced by the code object. + It's only valid when the *object* is a code object containing free + :term:`closure variables `. + The length of the tuple must exactly match the length of the code object'S + :attr:`~codeobject.co_freevars` attribute. .. audit-event:: exec code_object exec diff --git a/Doc/library/symtable.rst b/Doc/library/symtable.rst index 8ebcb3bcf1b7b4..930eb986617aec 100644 --- a/Doc/library/symtable.rst +++ b/Doc/library/symtable.rst @@ -167,11 +167,12 @@ Examining Symbol Tables .. method:: get_nonlocals() - Return a tuple containing names of nonlocals in this function. + Return a tuple containing names of explicitly declared nonlocals in this function. .. method:: get_frees() - Return a tuple containing names of free variables in this function. + Return a tuple containing names of free :term:`closure variables ` + in this function. .. class:: Class diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 116868c24be864..3aac9f91daeef5 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -199,7 +199,7 @@ Standard names are defined for the following types: .. data:: CellType The type for cell objects: such objects are used as containers for - a function's free variables. + a function's free :term:`closure variables `. .. versionadded:: 3.8 diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 2576f9a07284eb..16f2678cc09bfa 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -560,8 +560,9 @@ Special read-only attributes in which the function was defined. * - .. attribute:: function.__closure__ - - ``None`` or a :class:`tuple` of cells that contain bindings for the - function's free variables. + - ``None`` or a :class:`tuple` of cells that contain bindings for the names specified + in the :attr:`~codeobject.co_freevars` attribute of the function's + :attr:`code object `. A cell object has the attribute ``cell_contents``. This can be used to get the value of the cell, as well as set the value. @@ -1164,10 +1165,14 @@ Special read-only attributes * - .. attribute:: codeobject.co_cellvars - A :class:`tuple` containing the names of :ref:`local variables ` - that are referenced by nested functions inside the function + that are referenced from at least one :term:`nested scope` inside the function * - .. attribute:: codeobject.co_freevars - - A :class:`tuple` containing the names of free variables in the function + - A :class:`tuple` containing the names of free + :term:`closure variables ` that a :term:`nested scope` references + in an outer scope. See also :attr:`function.__closure__`. + + Note: references to global and builtin names are *not* included. * - .. attribute:: codeobject.co_code - A string representing the sequence of :term:`bytecode` instructions in diff --git a/Doc/reference/executionmodel.rst b/Doc/reference/executionmodel.rst index f24e1537af39ed..9ef328cfd0a4ea 100644 --- a/Doc/reference/executionmodel.rst +++ b/Doc/reference/executionmodel.rst @@ -90,7 +90,7 @@ If a name is bound in a block, it is a local variable of that block, unless declared as :keyword:`nonlocal` or :keyword:`global`. If a name is bound at the module level, it is a global variable. (The variables of the module code block are local and global.) If a variable is used in a code block but not -defined there, it is a :dfn:`free variable`. +defined there, it is a :term:`free variable`. Each occurrence of a name in the program text refers to the :dfn:`binding` of that name established by the following name resolution rules. @@ -330,6 +330,8 @@ enclosing namespace, but in the global namespace. [#]_ The :func:`exec` and :func:`eval` functions have optional arguments to override the global and local namespace. If only one namespace is specified, it is used for both. +.. XXX(ncoghlan) above is only accurate for string execution. When executing code objects, + closure cells may now be passed explicitly to resolve co_freevars references. .. _exceptions: diff --git a/Misc/NEWS.d/next/Documentation/2024-08-01-17-18-21.gh-issue-70870.fZnBM9.rst b/Misc/NEWS.d/next/Documentation/2024-08-01-17-18-21.gh-issue-70870.fZnBM9.rst new file mode 100644 index 00000000000000..ba607bf7a42cd9 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2024-08-01-17-18-21.gh-issue-70870.fZnBM9.rst @@ -0,0 +1,3 @@ +Clarified the dual usage of the term "free variable" (both the formal +meaning of any reference to names defined outside the local scope, and the +narrower pragmatic meaning of nonlocal variables named in ``co_freevars``). From fb0f8a046923d826dbd1cc6ea0a370614e3d639e Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Thu, 1 Aug 2024 17:29:35 +1000 Subject: [PATCH 2/9] Fix indentation --- Doc/reference/datamodel.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 16f2678cc09bfa..877ebb8045d7a4 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -561,8 +561,8 @@ Special read-only attributes * - .. attribute:: function.__closure__ - ``None`` or a :class:`tuple` of cells that contain bindings for the names specified - in the :attr:`~codeobject.co_freevars` attribute of the function's - :attr:`code object `. + in the :attr:`~codeobject.co_freevars` attribute of the function's + :attr:`code object `. A cell object has the attribute ``cell_contents``. This can be used to get the value of the cell, as well as set the value. From 9797c1b5b361c4a9a24b42319b39c9798e9bd7d5 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Sat, 3 Aug 2024 17:43:56 +1000 Subject: [PATCH 3/9] Tweaks after reviewing PR preview --- Doc/c-api/code.rst | 7 ++++--- Doc/glossary.rst | 8 +++++--- Doc/library/dis.rst | 6 +++--- Doc/library/symtable.rst | 2 +- Doc/library/types.rst | 2 +- 5 files changed, 14 insertions(+), 11 deletions(-) diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst index f42a4fa57fce86..be559b8e2e432e 100644 --- a/Doc/c-api/code.rst +++ b/Doc/c-api/code.rst @@ -32,11 +32,12 @@ bound into a function. .. c:function:: Py_ssize_t PyCode_GetNumFree(PyCodeObject *co) - Return the number of :attr:`free variables ` in a code object. + Return the number of free :term:`free (closure) variables ` + in a code object. .. c:function:: int PyUnstable_Code_GetFirstFree(PyCodeObject *co) - Return the position of the first :attr:`free variable ` + Return the position of the first :term:`free (closure) variable ` in a code object. .. versionchanged:: 3.13 @@ -145,7 +146,7 @@ bound into a function. Equivalent to the Python code ``getattr(co, 'co_freevars')``. Returns a new reference to a :c:type:`PyTupleObject` containing the names of - the free :term:`closure variables `. On error, ``NULL`` is returned + the :term:`free (closure) variables `. On error, ``NULL`` is returned and an exception is raised. .. versionadded:: 3.11 diff --git a/Doc/glossary.rst b/Doc/glossary.rst index ca9c50a12b3225..de5e6ef9753a00 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -231,6 +231,8 @@ Glossary rather than being resolved at runtime from the globals or builtin namespaces. May be explicitly defined with the :keyword:`nonlocal` keyword to allow write access, or implicitly defined if the variable is only read without being written to. + Due to the name of the :attr:`codeobject.co_freevars` attribute, sometimes instead + referred to as a :term:`free variable`. complex number An extension of the familiar real number system in which all numbers are @@ -452,10 +454,10 @@ Glossary free variable Formally, as defined in the :ref:`language execution model `, a free - variable is any variable used in a function which is not a local variable of that - function. + variable is any variable used in a namespace which is not a local variable in that + namespace. Pragmatically, due to the name of the :attr:`codeobject.co_freevars` attribute, - the term is also frequently used as a synonym for :term:`closure variable`. + the term is also sometimes used as a synonym for :term:`closure variable`. function A series of statements which returns some value to a caller. It can also diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index e19f73eee1188f..ea4056a10fc041 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -1426,8 +1426,8 @@ iterations of the loop. .. opcode:: COPY_FREE_VARS (n) - Copies the ``n`` free :term:`closure variables ` from the closure into - the frame. Removes the need for special code on the caller's side when calling + Copies the ``n`` :term:`free (closure) variables ` from the closure + into the frame. Removes the need for special code on the caller's side when calling closures. .. versionadded:: 3.11 @@ -1894,7 +1894,7 @@ instructions: .. data:: hasfree - Sequence of bytecodes that access a free :term:`closure variable`. + Sequence of bytecodes that access a :term:`free (closure) variable `. 'free' in this context refers to names in the current scope that are referenced by inner scopes or names in outer scopes that are referenced from this scope. It does *not* include references to global or builtin scopes. diff --git a/Doc/library/symtable.rst b/Doc/library/symtable.rst index 930eb986617aec..54e19af4bd69a6 100644 --- a/Doc/library/symtable.rst +++ b/Doc/library/symtable.rst @@ -171,7 +171,7 @@ Examining Symbol Tables .. method:: get_frees() - Return a tuple containing names of free :term:`closure variables ` + Return a tuple containing names of :term:`free (closure) variables ` in this function. diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 3aac9f91daeef5..188378f6200b83 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -199,7 +199,7 @@ Standard names are defined for the following types: .. data:: CellType The type for cell objects: such objects are used as containers for - a function's free :term:`closure variables `. + a function's :term:`closure variables `. .. versionadded:: 3.8 From 9ebc32ba9f7a816e05849332b117937a6022e91f Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Sat, 3 Aug 2024 17:49:27 +1000 Subject: [PATCH 4/9] Remove doubled word --- Doc/c-api/code.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst index be559b8e2e432e..fca9de54840c8c 100644 --- a/Doc/c-api/code.rst +++ b/Doc/c-api/code.rst @@ -32,7 +32,7 @@ bound into a function. .. c:function:: Py_ssize_t PyCode_GetNumFree(PyCodeObject *co) - Return the number of free :term:`free (closure) variables ` + Return the number of :term:`free (closure) variables ` in a code object. .. c:function:: int PyUnstable_Code_GetFirstFree(PyCodeObject *co) From e08d8f295039a3bbb19659bee541e8983f47c415 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Sat, 3 Aug 2024 17:53:15 +1000 Subject: [PATCH 5/9] Fix references missed due to line breaks --- Doc/library/functions.rst | 4 ++-- Doc/reference/datamodel.rst | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 772a0b5677f9c4..0072e108f80e58 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -682,8 +682,8 @@ are always available. They are listed here in alphabetical order. ``__builtins__`` dictionary into *globals* before passing it to :func:`exec`. The *closure* argument specifies a closure--a tuple of cellvars. - It's only valid when the *object* is a code object containing free - :term:`closure variables `. + It's only valid when the *object* is a code object containing + :term:`free (closure) variables `. The length of the tuple must exactly match the length of the code object'S :attr:`~codeobject.co_freevars` attribute. diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 877ebb8045d7a4..75131fad7f9941 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1168,9 +1168,9 @@ Special read-only attributes that are referenced from at least one :term:`nested scope` inside the function * - .. attribute:: codeobject.co_freevars - - A :class:`tuple` containing the names of free - :term:`closure variables ` that a :term:`nested scope` references - in an outer scope. See also :attr:`function.__closure__`. + - A :class:`tuple` containing the names of + :term:`free (closure) variables ` that a :term:`nested scope` + references in an outer scope. See also :attr:`function.__closure__`. Note: references to global and builtin names are *not* included. From d12a0f3ba3738037d55d967dcbc092c7b190347f Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Mon, 7 Oct 2024 23:08:16 +1000 Subject: [PATCH 6/9] Apply suggestions from code review --- Doc/glossary.rst | 17 ++++++++++++++--- Doc/reference/executionmodel.rst | 1 + 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst index de5e6ef9753a00..81e31ee9817f6a 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -227,10 +227,21 @@ Glossary class level (i.e., not in an instance of the class). closure variable - A variable referenced from a :term:`nested scope` that is defined in an outer scope - rather than being resolved at runtime from the globals or builtin namespaces. + A :term:`free variable` referenced from a :term:`nested scope` that is defined in an outer + scope rather than being resolved at runtime from the globals or builtin namespaces. May be explicitly defined with the :keyword:`nonlocal` keyword to allow write access, or implicitly defined if the variable is only read without being written to. + + For example, in the ``inner`` function in the following code, both ``x`` and ``print`` are + :term:`free variables `, but only ``x`` is a *closure variable*:: + + def outer(): + x = 0 + def inner(): + nonlocal x + x += 1 + print(x) + return inner Due to the name of the :attr:`codeobject.co_freevars` attribute, sometimes instead referred to as a :term:`free variable`. @@ -455,7 +466,7 @@ Glossary free variable Formally, as defined in the :ref:`language execution model `, a free variable is any variable used in a namespace which is not a local variable in that - namespace. + namespace. See :term:`closure variable` for an example. Pragmatically, due to the name of the :attr:`codeobject.co_freevars` attribute, the term is also sometimes used as a synonym for :term:`closure variable`. diff --git a/Doc/reference/executionmodel.rst b/Doc/reference/executionmodel.rst index 9ef328cfd0a4ea..2423d84457ab7c 100644 --- a/Doc/reference/executionmodel.rst +++ b/Doc/reference/executionmodel.rst @@ -332,6 +332,7 @@ namespace. If only one namespace is specified, it is used for both. .. XXX(ncoghlan) above is only accurate for string execution. When executing code objects, closure cells may now be passed explicitly to resolve co_freevars references. + Docs issue: https://github.com/python/cpython/issues/122826 .. _exceptions: From abb7fbc68147b1d6bc29a87ce3027fc72989420d Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Mon, 7 Oct 2024 23:11:47 +1000 Subject: [PATCH 7/9] Remove trailing whitespace --- Doc/glossary.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst index b5eb3e3ce5560e..73b6ba01e05e5e 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -236,10 +236,10 @@ Glossary scope rather than being resolved at runtime from the globals or builtin namespaces. May be explicitly defined with the :keyword:`nonlocal` keyword to allow write access, or implicitly defined if the variable is only read without being written to. - + For example, in the ``inner`` function in the following code, both ``x`` and ``print`` are :term:`free variables `, but only ``x`` is a *closure variable*:: - + def outer(): x = 0 def inner(): From bd8b52e720cc0674ea577de0a1835c0ed1efae09 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 8 Oct 2024 00:23:10 +1000 Subject: [PATCH 8/9] Terminate code example correctly. --- Doc/glossary.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 73b6ba01e05e5e..2a7557731a22d3 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -247,6 +247,7 @@ Glossary x += 1 print(x) return inner + Due to the name of the :attr:`codeobject.co_freevars` attribute, sometimes instead referred to as a :term:`free variable`. From a134ce138b5655e2d87c8ca4bb402b18c423ba3a Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 8 Oct 2024 17:27:22 +1000 Subject: [PATCH 9/9] Additional wording clarifications Co-authored-by: Carol Willing --- Doc/glossary.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 2a7557731a22d3..933fb0319452a6 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -235,7 +235,7 @@ Glossary A :term:`free variable` referenced from a :term:`nested scope` that is defined in an outer scope rather than being resolved at runtime from the globals or builtin namespaces. May be explicitly defined with the :keyword:`nonlocal` keyword to allow write access, - or implicitly defined if the variable is only read without being written to. + or implicitly defined if the variable is only being read. For example, in the ``inner`` function in the following code, both ``x`` and ``print`` are :term:`free variables `, but only ``x`` is a *closure variable*:: @@ -248,8 +248,10 @@ Glossary print(x) return inner - Due to the name of the :attr:`codeobject.co_freevars` attribute, sometimes instead - referred to as a :term:`free variable`. + Due to the :attr:`codeobject.co_freevars` attribute (which, despite its name, only + includes the names of closure variables rather than listing all referenced free + variables), the more general :term:`free variable` term is sometimes used even + when the intended meaning is to refer specifically to closure variables. complex number An extension of the familiar real number system in which all numbers are