`ast.dump()` elision of empty values should use field types

[JelleZijlstra]

Bug report

Bug description:

In 3.13 we changed ast.dump() to not show empty values (None or the empty list) by default. However, this is based purely on the value of individual attributes:

>>> ast.dump(ast.Name(id="x"))
"Name(id='x', ctx=Load())"
>>> ast.dump(ast.Name(id=None))
'Name(ctx=Load())'
>>> ast.dump(ast.Name(id=[]))
'Name(ctx=Load())'

Instead, this logic should look at the node's _field_types and use the same logic we use to determine whether to allow omitting the argument in calls to the constructor: elide None if the type is a union including None, elide [] if the type is a list, and elide Load() if the type is an expr_context.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-134718: Fix ast.dump() for empty non-default values #134926
• [3.14] gh-134718: Fix ast.dump() for empty non-default values (GH-134926) #134931
• gh-134718: Omit optional Load() values in ast.dump() #134934
• [3.13] gh-134718: Fix ast.dump() for empty non-default values (GH-134926) #134936

[hunterhogan]

I've been working on the missing None when show_empty=True. I tried to integrate into the current function.

ast.dump

from _ast import AST, Constant, MatchSingleton, MatchStar

def dump(
    node, annotate_fields=True, include_attributes=False,
    *,
    indent=None, show_empty=False,
):
    """
    Return a formatted dump of the tree in node.  This is mainly useful for
    debugging purposes.  If annotate_fields is true (by default),
    the returned string will show the names and the values for fields.
    If annotate_fields is false, the result string will be more compact by
    omitting unambiguous field names.  Attributes such as line
    numbers and column offsets are not dumped by default.  If this is wanted,
    include_attributes can be set to true.  If indent is a non-negative
    integer or string, then the tree will be pretty-printed with that indent
    level. None (the default) selects the single line representation.
    If show_empty is False, then empty lists and fields that are None
    will be omitted from the output for better readability.
    """
    def _format(node, level=0):
        if indent_str is not None:
            level += 1
            prefix = '\n' + indent_str * level
            sep = ',\n' + indent_str * level
        else:
            prefix = ''
            sep = ', '
        if isinstance(node, AST):
            cls = type(node)
            args = []
            args_buffer = []
            allsimple = True
            keywords = annotate_fields
            for name in node._fields:
                try:
                    value = getattr(node, name)
                except AttributeError:
                    keywords = True
                    continue
                if value is None and getattr(cls, name, ...) is None:
                    if show_empty:
                        args.append('%s=%s' % (name, value))
                    keywords = True
                    continue
                elif (
                    not show_empty
                    and (value is None or value == [])
                    # Special cases:
                    # `Constant(value=None)` and `MatchSingleton(value=None)` and `MatchStar(name=None)`
                    and not isinstance(node, (Constant, MatchSingleton, MatchStar))
                ):
                    args_buffer.append(repr(value))
                    continue
                elif not keywords:
                    args.extend(args_buffer)
                    args_buffer = []
                value_formatted, simple = _format(value, level)
                allsimple = allsimple and simple
                if keywords:
                    args.append('%s=%s' % (name, value_formatted))
                else:
                    args.append(value_formatted)
            if include_attributes and node._attributes:
                for name_attributes in node._attributes:
                    try:
                        value_attributes = getattr(node, name_attributes)
                    except AttributeError:
                        continue
                    if value_attributes is None and getattr(cls, name_attributes, ...) is None:
                        continue
                    value_attributes_formatted, simple = _format(value_attributes, level)
                    allsimple = allsimple and simple
                    args.append('%s=%s' % (name_attributes, value_attributes_formatted))
            if allsimple and len(args) <= 3:
                return ('%s(%s)' % (f"{node.__class__.__module__}.{node.__class__.__name__}", ', '.join(args)), not args)
            return ('%s(%s%s)' % (f"{node.__class__.__module__}.{node.__class__.__name__}", prefix, sep.join(args)), False)
        elif isinstance(node, list):
            if not node:
                return ('[]', True)
            return '[%s%s]' % (prefix, sep.join(_format(x, level)[0] for x in node)), False
        return (repr(node), True)

    if not isinstance(node, AST):
        raise TypeError('expected AST, got %r' % node.__class__.__name__)
    if indent is not None and not isinstance(indent, str):
        indent_str = ' ' * indent
    else:
        indent_str = indent
    return _format(node)[0]

type annotations

from _ast import AST
from types import EllipsisType, NotImplementedType
from typing import TypeAlias

_Constant: TypeAlias = bool | bytes | complex | EllipsisType | float | int | None | NotImplementedType | range | str

ConstantType: TypeAlias = _Constant | frozenset['ConstantType'] | tuple['ConstantType', ...]

def dump(
    node: AST,
    annotate_fields: bool = ...,
    include_attributes: bool = ...,
    *,
    indent: int | str | None = ...,
    show_empty: bool = ...,
) -> str:
    def _format(node: ConstantType | AST | list[AST] | list[str], level: int = ...) -> tuple[str, bool]:
        # Local variables in _format function
        prefix: str
        sep: str
        cls: type[AST]
        args: list[str]
        args_buffer: list[str]
        allsimple: bool
        keywords: bool
        
        name: str
        value: AST | list[AST] | list[str] | ConstantType | bool | bytes | complex | float | int | str
        value_formatted: str
        simple: bool

        name_attributes: str
        value_attributes: int | None
        value_attributes_formatted: str

        x: AST | str
        
        ...
    
    # Local variables in dump function
    indent_str: str | None
    
    ...

stdlib/builtins.pyi type change

@final
class _NotImplementedType(object):
    __call__: None

side by side comparison file

I guess add dump, above, to the file below instead of the import I'm using.

from astToolFactory.ast_dump import dump
import ast

nodeConstructors: list[str] = [
	"ast.Add()",
	"ast.alias(name='name', asname=None)",
	"ast.And()",
	"ast.AnnAssign(target=ast.Name(id='target', ctx=ast.Load()), annotation=ast.Name(id='annotation', ctx=ast.Load()), value=None, simple=0)",
	"ast.arg(arg='arg', annotation=None, type_comment=None)",
	"ast.arguments(posonlyargs=[], args=[], vararg=None, kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[])",
	"ast.Assert(test=ast.Constant(value=..., kind=None), msg=None)",
	"ast.Assign(targets=[], value=ast.Constant(value=None, kind=None), type_comment=None)",
	"ast.AsyncFor(target=ast.Name(id='target', ctx=ast.Load()), iter=ast.Name(id='iter', ctx=ast.Load()), body=[], orelse=[], type_comment=None)",
	"ast.AsyncFunctionDef(name='name', args=ast.arguments(posonlyargs=[], args=[], vararg=None, kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[]), body=[], decorator_list=[], returns=None, type_comment=None, type_params=[])",
	"ast.AsyncWith(items=[], body=[], type_comment=None)",
	"ast.Attribute(value=ast.Name(id='value', ctx=ast.Load()), attr='attr', ctx=ast.Load())",
	"ast.AugAssign(target=ast.Name(id='target', ctx=ast.Load()), op=ast.Add(), value=ast.Constant(value=None, kind=None))",
	"ast.Await(value=ast.Constant(value=None, kind=None))",
	"ast.BinOp(left=ast.Constant(value=None, kind=None), op=ast.Add(), right=ast.Constant(value=None, kind=None))",
	"ast.BitAnd()",
	"ast.BitOr()",
	"ast.BitXor()",
	"ast.boolop()",
	"ast.BoolOp(op=ast.And(), values=[])",
	"ast.Break()",
	"ast.Call(func=ast.Name(id='func', ctx=ast.Load()), args=[], keywords=[])",
	"ast.ClassDef(name='name', bases=[], keywords=[], body=[], decorator_list=[], type_params=[])",
	"ast.cmpop()",
	"ast.Compare(left=ast.Constant(value=None, kind=None), ops=[], comparators=[])",
	"ast.comprehension(target=ast.Name(id='target', ctx=ast.Load()), iter=ast.Name(id='iter', ctx=ast.Load()), ifs=[], is_async=0)",
	"ast.Constant(value=None, kind=None)",
	"ast.Continue()",
	"ast.Del()",
	"ast.Delete(targets=[])",
	"ast.Dict(keys=[], values=[])",
	"ast.DictComp(key=ast.Constant(value=None, kind=None), value=ast.Constant(value=None, kind=None), generators=[])",
	"ast.Div()",
	"ast.Eq()",
	"ast.excepthandler()",
	"ast.ExceptHandler(type=None, name=None, body=[])",
	"ast.expr_context()",
	"ast.expr()",
	"ast.Expr(value=ast.Constant(value=None, kind=None))",
	"ast.Expression(body=ast.Constant(value=None, kind=None))",
	"ast.FloorDiv()",
	"ast.For(target=ast.Name(id='target', ctx=ast.Load()), iter=ast.Name(id='iter', ctx=ast.Load()), body=[], orelse=[], type_comment=None)",
	"ast.FormattedValue(value=ast.Constant(value=None, kind=None), conversion=0, format_spec=None)",
	"ast.FunctionDef(name='name', args=ast.arguments(posonlyargs=[], args=[], vararg=None, kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[]), body=[], decorator_list=[], returns=None, type_comment=None, type_params=[])",
	"ast.FunctionType(argtypes=[], returns=ast.Constant(value=None, kind=None))",
	"ast.GeneratorExp(elt=ast.Constant(value=None, kind=None), generators=[])",
	"ast.Global(names=[])",
	"ast.Gt()",
	"ast.GtE()",
	"ast.If(test=ast.Constant(value=None, kind=None), body=[], orelse=[])",
	"ast.IfExp(test=ast.Constant(value=None, kind=None), body=ast.Constant(value=None, kind=None), orelse=ast.Constant(value=None, kind=None))",
	"ast.Import(names=[])",
	"ast.ImportFrom(module=None, names=[], level=0)",
	"ast.In()",
	"ast.Interactive(body=[])",
	"ast.Invert()",
	"ast.Is()",
	"ast.IsNot()",
	"ast.JoinedStr(values=[])",
	"ast.keyword(arg=None, value=ast.Constant(value=None, kind=None))",
	"ast.Lambda(args=ast.arguments(posonlyargs=[], args=[], vararg=None, kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[]), body=ast.Constant(value=None, kind=None))",
	"ast.List(elts=[], ctx=ast.Load())",
	"ast.ListComp(elt=ast.Constant(value=None, kind=None), generators=[])",
	"ast.Load()",
	"ast.LShift()",
	"ast.Lt()",
	"ast.LtE()",
	"ast.match_case(pattern=ast.MatchValue(value=ast.Constant(value=None, kind=None)), guard=None, body=[])",
	"ast.Match(subject=ast.Constant(value=None, kind=None), cases=[])",
	"ast.MatchAs(pattern=None, name=None)",
	"ast.MatchClass(cls=ast.Name(id='cls', ctx=ast.Load()), patterns=[], kwd_attrs=[], kwd_patterns=[])",
	"ast.MatchMapping(keys=[], patterns=[], rest=None)",
	"ast.MatchOr(patterns=[])",
	"ast.MatchSequence(patterns=[])",
	"ast.MatchSingleton(value=None)",
	"ast.MatchStar(name=None)",
	"ast.MatchValue(value=ast.Constant(value=None, kind=None))",
	"ast.MatMult()",
	"ast.Mod()",
	"ast.mod()",
	"ast.Module(body=[], type_ignores=[])",
	"ast.Mult()",
	"ast.Name(id='id', ctx=ast.Load())",
	"ast.NamedExpr(target=ast.Name(id='target', ctx=ast.Load()), value=ast.Constant(value=None, kind=None))",
	"ast.Nonlocal(names=[])",
	"ast.Not()",
	"ast.NotEq()",
	"ast.NotIn()",
	"ast.operator()",
	"ast.Or()",
	"ast.ParamSpec(name='name', default_value=None)",
	"ast.Pass()",
	"ast.pattern()",
	"ast.Pow()",
	"ast.Raise(exc=None, cause=None)",
	"ast.Return(value=None)",
	"ast.RShift()",
	"ast.Set(elts=[])",
	"ast.SetComp(elt=ast.Constant(value=None, kind=None), generators=[])",
	"ast.Slice(lower=None, upper=None, step=None)",
	"ast.Starred(value=ast.Name(id='value', ctx=ast.Load()), ctx=ast.Load())",
	"ast.stmt()",
	"ast.Store()",
	"ast.Sub()",
	"ast.Subscript(value=ast.Name(id='value', ctx=ast.Load()), slice=ast.Constant(value=None, kind=None), ctx=ast.Load())",
	"ast.Try(body=[], handlers=[], orelse=[], finalbody=[])",
	"ast.TryStar(body=[], handlers=[], orelse=[], finalbody=[])",
	"ast.Tuple(elts=[], ctx=ast.Load())",
	"ast.type_ignore()",
	"ast.type_param()",
	"ast.TypeAlias(name=ast.Name(id='name', ctx=ast.Load()), type_params=[], value=ast.Name(id='value', ctx=ast.Load()))",
	"ast.TypeIgnore(lineno=1, tag='')",
	"ast.TypeVar(name='name', bound=None, default_value=None)",
	"ast.TypeVarTuple(name='name', default_value=None)",
	"ast.UAdd()",
	"ast.unaryop()",
	"ast.UnaryOp(op=ast.Not(), operand=ast.Constant(value=None, kind=None))",
	"ast.USub()",
	"ast.While(test=ast.Constant(value=None, kind=None), body=[], orelse=[])",
	"ast.With(items=[], body=[], type_comment=None)",
	"ast.withitem(context_expr=ast.Name(id='name', ctx=ast.Load()), optional_vars=None)",
	"ast.Yield(value=None)",
	"ast.YieldFrom(value=ast.Constant(value=..., kind=None))",
]
for nodeAsStr in nodeConstructors:
	print(nodeAsStr, ast.dump(eval(nodeAsStr), show_empty=True), dump(eval(nodeAsStr), show_empty=True), '', sep='\n')

[hunterhogan]

FWIW, I think the elided idea for show_empty=False is far better than

                elif (
                    not show_empty
                    and (value is None or value == [])
                    # Special cases:
                    # `Constant(value=None)` and `MatchSingleton(value=None)`
                    and not isinstance(node, (Constant, MatchSingleton))
                ):

I didn't like that code block, but I didn't know enough to figure out a different way.

I have a request for whomever rewrites the function: please change the identifiers so that a stub file can type the identifiers. There is a difference between "not typed" and "not typeable."

[grantlouisherman]

@hunterhogan could I take a look at this? Im new but wanted to take a crack if thats ok.

[hunterhogan]

@grantlouisherman Sure. I threw everything into a directory here.
https://github.com/hunterhogan/cpython/tree/804191081b49aa1943f8dfcb8055d875ea6f8d07/Z0Z_workingFiles

I have a lightly refactored and functioning version in astToolkit. import astToolkit; astToolkit.dump(...).

The major barrier to the pull request, afaik, is that the current tests are reporting failures because the tests are wrong.

I don't have enough experience with this labyrinthian process or the health to cope and to figure out how to publish working code.

[grantlouisherman]

@serhiy-storchaka are you taking a look? I see you assigned it to yourself. If so Ill look for a new issue.

[serhiy-storchaka]

This is a great idea. There is an issue with the current code -- it omits None and [] not only for optional arguments. It has a special case for Constant and MatchSingleton which can legally have the non-optional None value, but a new Interpolation node was added recently. It's str parameter has type constant, so it can theoretically be any value.

Also, None and [] wrongly assigned to other attributes should not be omitted.

For None, we can detect an optional argument by simply testing if the class variable was set to None. For [], the only way to do this is using a field_type.

So this part is qualified as a bug fix and should be backported back to 3.13. Omission of the Load() value is a new feature.