Test failures with Python 3.10.0a7

[hroncok]

Hello, we see some test failures due to different reprs in Python 3.10.0a7.

This is on branch main, @ 9653a0e.

Fedora Linux 33, 34 or 35.

$ tox -e py310
... installs deps, kill ...
$ .tox/py310/bin/pip install -U https://github.com/benjaminp/six/archive/refs/heads/master.zip
$ tox -e py310 -- testing/test_pytester.py testing/python/metafunc.py
py310 inst-nodeps: /home/churchyard/Dokumenty/RedHat/pytest/.tox/.tmp/package/3/pytest-6.3.0.dev355+g9653a0e9f.tar.gz
py310 installed: argcomplete==1.12.2,attrs==20.3.0,certifi==2020.12.5,chardet==4.0.0,elementpath==2.2.1,hypothesis==6.9.1,idna==2.10,iniconfig==1.1.1,mock==4.0.3,nose==1.3.7,packaging==20.9,pluggy==0.13.1,py==1.10.0,Pygments==2.8.1,pyparsing==2.4.7,pytest @ file:///home/churchyard/Dokumenty/RedHat/pytest/.tox/.tmp/package/3/pytest-6.3.0.dev355%2Bg9653a0e9f.tar.gz,requests==2.25.1,six @ https://github.com/benjaminp/six/archive/refs/heads/master.zip,sortedcontainers==2.3.0,toml==0.10.2,urllib3==1.26.4,xmlschema==1.6.1
py310 run-test-pre: PYTHONHASHSEED='3484571371'
py310 run-test: commands[0] | pytest testing/test_pytester.py testing/python/metafunc.py
============================= test session starts ==============================
platform linux -- Python 3.10.0a7, pytest-6.3.0.dev355+g9653a0e9f, py-1.10.0, pluggy-0.13.1
cachedir: .tox/py310/.pytest_cache
rootdir: /home/churchyard/Dokumenty/RedHat/pytest, configfile: pyproject.toml
plugins: hypothesis-6.9.1
collected 155 items

testing/test_pytester.py ...................F.                           [ 13%]
testing/python/metafunc.py .......................F.................     [ 40%]
testing/test_pytester.py x...........................                    [ 58%]
testing/python/metafunc.py ............................................. [ 87%]
...............                                                          [ 96%]
testing/test_pytester.py ....s                                           [100%]

=================================== FAILURES ===================================
_____________________________ test_run_result_repr _____________________________

    def test_run_result_repr() -> None:
        outlines = ["some", "normal", "output"]
        errlines = ["some", "nasty", "errors", "happened"]
    
        # known exit code
        r = pytester_mod.RunResult(1, outlines, errlines, duration=0.5)
>       assert (
            repr(r) == "<RunResult ret=ExitCode.TESTS_FAILED len(stdout.lines)=3"
            " len(stderr.lines)=4 duration=0.50s>"
        )
E       AssertionError: assert '<RunResult r...ration=0.50s>' == '<RunResult r...ration=0.50s>'
E         - <RunResult ret=ExitCode.TESTS_FAILED len(stdout.lines)=3 len(stderr.lines)=4 duration=0.50s>
E         ?                ---------
E         + <RunResult ret=TESTS_FAILED len(stdout.lines)=3 len(stderr.lines)=4 duration=0.50s>

testing/test_pytester.py:747: AssertionError
________________________ TestMetafunc.test_idmaker_enum ________________________

self = <metafunc.TestMetafunc object at 0x7f666fa0a710>

    def test_idmaker_enum(self) -> None:
        enum = pytest.importorskip("enum")
        e = enum.Enum("Foo", "one, two")
        result = idmaker(("a", "b"), [pytest.param(e.one, e.two)])
>       assert result == ["Foo.one-Foo.two"]
E       AssertionError: assert ['one-two'] == ['Foo.one-Foo.two']
E         At index 0 diff: 'one-two' != 'Foo.one-Foo.two'
E         Use -v to get the full diff

testing/python/metafunc.py:451: AssertionError
=========================== short test summary info ============================
FAILED testing/test_pytester.py::test_run_result_repr - AssertionError: asser...
FAILED testing/python/metafunc.py::TestMetafunc::test_idmaker_enum - Assertio...
============= 2 failed, 151 passed, 1 skipped, 1 xfailed in 5.27s ==============
ERROR: InvocationError for command /home/churchyard/Dokumenty/RedHat/pytest/.tox/py310/bin/pytest testing/test_pytester.py testing/python/metafunc.py (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py310: commands failed

The first problem is here:

pytest/testing/test_pytester.py

Line 748 in 9653a0e

repr(r) == "<RunResult ret=ExitCode.TESTS_FAILED len(stdout.lines)=3"

The second is here:

pytest/testing/python/metafunc.py

Line 451 in 9653a0e

assert result == ["Foo.one-Foo.two"]

This mgiht be related:

https://docs.python.org/3.10/whatsnew/3.10.html#enum says:

Enum repr() now returns enum_name.member_name and str() now returns member_name. Stdlib enums available as module constants have a repr() of module_name.member_name. (Contributed by Ethan Furman in bpo-40066.)

Package          Version
---------------- -----------------------
argcomplete      1.12.2
attrs            20.3.0
certifi          2020.12.5
chardet          4.0.0
elementpath      2.2.1
hypothesis       6.9.1
idna             2.10
iniconfig        1.1.1
mock             4.0.3
nose             1.3.7
packaging        20.9
pip              21.0.1
pluggy           0.13.1
py               1.10.0
Pygments         2.8.1
pyparsing        2.4.7
pytest           6.3.0.dev355+g9653a0e9f
requests         2.25.1
setuptools       53.0.0
six              1.15.0
sortedcontainers 2.3.0
toml             0.10.2
urllib3          1.26.4
wheel            0.36.2
xmlschema        1.6.1

[hroncok]

As a fix, should I if-else the expected string on Python version?

[bluetech]

Thanks for the detailed report!

Pretty strange that they just break Enum's str and repr like this after so long. There's a bunch of code everywhere which depends on those. But anyway...

The first problem is here:

For this I think whichever way we fix it is fine - condition, make the ExitCode. part optional, etc.

The second is here:

This is more tricky, because it may in turn break users who match on Enum-generated IDs. Probably not very common but still.

The choices I see are (relevant code is here):

1. Keep the behavior (i.e. str(val)).
   • Good: consistent behavior (str), arguably better looking?
   • Bad: inconsistent across versions, breaks compat
2. Keep old behavior (i.e. f"{type(val).__name__}.{val.name}")
   • Good: consistent across versions, no compat break, arguably better looking?
   • Bad: inconsistent with str
3. Move to new behavior (will need to emulate on older versions -- apparently tricky because they keep the module for stdlib enums...)
   • Good: consistent across versions, consistent with str (at least over time)
   • Bad: breaks compat

I think 3 is strictly better than 1, and 2 is better than 3. So my preference is 2.

[RonnyPfannschmidt]

If all we need is a custom repr on the enum let's go for it

[frenzymadness]

I have a little bit newer Python 3.10.0a7 python/cpython@23acadc and the pytest does not work for me for a different reason:

# cat test_pytest.py
def foo():
    return "bar"

def test_foo():
    assert foo() == "bar"

# python -m pytest test_pytest.py
==================================== test session starts =====================================
platform linux -- Python 3.10.0a7+, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /home/lbalhar/Software/cpython
collected 0 items / 1 error                                                                  

=========================================== ERRORS ===========================================
______________________________ ERROR collecting test_pytest.py _______________________________
Lib/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:688: in _load_unlocked
    ???
pytest2/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:161: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
pytest2/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:356: in _rewrite_test
    co = compile(tree, fn_, "exec", dont_inherit=True)
E   TypeError: required field "lineno" missing from alias
================================== short test summary info ===================================
ERROR test_pytest.py - TypeError: required field "lineno" missing from alias
!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================== 1 error in 0.10s ======================================

I haven't had time to investigate more yet. It seems to be something very fresh in Python itself. I'll try to bisect the change today.

[The-Compiler]

@frenzymadness See #8539 and #8540.

[nicoddemus]

Fixed by #8540.