Tests: test_fail_and_continue_with_stepwise is flaky

[blueyed]

Seen it before already, and here it is again:

https://ci.appveyor.com/project/pytestbot/pytest/builds/20191315/job/431p82b80ai2je82#L148

Seems to be related to xdist. TOXENV=py27-trial,py27-numpy,py27-nobyte in this case.

================================== FAILURES ===================================
____________________ test_fail_and_continue_with_stepwise _____________________
[gw1] win32 -- Python 2.7.15 c:\projects\pytest\.tox\py27-nobyte\scripts\python.exe
stepwise_testdir = <Testdir local('c:\\users\\appveyor\\appdata\\local\\temp\\1\\pytest-of-appveyor\\pytest-2\\popen-gw1\\test_fail_and_continue_with_stepwise0')>
    def test_fail_and_continue_with_stepwise(stepwise_testdir):
        # Run the tests with a failing second test.
        result = stepwise_testdir.runpytest("-v", "--strict", "--stepwise", "--fail")
        assert not result.stderr.str()
    
        stdout = result.stdout.str()
        # Make sure we stop after first failing test.
        assert "test_success_before_fail PASSED" in stdout
        assert "test_fail_on_flag FAILED" in stdout
        assert "test_success_after_fail" not in stdout
    
        # "Fix" the test that failed in the last run and run it again.
        result = stepwise_testdir.runpytest("-v", "--strict", "--stepwise")
        assert not result.stderr.str()
    
        stdout = result.stdout.str()
        # Make sure the latest failing test runs and then continues.
>       assert "test_success_before_fail" not in stdout
E       AssertionError: assert 'test_success_before_fail' not in '============================= test session starts =============================\nplatform win32 -- Python 2.7.15, pyte...                              [100%]\n\n========================== 6 passed in 0.08 seconds ===========================\n'
C:\projects\pytest\testing\test_stepwise.py:95: AssertionError
---------------------------- Captured stdout call -----------------------------
============================= test session starts =============================
platform win32 -- Python 2.7.15, pytest-3.10.1.dev65+g9d11428a, py-1.7.0, pluggy-0.8.0 -- c:\projects\pytest\.tox\py27-nobyte\scripts\python.exe
cachedir: C:\projects\pytest\.tox\py27-nobyte\.pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('C:\\projects\\pytest\\.hypothesis\\examples')
rootdir: c:\users\appveyor\appdata\local\temp\1\pytest-of-appveyor\pytest-2\popen-gw1\test_fail_and_continue_with_stepwise0, inifile:
plugins: xdist-1.24.0, forked-0.2, hypothesis-3.82.1
collecting ... collected 6 items
test_a.py::test_success_before_fail PASSED                               [ 16%]
test_a.py::test_fail_on_flag FAILED                                      [ 33%]
================================== FAILURES ===================================
______________________________ test_fail_on_flag ______________________________
request = <FixtureRequest for <Function 'test_fail_on_flag'>>
    def test_fail_on_flag(request):
>       assert not request.config.getvalue('fail')
E       AssertionError: assert not True
E        +  where True = <bound method Config.getvalue of <_pytest.config.Config object at 0x0566DA70>>('fail')
E        +    where <bound method Config.getvalue of <_pytest.config.Config object at 0x0566DA70>> = <_pytest.config.Config object at 0x0566DA70>.getvalue
E        +      where <_pytest.config.Config object at 0x0566DA70> = <FixtureRequest for <Function 'test_fail_on_flag'>>.config
test_a.py:5: AssertionError
!!!!!!!! Interrupted: Test failed, continuing from this test next run. !!!!!!!!
===================== 1 failed, 1 passed in 0.17 seconds ======================
============================= test session starts =============================
platform win32 -- Python 2.7.15, pytest-3.10.1.dev65+g9d11428a, py-1.7.0, pluggy-0.8.0 -- c:\projects\pytest\.tox\py27-nobyte\scripts\python.exe
cachedir: C:\projects\pytest\.tox\py27-nobyte\.pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('C:\\projects\\pytest\\.hypothesis\\examples')
rootdir: c:\users\appveyor\appdata\local\temp\1\pytest-of-appveyor\pytest-2\popen-gw1\test_fail_and_continue_with_stepwise0, inifile:
plugins: xdist-1.24.0, forked-0.2, hypothesis-3.82.1
collecting ... collected 6 items
test_a.py::test_success_before_fail PASSED                               [ 16%]
test_a.py::test_fail_on_flag PASSED                                      [ 33%]
test_a.py::test_success_after_fail PASSED                                [ 50%]
test_a.py::test_fail_last_on_flag PASSED                                 [ 66%]
test_a.py::test_success_after_last_fail PASSED                           [ 83%]
test_b.py::test_success PASSED                                           [100%]
========================== 6 passed in 0.08 seconds ===========================
=========================== short test summary info ===========================
FAIL testing/test_stepwise.py::test_fail_and_continue_with_stepwise

[henrykironde]

I looked through the code and I think we need to initialize the default property related to stepwise in both Config and StepwisePlugin or just in the StepwisePlugin

class StepwisePlugin:
    def __init__(self, config):
        self.config = config
        self.config.cache = {}  # could be None or OrderedDict

class Config(object):
    """ access to configuration values, pluginmanager and plugin hooks.  """

    def __init__(self, pluginmanager):
        #: access to command line option as attributes.
        #: (deprecated), use :py:func:`getoption() <_pytest.config.Config.getoption>` instead
        self.option = argparse.Namespace()
        self..cache = {}  # could be None or OrderedDict

I think that is why this happens <_pytest.config.Config object at 0x7f47beeff2e8>.getvalue

[blueyed]

Hmm, not sure what you mean.. can you post a diff instead?

Apart from that I think it is more related to tests being run on different xdist nodes maybe?!

[blueyed]

Fixes in #4382.