@pytest.mark.xfail has no option to raise all Exceptions

[Jacob-Stevens-Haas]

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

Desired/Expected behavior

I'm looking to XFAIL a test, but only at an assert False and not for any Exception. I would like exceptions to be reported as ERRORs and not XFAILs. I opened this as a bug report because its unclear to me whether such behavior should exist with @pytest.mark.xfail(raises=[]) (currently raises an internal error in pytest, see MWE 3), or whether the desired behavior is a feature request.

In addition, I'm unclear whether the right behavior with @pytest.mark.xfail(raises=None) should differ between an Exception in the test (see MWE 1) versus an Exception in the test setup (see MWE 2). @RonnyPfannschmidt's comment on #5044 seems to indicate tests marked xfail should ERROR instead of XFAIL when the Exception occurs in setup code:

pytest in general consider any failure in the setup instead of the execution of a test a error instead of a failure (its considered a harsh issue when the creation of test conditions fails instead of the test)

Rationale:

I added an integration test for an app's workflow. Because I'm still creating low-level fixtures that are not yet deterministic, I am not sure what the expected output of the integration test should be. However, the test should still be able to run without errors. I understand that the test could be written:

def test_workflow_runs_without_errors(bad_fixture):
    do_something_and_return_something(bad_fixture)
    assert True

Then later, I could populate expected with the correct fixture and change the name of the test. The test and coverage report seems clearer, however, to have a test with an XFAIL mark. It would be removed when I write the correct fixture and can determine the expected output.

@pytest.mark.xfail(raises=[])
def test_workflow(bad_fixture):
    result = do_something_and_return_something(bad_fixture)
    expected = None
    assert result == expected

MWEs

Case 1 is an Exception in the test, case 2 is an Exception in test setup. In both cases, the test runs as an XFAIL rather than an ERROR. Case 3 includes the expected calling signature, along with the traceback.

Case 1

import pytest

@pytest.mark.xfail
def test_nothing():
    raise OSError
    assert False

Case 2

import pytest


@pytest.fixture
def bad_fixture():
    raise OSError

@pytest.mark.xfail
def test_nothing(bad_fixture):
    assert False

Case 3:

import pytest

@pytest.mark.xfail(raises=[])
def test_nothing():
    # raise OSError
    assert False

Case 3 traceback

>python -W ignore -m pytest -k test_nothing
========================================= test session starts =========================================
platform win32 -- Python 3.7.8, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\600301\Documents\GitHub\drivercast, configfile: pyproject.toml, testpaths: src/r2d2/
plugins: dash-1.20.0
collected 21 items / 20 deselected / 1 selected

src\r2d2\tests\test_nothing.py
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\_pytest\main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\_pytest\main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\_pytest\main.py", line 348, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\_pytest\runner.py", line 109, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\_pytest\runner.py", line 120, in runtestprotocol
INTERNALERROR>     rep = call_and_report(item, "setup", log)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\_pytest\runner.py", line 217, in call_and_report
INTERNALERROR>     report: TestReport = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\pluggy\callers.py", line 203, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "C:\Users\600301\Documents\GitHub\drivercast\r2d2-env\lib\site-packages\_pytest\skipping.py", line 291, in pytest_runtest_makereport
INTERNALERROR>     if raises is not None and not isinstance(call.excinfo.value, raises):
INTERNALERROR> TypeError: isinstance() arg 2 must be a type or tuple of types

======================================= 20 deselected in 2.94s ========================================

Versions

OS: Windows 10 Enterprise 10.0.16299 Build 16299 
Pytest version 6.2.3

pip list

>pip list
Package                       Version   Location
----------------------------- --------- -----------------------------------------------
alabaster                     0.7.12
appdirs                       1.4.4
atomicwrites                  1.4.0
attrs                         20.3.0
Babel                         2.9.1
backcall                      0.2.0
black                         21.4b2
Brotli                        1.0.9
certifi                       2020.12.5
cfgv                          3.2.0
chardet                       4.0.0
click                         7.1.2
colorama                      0.4.4
coverage                      5.5
dash                          1.20.0
dash-core-components          1.16.0
dash-html-components          1.1.3
dash-renderer                 1.9.1
dash-table                    4.11.3
decorator                     5.0.7
distlib                       0.3.1
docutils                      0.16
filelock                      3.0.12
flake8                        3.9.1
Flask                         1.1.2
Flask-Compress                1.9.0
future                        0.18.2
greenlet                      1.0.0
identify                      2.2.4
idna                          2.10
imagesize                     1.2.0
importlib-metadata            4.0.1
iniconfig                     1.1.1
ipykernel                     5.5.3
ipython                       7.23.0
ipython-genutils              0.2.0
itsdangerous                  1.1.0
jedi                          0.18.0
Jinja2                        2.11.3
joblib                        1.0.1
jupyter-client                6.1.12
jupyter-core                  4.7.1
MarkupSafe                    1.1.1
matplotlib-inline             0.1.2
mccabe                        0.6.1
mypy-extensions               0.4.3
nodeenv                       1.6.0
numpy                         1.20.2
packaging                     20.9
pandas                        1.2.4
parso                         0.8.2
pathspec                      0.8.1
patsy                         0.5.1
pickleshare                   0.7.5
pip                           20.1.1
plotly                        4.14.3
pluggy                        0.13.1
pre-commit                    2.12.1
prompt-toolkit                3.0.18
py                            1.10.0
pycodestyle                   2.7.0
pyflakes                      2.3.1
Pygments                      2.9.0
PyKrige                       1.6.0
pyparsing                     2.4.7
pytest                        6.2.3
python-dateutil               2.8.1
pytz                          2021.1
pywin32                       300
PyYAML                        5.4.1
pyzmq                         22.0.3
r2d2                          0.2.0     c:\users\600301\documents\github\drivercast\src
regex                         2021.4.4
requests                      2.25.1
retrying                      1.3.3
scikit-learn                  0.23.2
scipy                         1.6.3
setuptools                    47.1.0
six                           1.15.0
snowballstemmer               2.1.0
Sphinx                        3.5.4
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        1.0.3
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.4
SQLAlchemy                    1.4.12
statsmodels                   0.12.2
threadpoolctl                 2.1.0
toml                          0.10.2
tornado                       6.1
traitlets                     5.0.5
typed-ast                     1.4.3
typing-extensions             3.10.0.0
urllib3                       1.26.4
virtualenv                    20.4.4
wcwidth                       0.2.5
Werkzeug                      1.0.1
zipp                          3.4.1
WARNING: You are using pip version 20.1.1; however, version 21.1.1 is available.
You should consider upgrading via the 'c:\users\600301\documents\github\drivercast\r2d2-env\scripts\python.exe -m pip install --upgrade pip' command.

[Jacob-Stevens-Haas]

On MWE 3:
The docs say:

you can specify a single exception, or a list of exceptions, in the raises argument

So I assumed an empty list was also allowed. Related to #8646?

[The-Compiler]

A failing assertion is an exception as well (an AssertionError). So it sounds like you want @pytest.mark.xfail(raises=[AssertionError]) probably?

Agreed that raises=[] should do something more sensible, though.

[Zac-HD]

Can you link to the docs you're referencing? Because at https://docs.pytest.org/en/latest/how-to/skipping.html#raises-parameter I see:

If you want to be more specific as to why the test is failing, you can specify a single exception, or a tuple of exceptions, in the raises argument.

and I would expect passing a list to be an error, since you can't pass it to isinstance().

---

IMO passing an empty sequence of exceptions should be an explicit error, though of course via a deprecation period if the current behaviour is to silently ignore the argument.

[Jacob-Stevens-Haas]

A failing assertion is an exception as well (an AssertionError). So it sounds like you want @pytest.mark.xfail(raises=[AssertionError]) probably?

Yes! Thank you :) That solves the immediate problem I needed.

@Zac-HD, I could have sworn when I looked a few days ago, the docs said "list" and not "tuple", but maybe I read it wrong or was looking at an old version of the docs. EDIT: I looked at https://docs.pytest.org/en/reorganize-docs/skipping.html, which came up in a google searchfor this issue. I thought "reorganize-docs" meant "latest". Modifying MWE3 to use a tuple doesn't generate an error, but it does throw out AssertionError as an allowed exception. tl;dr:

@pytest.mark.xfail(raises=())

only allows a test to XPASS or FAIL, without possibility of XFAIL. This is perhaps a bit less jarring than silently ignoring the argument. Now that I'm reading the correct docs, 🤦‍♂️, I'm personally fine with the current behavior.

So I guess there's three questions at play:

1. Does @pytest.mark.xfail(raises=[AssertionError]) deserve special mention in the documentation (I'm happy to write a documentation PR)
2. How to handle @pytest.mark.xfail(raises=()).
3. Should @pytest.mark.xfail handle Exceptions in fixtures the same as in the code under test?

[Zac-HD]

1. Maybe - if you have wording that would have helped you, I'd be happy to at least consider adding it.

2. IMO xfail(raises=()) should be an error, because it can never have status XFAIL (only FAIL or XPASS). We should probably have a deprecation period though; I've added this to Improved error when () (empty tuple) is passed to pytest.raises() or pytest.warns() #8646.

3. No strong opinion here, so I'd keep the status quo (at least until we have a specific better design, and have planned all the related changes to avoid making them piecemeal).

---

I'm looking to XFAIL a test, but only at an assert False and not for any Exception.

@pytest.mark.xfail(raises=AssertionError). This could be documented as per (1) above.

I would like exceptions to be reported as ERRORs and not XFAILs.

assert False will be an XFAIL, and other exceptions will be FAIL-ures. I don't think we could change that to ERROR, which is reserved for errors in test setup, without breaking many other things across the ecosystem.