OutcomeExceptions should be handled during collection (not causing INTERNALERROR)

[blueyed]

When trying to use this from pytest-cov's source:

% p -m 'not slow'
===================================== test session starts ======================================
platform linux -- Python 3.8.0, pytest-5.3.0, py-1.8.0, pluggy-0.13.1
rootdir: …/Vcs/pytest-cov, inifile: setup.cfg, testpaths: tests
plugins: forked-1.1.3, xdist-1.30.0, cov-2.8.1
collected 124 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/main.py", line 196, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/main.py", line 239, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/main.py", line 249, in pytest_collection
INTERNALERROR>     return session.perform_collect()
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/main.py", line 450, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 182, in _multicall
INTERNALERROR>     next(gen)  # first yield
INTERNALERROR>   File "…/Vcs/pytest-cov/tests/conftest.py", line 34, in pytest_collection_modifyitems
INTERNALERROR>     item.add_marker(pytest.mark.slow)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/mark/structures.py", line 313, in __getattr__
INTERNALERROR>     fail(
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/outcomes.py", line 128, in fail
INTERNALERROR>     raise Failed(msg=msg, pytrace=pytrace)
INTERNALERROR> Failed: 'slow' not found in `markers` configuration option

[Zac-HD]

INTERNALERROR>   File "…/Vcs/pytest-cov/tests/conftest.py", line 34, in pytest_collection_modifyitems
INTERNALERROR>     item.add_marker(pytest.mark.slow)

pytest-cov should register it's slow marker, but I agree that we should have a better UX when this hasn't happened (including suggesting how to fix it from the error message).

[blueyed]

@Zac-HD
There's no "slow" marker in pytest-cov.
This issue is about the UX in case of the error/warning in that case.

[Zac-HD]

There's no registered slow marker, but pytest-cov/tests/conftest.py tries to add it anyway.

It's definitely a UX problem for pytest, but it's reporting a real problem for pytest-cov.

[blueyed]

Ok.
Just to be clear: it was me adding item.add_marker(pytest.mark.slow) to its conftest (locally), and this issue is about improving the UX / error in that case (for when it is not registered).
IIRC this error is better in general, so is likely about fail.Exception not being handled here properly.

[RonnyPfannschmidt]

originally this was an attribute error, what happens in that particular case, if we add that back?

[blueyed]

@RonnyPfannschmidt
It was changed in 5436e42.
(btw: in #6232 I've changed it to not use pytrace=False already, because I found the traceback/location to be useful.)

The issue is that "fail.Exception" is not handled here. This also applies to AttributeErrors (just tried it directly from pytest_collection_modifyitems).

It might make more sense for non-outcome-errors (altough not really "internal"), but outcome-exceptions should be handled probably.

[RonnyPfannschmidt]

@blueyed i think we should change all those fails to actual UsageError

[blueyed]

@RonnyPfannschmidt
sounds good in general, but is unclear to me: do you mean all failures (pytest.fail and/or Outcome exceptions in general) during collection then?

Likely you've meant that the initial reported one ("'X' not found in markers configuration option") should become an UsageError by itself, which makes sense.
This leaves us with what should happen with pytest.fail etc.

[RonnyPfannschmidt]

outcome exceptions should be handled where we have actual outcomes, else they should be a internalerror

the modify-items hook is something where we do not expect outcomes