`pytest.warns` hides type errors in `warnings.warn` calls

[zanieb]

• a detailed description of the bug or problem you are having
• [ ] output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

When using pytest.warns, incorrect values sent to warnings.warn are not reported obscuring bugs:

import pytest
import warnings


def test_example_one():
    with pytest.warns(UserWarning):
        warnings.warn(1)

    # This test passes but should raise a `TypeError` as demonstrated in `example_two`


def test_example_two():
    with pytest.raises(TypeError):
        warnings.warn(1)

macOS 13.2.1
pytest 7.2.2
python 3.11

Related to #9288

[nicoddemus]

Hmm that's unfortunate, thanks for the report!

Should be simple to fix.

[zanieb]

You're welcome! Certainly confused me for a minute :)

I'm happy to contribute a fix — want to just raise a type error if it's not a string? Looks like CPython allows Warning types too https://github.com/python/cpython/blob/f513d5c80672c76acbdaf7d5b601f4bbe9fae56a/Lib/warnings.py#L299-L301

[zanieb]

While trying to find the logic where this TypeError is raised I've gotten a bit confused. Weirdly this is fine

❯ python -c "import warnings; warnings.warn(1)"
<string>:1: UserWarning: 1

but the TypeError is definitely raised in pytest i.e. if I remove the pytest.raises:

    def test_example_two():
>       warnings.warn(1)
E       TypeError: expected string or bytes-like object, got 'int'

I feel like this caused me an error outside of pytest as well though

[nicoddemus]

Are you sure? This works for me:

import warnings
def test_example_two():
    warnings.warn(1)

Which Python and pytest versions are you using?

[zanieb]

The Python and pytest versions are included in the original post, but consider me very confused — I did not reproduce this with the minimal example and a test matrix at https://github.com/madkinsz/pytest-10865/actions/runs/4714285419/jobs/8360551859

It does however reproduce quite clearly when I'm working in PrefectHQ/prefect. Here's the example running in our test suite PrefectHQ/prefect#9231

[zanieb]

The issue appears to be related to filterwarnings, for example:

import warnings

# Does not raise a type error
warnings.warn(1)

# Raises a type error
with warnings.catch_warnings():
    warnings.filterwarnings("ignore", "test")
    warnings.warn(1)

with the addition of a warnings filter to pytest zanieb/pytest-10865@1c3e4e3 the type error is raised on all Python and 7.x pytest versions https://github.com/madkinsz/pytest-10865/actions/runs/4714336351

[zanieb]

I've opened a CPython issue as this doesn't really seem like pytest's fault python/cpython#103577 but pytest probably shouldn't hide this error still.

[zanieb]

@nicoddemus I don't expect there to be changes upstream for this. Should we validate that argument is of the correct type in pytest?

[RonnyPfannschmidt]

as far as im concerned, type metadata in typeshed is provided, and my understandig is, that it would point to the error