Feature request: refactor the pytest.raises interface

[wimglenn]

Interface for pytest.raises is a stumbling block. Problems:

1. The "message" kwarg violates principle of least surprise. User probably expected: simple match on exception message (useful), but it's actually for providing custom failure message (not very useful). Recommend deprecation. message has now been deprecated.
2. The "match" kwarg violates principle of least surprise. It will catch substrings, for example. And it's annoying when the error message itself contains regex syntax that you'll need to escape. Could possibly be renamed to "regexp" or something? ...don't care / don't use it
3. No easy way to achieve the most simple and basic use-case: assert exact match on an exception instance args with basic types such as strings and/or numbers. use the plugin instead: pytest-raisin

Here are some examples of problematic usage.

# looks right, but doesn't assert what it seems like it's asserting. trappy for beginners!
with pytest.raises(Exception, message="boom"):
    Exception("boom")

# whoops!
with pytest.raises(Exception, message="boom"):
    Exception("bang")

# whoops again! still not there yet..
with pytest.raises(Exception, match="boom"):
    Exception("kaboom")

# correct usage
with pytest.raises(Exception, match="^boom$"):
    Exception("kaboom")

What if the thing you want to match is not really a string? Can encounter this frequently when testing the CLI parts of code (since argparse is gonna raise SystemExit with an integer return code). Problems:

# passes, but not why you thought
with pytest.raises(SystemExit, match=0):
    sys.exit(0)

# passes??
with pytest.raises(SystemExit, match=0):
    sys.exit(1)

# crashes, but crashes too late (`TypeError` in re compile)
with pytest.raises(SystemExit, match=1):
    sys.exit(1)

# correct usage
with pytest.raises(SystemExit) as cm:
    sys.exit(0)
assert cm.value.args == (0,)

# another trap! d'oh!
with pytest.raises(SystemExit) as cm:
    sys.exit(0)
    assert cm.value.args == (0,)

I would like to propose using exception instances to overcome these stumbling blocks.

Example usage:

with pytest.raises(SystemExit(2)):
    sys.exit(2)

testfixtures.ShouldRaise is an example of this usage, and I'd love to have that interface in pytest directly. The comparison objects which make it work are non-trivial. A much easier to implement alternative, though less beautiful, might be just to support matching the args attribute as tuple.

with pytest.raises(SomeError, exc_args=(2, "code two")):
    raise SomeError(2, "code two")

Admittedly it won't help people who put too much logic into their custom exception classes, but it should cover majority standard use-cases (where exceptions are dumb subclasses, just with a custom numeric error code or message).

What do you think?

[nicoddemus]

Hi @wimglenn,

You are absolutely right. In fact we had a long discussion about this in #3974.

I would like to avoid breaking things, even if match might be a little misleading as you comment.

We had some consensus with the proposal in #3974 (comment) about the message parameter.

Your proposal then would be improve pytest.raises to match an exception exactly. I particularly like the idea of passing the exception instance that should be matched exactly (SystemExit(2)).

[asottile]

one problem I see with passing exception instances into raises is few exception types properly implement __eq__ and so you'd potentially enable another pitfall. I'd rather lean towards fewer behaviours of raises and leave the assertion up to the user.

I think we should steer people more towards the context manager -- that said, I don't have any counters to this example other than linting 🤷‍♂️ (pytest knows the full ast of your tests, maybe it could warn on this behaviour during discovery)

with pytest.raises(SystemExit) as cm:
    sys.exit(0)
    assert cm.value.args == (0,)

actually, now that I think about it -- any code coverage tool is going to expose this mistake so maybe it's less of a bad problem than I think.

I'm in favor of deprecating and removing `message=

Should we consolidate this discussion into #3974 (duplicate)?

[nicoddemus]

I'm in favor of deprecating and removing `message=

Too slow! 😁 #4539

[wimglenn]

@asottile To clarify, my suggestion wasn't to rely on exception __eq__ (which in almost all cases will be identity-based / useless) but for pytest.raises itself to do a more nuanced comparison of exception instances.

[asottile]

@asottile To clarify, my suggestion was not be to rely on exception's __eq__ (which in almost all cases will be identity-based) but for pytest.raises itself to do a more nuanced comparison of exception instances.

yes I understand that, implementing our own custom idea of "equality" seems like a place to write bugs and surprising behaviour.

[wimglenn]

I see the point, and could be convinced either way, however this wouldn't be using anywhere an equality assertion like assert exc1 == exc2. Rather, the means of comparison would be a documented behaviour of pytest.raises (probably just something simple like comparing equality of the args attribute).

For the surprising behaviour, we already have it ;)

[asottile]

Other than message= I don't find anything else you've posted "surprising". Adding a custom non-equality "equality" check seems like a bad idea

[wimglenn]

Not even this one?

with pytest.raises(SystemExit, match=0):
    sys.exit(2)

Why match=0 was treated the same as match=None or match=".*"?

Adding a custom non-equality "equality" check seems like a bad idea

Yes, agreed.

[asottile]

Ooh missed that one. Seems like a micro bug needing an is somewhere

[asottile]

Here's a fix for that one: #4540

[nicoddemus]

message is now deprecated as of #4539. 🎉

I guess what's left here is to discuss if/how we can improve pytest.raises to also check for the exception's args, correct?

[wimglenn]

Correct.

Sensible options I see:

1. allow to pass exception instance, and equality compare args attributes. any more sophisticated comparison has to use cm.value instead
2. allow to pass exception instance and allow user to provide a callable, takes two arguments, and compares exc. optional - default callable just compares exc args.
3. do not allow to pass exception instances. add an "exc_args" keyword for pytest.raises to allow easier assertion on those.
4. do nothing.

[asottile]

my vote is for the simplest interface: 4 😆