Enhance `RunResult` warning assertion capabilities

[symonk]

while writing some other bits and pieces, I had a use case for checking the warnings omitted, RunResult has a assert_outcomes() that doesn't quite offer warnings= yet the information is already available in there, I suspect there is a good reason why we don't have assert_outcomes(warnings=...) so I propose some additional capabilities on RunResult to handle warnings in isolation.

With assert_outcomes() the full dict comparison may get a bit intrusive as far as warning capture is concerned.

something simple like:

result = pytester.runpytest(...)
result.assert_warnings(count=1)

Thoughts?

[nicoddemus]

A separate function seems good, as warnings are not really part of the test outcome, but something which is captured and displayed separately.

Another use case is perhaps to also check which warnings matched exactly?

result.assert_warnings(warnings=[UserWarning("some message"), RuntimeWarning("other message")])

[symonk]

@nicoddemus sweet, will implement it this weekend, I wanna get a look at per-logger configuration/suppression as a capability as well

[symonk]

@nicoddemus should we use regex for this on stdout/stderr? I'm not sure if theres a way to get actual warning information in there (or if we'd want too but it's a decent learning exercise so i'll have a look around!)

[nicoddemus]

Ahh good point, when running as subprocess this gets tricky.

Scrap that then, let's just go back to your original proposal.

In that sense, when you say:

the full dict comparison may get a bit intrusive as far as warning capture is concerned.

What do you mean exactly? I ask because:

result = pytester.runpytest(...)
result.assert_outcomes(warnings=2)

At first glance seems like it would fit in the purpose of the method nicely, because assert_outcomes just parses the final summary, and we have a warnings count in there.

[symonk]

Thanks @nicoddemus are we thinking something like this: #8952 might be acceptible? I closed it as I felt the assert_outcomes(warnings=n) didn't have any merit, I'm happy to re-open it (and improve the test, i don't think doing it with --strict is a good idea. I've re-opened it, let me know what you think

[nicoddemus]

Yeah I think that's a natural addition. 👍 I didn't take a closer look at #8952 because it was closed by the time I got to it, hehehe.