Option to disable impending change

[aroberge]

I just installed the latest pytest version (7.2.0) and saw multiple examples which stating the following:

 PytestReturnNotNoneWarning: Expected None, but test_example.py::test returned 'Hello', which will be an error in a future version of pytest.  Did you mean to use `assert` instead of `return`?

In one of my projects, I have over 400 tests which do use assert and return. The return statement is used to create documentation. See for example https://friendly-traceback.github.io/docs/tracebacks_en_3.9.html

A typical test function will have the following outline:

import friendly_traceback

def test_example():
    try:
        SpecificException("specific message")  # simplification for this issue on Github
    except SpecificException as e:
        message = str(e)
        friendly_traceback.explain_traceback(redirect="capture")
    
    result = friendly_traceback.get_output()
    assert "specific message" in result  # often varies in different Python versions
    if friendly_traceback.get_lang() == "en":
        assert "expected explanation in English from friendly_traceback" in result
    return result, message

If you browse the documentation of friendly_traceback, you will see that these 400+ tests help to document what the output is for various Python version and languages (English, French, Spanish, ...). This is invaluable to understand in context if a translation has been done correctly.
Returning the message helps to document changes from one Python version to the next.

I would really like if there were a way to enable having test functions returning something else than None not being an error (and being able to turn off PytestReturnNotNoneWarning.

[asottile]

imo you're trying to do too much in tests -- the warning is following the precedent set by python unittest

another way I've done this in the past is to explicitly save off the results using a fixture -- but even that seems a little to involved for a testsuite

[aroberge]

imo you're trying to do too much in tests

I'm doing this to avoid duplication. The ultimate goals of friendly_traceback are to add beginner-friendly explanation to every error messages arising from the Python standard library (and other popular ones, such as pandas and numpy), with translations in as many human languages as possible.

Just today, I found out by running my test suite with Python 3.12.0a1 that a new SyntaxError message had been added, and that an extra spurious space had been added between two word in an error message.
By running the same code for the tests and the documentation, I can keep adding cases without having to duplicate them (which would be error prone).

If I write something like:

def test_example():
    ...
    if running_with_pytest:
        return
    return output_for_doc

would pytest, with its AST manipulation, still detect that I can potentially return something other than None and flag this as an error? (I just tested, and it appears to work for now, without producing a warning.)

[asottile]

python's unittest and pytest will warn when a non-None value is returned -- it's not at the ast level

[aroberge]

Thanks. I'll figure out a way to modify my tests to return something only when not run with pytest.