pytest-asyncio breaks on pytest 5.4.2 when subclassing from unittest.TestCase

[JonathanBrouwer]

System information:
Manjaro Linux with 5.7 kernel
Python 3.8.3
pip 20.1.1

I am using pytest with pytest-asyncio.

I am running the following test:

import unittest
import pytest

class SimpleTest(unittest.TestCase):

    @pytest.mark.asyncio
    async def test_service_request(self):
        assert False

Using pytest 5.4.1 and pytest-asyncio 0.14.0 this test fails (as it should)
Using pytest 5.4.2 and pytest-asyncio 0.14.0 this test passes
Using pytest 6.0.1 and pytest-asyncio 0.14.0 this test passes

On pytest 5.4.2, the following is printed in the console (Note the warning)

============================= test session starts ==============================
platform linux -- Python 3.8.3, pytest-5.4.2, py-1.9.0, pluggy-0.13.1 -- /home/jonathan/projects/LOB-star/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/jonathan/projects/LOB-star/test, inifile: pytest.ini
plugins: asyncio-0.14.0, timeout-1.4.2, repeat-0.8.0, cov-2.10.0
timeout: 10.0s
timeout method: signal
timeout func_only: False
collecting ... collected 1 item

test_x.py::SimpleTest::test_service_request PASSED                       [100%]

=============================== warnings summary ===============================
test_x.py::SimpleTest::test_service_request
  /usr/lib/python3.8/unittest/case.py:633: RuntimeWarning: coroutine 'SimpleTest.test_service_request' was never awaited
    method()

-- Docs: https://docs.pytest.org/en/latest/warnings.html
========================= 1 passed, 1 warning in 0.01s =========================

Pip list output:

Package            Version
------------------ ---------
Adafruit-ADS1x15   1.0.2
Adafruit-GPIO      1.0.3
Adafruit-PureIO    1.1.5
astroid            2.4.2
asyncio            3.4.3
attrs              19.3.0
backcall           0.2.0
bleach             3.1.5
certifi            2020.6.20
chardet            3.0.4
click              7.1.2
colorama           0.4.3
coverage           5.2.1
cycler             0.10.0
decorator          4.4.2
defusedxml         0.6.0
demjson            2.2.4
entrypoints        0.3
flake8             3.8.3
Flask              1.1.2
Flask-Sockets      0.2.1
gevent             20.6.2
gevent-websocket   0.10.1
greenlet           0.4.16
idna               2.10
iniconfig          1.0.0
ipykernel          5.3.4
ipython            7.16.1
ipython-genutils   0.2.0
ipywidgets         7.5.1
isort              4.3.21
itsdangerous       1.1.0
jedi               0.17.2
Jinja2             2.11.2
jsonschema         3.2.0
jupyter            1.0.0
jupyter-client     6.1.6
jupyter-console    6.1.0
jupyter-core       4.6.3
kiwisolver         1.2.0
lazy-object-proxy  1.4.3
lobster-simulator  0.0.3
MarkupSafe         1.1.1
matplotlib         3.3.0
mccabe             0.6.1
mistune            0.8.4
mock               4.0.2
more-itertools     8.4.0
mypy               0.782
mypy-extensions    0.4.3
nbconvert          5.6.1
nbformat           5.0.7
notebook           6.0.3
numpy              1.19.1
packaging          20.4
pandocfilters      1.4.2
parso              0.7.1
pexpect            4.8.0
pickleshare        0.7.5
pigpio             1.46
Pillow             7.2.0
pip                20.2
pluggy             0.13.1
prometheus-client  0.8.0
prompt-toolkit     3.0.5
property           2.2
ptyprocess         0.6.0
py                 1.9.0
pybullet           2.8.4
pycodestyle        2.6.0
pyflakes           2.2.0
Pygments           2.6.1
pygtrie            2.3.3
pylint             2.5.3
pyparsing          2.4.7
pyrsistent         0.16.0
pytest             5.4.1
pytest-asyncio     0.14.0
pytest-cov         2.10.0
pytest-repeat      0.8.0
pytest-timeout     1.4.2
python-dateutil    2.8.1
PyYAML             5.3.1
pyzmq              19.0.1
qtconsole          4.7.5
QtPy               1.9.0
requests           2.24.0
Send2Trash         1.5.0
setuptools         49.2.0
simple-pid         0.2.4
six                1.15.0
spidev             3.5
terminado          0.8.3
testpath           0.4.4
toml               0.10.1
tornado            6.0.4
traitlets          4.3.3
typed-ast          1.4.1
typing-extensions  3.7.4.2
urllib3            1.25.10
wcwidth            0.2.5
webencodings       0.5.1
websockets         8.1
Werkzeug           1.0.1
wheel              0.34.2
widgetsnbextension 3.5.1
wrapt              1.12.1
zope.event         4.4
zope.interface     5.1.0

[The-Compiler]

Bisected to pytest-dev/pytest@da615a4 which is the backport of pytest-dev/pytest#7144.

cc @nicoddemus

[nicoddemus]

Thanks! I will take a look ASAP. 👍

[nicoddemus]

async test methods of unittest.TestCase subclasses worked as a side-effect before because we used to call pytest_pyfunc_call as part of running a unittest test method, but that unfortunately brings a number of problems, including preventing unittest-async frameworks (such as asynctest) from running properly.

Changing the example to use asynctest fixes it:

import asynctest

import pytest

class SimpleTest(asynctest.TestCase):
    
    async def test_service_request(self):
        assert False

You don't even need the mark anymore, as now asynctest handles the test execution.

If you don't subclass from TestCase, then pytest-asyncio will handle the test run and everything works:

import asynctest

import pytest

class SimpleTest(asynctest.TestCase):

    async def test_service_request(self):
        assert False

class Test:

    @pytest.mark.asyncio
    async def test_service_request2(self):
        assert False

@pytest.mark.asyncio
async def test_service_request3():
    assert False

(all 3 tests above fail as expected)

---

The pytest-asyncio documentation doesn't mention support for unittest.TestCase subclasses specifically, so I'm not sure it was intended to be supported or it just so happened to be the case by accident, but adding that support will not be trivial because it would need somehow to interact with the internal pytest plugin.

[JonathanBrouwer]

Ah I did not know asynctest.TestCase was a thing, this resolves my use case. Feel free to close the issue if using unittest.TestCase is not supported.
I would recommend documenting this somewhere visible

[bluetech]

Don't know if it's relevant but Python 3.8 also has unittest.IsolatedAsyncioTestCase.

[nicoddemus]

Thanks @JonathanBrouwer!

I'm closing this for now then. I believe if support for plain unittest.TestCase is desired, we should open new issue. 👍