Integrate pytest-subtests

changelog/1367.feature.rst

@@ -0,0 +1,26 @@
+**Support for subtests** has been added.
+
+:ref:`subtests <subtests>` are an alternative to parametrization, useful in situations where the parametrization values are not all known at collection time.
+
+**Example**
+
+.. code-block:: python
+
+    def contains_docstring(p: Path) -> bool:
+        """Return True if the given Python file contains a top-level docstring."""
+        ...
+
+
+    def test_py_files_contain_docstring(subtests: pytest.Subtests) -> None:
+        for path in Path.cwd().glob("*.py"):
+            with subtests.test(path=str(path)):
+                assert contains_docstring(path)
+
+
+Each assert failure or error is caught by the context manager and reported individually, giving a clear picture of all files that are missing a docstring.
+
+In addition, :meth:`unittest.TestCase.subTest` is now also supported.
+
+.. note::
+
+    This feature is experimental and will likely evolve in future releases. By that we mean that we might change how subtests are reported on failure, but the functionality and how to use it are stable.

doc/en/how-to/index.rst

@@ -16,6 +16,7 @@ Core pytest functionality
    fixtures
    mark
    parametrize
+   subtests
    tmp_path
    monkeypatch
    doctest

doc/en/how-to/parametrize.rst

@@ -20,6 +20,11 @@ pytest enables test parametrization at several levels:
 * `pytest_generate_tests`_ allows one to define custom parametrization
   schemes or extensions.
 
+
+.. note::
+
+    See :ref:`subtests` for an alternative to parametrization.
+
 .. _parametrizemark:
 .. _`@pytest.mark.parametrize`:
 
@@ -194,6 +199,7 @@ To get all combinations of multiple parametrized arguments you can stack
 This will run the test with the arguments set to ``x=0/y=2``, ``x=1/y=2``,
 ``x=0/y=3``, and ``x=1/y=3`` exhausting parameters in the order of the decorators.
 
+
 .. _`pytest_generate_tests`:
 
 Basic ``pytest_generate_tests`` example

doc/en/how-to/subtests.rst

@@ -0,0 +1,95 @@
+.. _subtests:
+
+How to use subtests
+===================
+
+.. versionadded:: 9.0
+
+.. note::
+
+    This feature is experimental. Its behavior, particularly how failures are reported, may evolve in future releases. However, the core functionality and usage are considered stable.
+
+pytest allows for grouping assertions within a normal test, known as *subtests*.
+
+Subtests are an alternative to parametrization, particularly useful when the exact parametrization values are not known at collection time.
+
+
+.. code-block:: python
+
+    # content of test_subtest.py
+
+
+    def test(subtests):
+        for i in range(5):
+            with subtests.test(msg="custom message", i=i):
+                assert i % 2 == 0
+
+Each assertion failure or error is caught by the context manager and reported individually:
+
+.. code-block:: pytest
+
+    $ pytest -q test_subtest.py
+
+
+In the output above:
+
+* Each subtest is reported with the ``,`` character.
+* Subtests are reported first and the "top-level" test is reported at the end on its own.
+* Subtest failures are reported as ``SUBFAIL``.
+
+Note that it is possible to use ``subtests`` multiple times in the same test, or even mix and match with normal assertions
+outside the ``subtests.test`` block:
+
+.. code-block:: python
+
+    def test(subtests):
+        for i in range(5):
+            with subtests.test("stage 1", i=i):
+                assert i % 2 == 0
+
+        assert func() == 10
+
+        for i in range(10, 20):
+            with subtests.test("stage 2", i=i):
+                assert i % 2 == 0
+
+.. note::
+
+    See :ref:`parametrize` for an alternative to subtests.
+
+
+Typing
+------
+
+:class:`pytest.Subtests` is exported so it can be used in type annotations:
+
+.. code-block:: python
+
+    def test(subtests: pytest.Subtests) -> None: ...
+
+.. _parametrize_vs_subtests:
+
+Parametrization vs Subtests
+---------------------------
+
+While :ref:`traditional pytest parametrization <parametrize>` and ``subtests`` are similar, they have important differences and use cases.
+
+
+Parametrization
+~~~~~~~~~~~~~~~
+
+* Happens at collection time.
+* Generates individual tests.
+* Parametrized tests can be referenced from the command line.
+* Plays well with plugins that handle test execution, such as ``--last-failed``.
+* Ideal for decision table testing.
+
+Subtests
+~~~~~~~~
+
+* Happen during test execution.
+* Are not known at collection time.
+* Can be generated dynamically.
+* Cannot be referenced individually from the command line.
+* Plugins that handle test execution cannot target individual subtests.
+* An assertion failure inside a subtest does not interrupt the test, letting users see all failures in the same report.

doc/en/how-to/unittest.rst

@@ -22,17 +22,14 @@ their ``test`` methods in ``test_*.py`` or ``*_test.py`` files.
 
 Almost all ``unittest`` features are supported:
 
-* ``@unittest.skip`` style decorators;
-* ``setUp/tearDown``;
-* ``setUpClass/tearDownClass``;
-* ``setUpModule/tearDownModule``;
+* :func:`unittest.skip`/:func:`unittest.skipIf` style decorators
+* :meth:`unittest.TestCase.setUp`/:meth:`unittest.TestCase.tearDown`
+* :meth:`unittest.TestCase.setUpClass`/:meth:`unittest.TestCase.tearDownClass`
+* :func:`unittest.setUpModule`/:func:`unittest.tearDownModule`
+* :meth:`unittest.TestCase.subTest` (since version ``9.0``)
 
-.. _`pytest-subtests`: https://github.com/pytest-dev/pytest-subtests
 .. _`load_tests protocol`: https://docs.python.org/3/library/unittest.html#load-tests-protocol
 
-Additionally, :ref:`subtests <python:subtests>` are supported by the
-`pytest-subtests`_ plugin.
-
 Up to this point pytest does not have support for the following features:
 
 * `load_tests protocol`_;

doc/en/reference/fixtures.rst

@@ -52,6 +52,9 @@ Built-in fixtures
    :fixture:`pytestconfig`
         Access to configuration values, pluginmanager and plugin hooks.
 
+   :fixture:`subtests`
+        Enable declaring subtests inside test functions.
+
    :fixture:`record_property`
        Add extra properties to the test.
 

doc/en/reference/reference.rst

@@ -572,6 +572,19 @@ The ``request`` fixture is a special fixture providing information of the reques
     :members:
 
 
+.. fixture:: subtests
+
+subtests
+~~~~~~~~
+
+The ``subtests`` fixture enables declaring subtests inside test functions.
+
+**Tutorial**: :ref:`subtests`
+
+.. autoclass:: pytest.Subtests()
+    :members:
+
+
 .. fixture:: testdir
 
 testdir

src/_pytest/config/__init__.py

@@ -278,6 +278,7 @@ def directory_arg(path: str, optname: str) -> str:
     "logging",
     "reports",
     "faulthandler",
+    "subtests",
 )
 
 builtin_plugins = {

src/_pytest/deprecated.py

@@ -25,6 +25,7 @@
     "pytest_catchlog",
     "pytest_capturelog",
     "pytest_faulthandler",
+    "pytest_subtests",
 }
 
 

src/_pytest/reports.py

@@ -251,7 +251,6 @@ def _report_unserialization_failure(
     raise RuntimeError(stream.getvalue())
 
 
-@final
 class TestReport(BaseReport):
     """Basic test report object (also used for setup and teardown calls if
     they fail).