Python 3.9: DeprecationWarning: The parser module

[hroncok]

Hello. We are trying to build pytest with Python 3.9 in Fedora.

We are still on 4.x for various reasons, but the code raising DeprecationWarning is still present on 5.x.

This is outptut of the tests:

+ /builddir/build/BUILDROOT/pytest-4.6.8-1.fc32.x86_64/usr/bin/pytest-3.9 -r s testing
============================= test session starts ==============================
platform linux -- Python 3.9.0a2, pytest-4.6.8, py-1.8.0, pluggy-0.13.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/builddir/build/BUILD/pytest-4.6.8/.hypothesis/examples')
rootdir: /builddir/build/BUILD/pytest-4.6.8, inifile: tox.ini, testpaths: testing
plugins: hypothesis-4.23.8
collected 2452 items                                                           

testing/acceptance_test.py .......                                       [  0%]
testing/deprecated_test.py ....                                          [  0%]
testing/test_argcomplete.py ..                                           [  0%]
testing/test_assertion.py .............................................. [  2%]
.....s.                                                                  [  2%]
testing/test_assertrewrite.py ....................                       [  3%]
testing/test_capture.py ..........s.........s............s........s...s. [  5%]
                                                                         [  5%]
testing/test_collection.py .                                             [  5%]
testing/test_compat.py ......                                            [  5%]
testing/test_config.py .............................                     [  6%]
testing/test_conftest.py ...........                                     [  7%]
testing/test_doctest.py .....                                            [  7%]
testing/test_entry_points.py .                                           [  7%]
testing/test_junitxml.py ....                                            [  7%]
testing/test_mark.py ................                                    [  8%]
testing/test_monkeypatch.py .................ss.............s.           [  9%]
testing/test_nodes.py .........                                          [ 10%]
testing/test_nose.py ss                                                  [ 10%]
testing/test_parseopt.py ............................                    [ 11%]
testing/test_pastebin.py .                                               [ 11%]
testing/test_pathlib.py .....................................            [ 12%]
testing/test_pdb.py .                                                    [ 13%]
testing/test_pluginmanager.py .........                                  [ 13%]
testing/test_pytester.py ...............                                 [ 13%]
testing/test_recwarn.py ....................................             [ 15%]
testing/test_resultlog.py ....                                           [ 15%]
testing/test_runner.py .............                                     [ 16%]
testing/test_skipping.py .                                               [ 16%]
testing/test_terminal.py .............................                   [ 17%]
testing/test_tmpdir.py ....s............                                 [ 18%]
testing/test_warnings.py .                                               [ 18%]
testing/code/test_code.py .........s......                               [ 18%]
testing/code/test_excinfo.py ..............s........s................... [ 20%]
..................................................                       [ 22%]
testing/code/test_source.py ...........FF............................... [ 24%]
.......................                                                  [ 25%]
testing/freeze/tests/test_trivial.py ..                                  [ 25%]
testing/io/test_saferepr.py .......                                      [ 25%]
testing/logging/test_fixture.py ........                                 [ 25%]
testing/logging/test_formatter.py ..                                     [ 26%]
testing/logging/test_reporting.py ..                                     [ 26%]
testing/python/approx.py .sssss..........................sssss.......... [ 28%]
.ss.                                                                     [ 28%]
testing/python/collect.py ..                                             [ 28%]
testing/python/fixtures.py ....                                          [ 28%]
testing/python/integration.py ..x.                                       [ 28%]
testing/python/metafunc.py .................................             [ 29%]
testing/python/raises.py .....................ssssss                     [ 31%]
testing/freeze/tests/test_doctest.txt .                                  [ 31%]
testing/acceptance_test.py ............................................. [ 32%]
..x...................                                                   [ 33%]
testing/deprecated_test.py ...........                                   [ 34%]
testing/test_assertion.py .......................................        [ 35%]
testing/test_assertrewrite.py ........................s..ssss........... [ 37%]
......                                                                   [ 37%]
testing/test_cacheprovider.py .......................................... [ 39%]
....                                                                     [ 39%]
testing/test_capture.py ......x....................s.................... [ 41%]
.s..................                                                     [ 42%]
testing/test_collection.py ............................x................ [ 44%]
..........................                                               [ 45%]
testing/test_compat.py ..                                                [ 45%]
testing/test_config.py ..........x...................................... [ 47%]
......................................                                   [ 49%]
testing/test_conftest.py .........................................       [ 50%]
testing/test_doctest.py ................................................ [ 52%]
...................................................s.                    [ 54%]
testing/test_helpconfig.py .......                                       [ 55%]
testing/test_junitxml.py ............................................... [ 57%]
......s..x..............                                                 [ 57%]
testing/test_mark.py ..................................................x [ 60%]
.........                                                                [ 60%]
testing/test_monkeypatch.py ....                                         [ 60%]
testing/test_nodes.py .                                                  [ 60%]
testing/test_nose.py sssssssssssssssss                                   [ 61%]
testing/test_pastebin.py ...                                             [ 61%]
testing/test_pdb.py ...................                                  [ 62%]
testing/test_pluginmanager.py ...............                            [ 62%]
testing/test_pytester.py x.....................                          [ 63%]
testing/test_recwarn.py ..                                               [ 63%]
testing/test_reports.py ..............                                   [ 64%]
testing/test_resultlog.py ......                                         [ 64%]
testing/test_runner.py .....................sssssss....ssss.x........... [ 66%]
........                                                                 [ 66%]
testing/test_runner_xunit.py .............                               [ 67%]
testing/test_session.py .......................                          [ 68%]
testing/test_skipping.py ...s........................................... [ 70%]
............................                                             [ 71%]
testing/test_stepwise.py ........                                        [ 71%]
testing/test_terminal.py .......................................s....... [ 73%]
............................................sss.....s                    [ 75%]
testing/test_tmpdir.py ..........                                        [ 76%]
testing/test_unittest.py .....................sssssss..................s [ 78%]
.....                                                                    [ 78%]
testing/test_warnings.py .......s.....................s........          [ 79%]
testing/code/test_excinfo.py ...                                         [ 80%]
testing/examples/test_issue519.py .                                      [ 80%]
testing/logging/test_fixture.py ..                                       [ 80%]
testing/logging/test_reporting.py .....................................  [ 81%]
testing/python/approx.py .                                               [ 81%]
testing/python/collect.py .............................................. [ 83%]
...........................                                              [ 84%]
testing/python/fixtures.py ............................................. [ 86%]
........................x............................................... [ 89%]
.....................................................                    [ 91%]
testing/python/integration.py ....ssss.......                            [ 92%]
testing/python/metafunc.py ............................................. [ 94%]
.....x..............                                                     [ 94%]
testing/python/raises.py ...                                             [ 95%]
testing/python/setup_only.py ........................                    [ 96%]
testing/python/setup_plan.py .                                           [ 96%]
testing/python/show_fixtures_per_test.py ......                          [ 96%]
testing/test_modimport.py .............................................. [ 98%]
....                                                                     [ 98%]
testing/test_parseopt.py s                                               [ 98%]
testing/test_pdb.py sssssssssssssssssssssssssssssssssss                  [ 99%]
testing/test_pytester.py s                                               [ 99%]
testing/test_terminal.py ss                                              [ 99%]
testing/test_unittest.py s                                               [100%]

=================================== FAILURES ===================================
_______________________________ test_isparseable _______________________________

    def test_isparseable():
>       assert Source("hello").isparseable()

/builddir/build/BUILD/pytest-4.6.8/testing/code/test_source.py:130: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest._code.source.Source object at 0x7f25d53afaf0>, deindent = True

    def isparseable(self, deindent=True):
        """ return True if source is parseable, heuristically
            deindenting it by default.
        """
>       from parser import suite as syntax_checker
E       DeprecationWarning: The parser module is deprecated and will be removed in future versions of Python

/builddir/build/BUILDROOT/pytest-4.6.8-1.fc32.x86_64/usr/lib/python3.9/site-packages/_pytest/_code/source.py:126: DeprecationWarning
__________________________ TestAccesses.test_getrange __________________________

self = <test_source.TestAccesses object at 0x7f25d5c02670>

    def test_getrange(self):
        x = self.source[0:2]
>       assert x.isparseable()

/builddir/build/BUILD/pytest-4.6.8/testing/code/test_source.py:150: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest._code.source.Source object at 0x7f25d59a7190>, deindent = True

    def isparseable(self, deindent=True):
        """ return True if source is parseable, heuristically
            deindenting it by default.
        """
>       from parser import suite as syntax_checker
E       DeprecationWarning: The parser module is deprecated and will be removed in future versions of Python

/builddir/build/BUILDROOT/pytest-4.6.8-1.fc32.x86_64/usr/lib/python3.9/site-packages/_pytest/_code/source.py:126: DeprecationWarning
=============================== warnings summary ===============================
testing/test_warnings.py::test_group_warnings_by_message
  /builddir/build/BUILD/pytest-4.6.8/testing/test_warnings.py:701: PytestExperimentalApiWarning: testdir.copy_example is an experimental api that may change over time
    testdir.copy_example("warnings/test_group_warnings_by_message.py")

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== short test summary info ============================
...
= 2 failed, 2309 passed, 130 skipped, 11 xfailed, 1 warnings in 167.85 seconds =

I will try to reproduce outside of our build environment.

[hroncok]

$ python3.9 -m venv __venv__
$ . __venv__/bin/activate
$ pip install -e .
$ pip install xmlschema hypothesis
$ pytest -r s testing --maxfail 2
============================= test session starts ==============================
platform linux -- Python 3.9.0a2, pytest-5.3.3.dev36+g26a2e1aba, py-1.8.1, pluggy-0.13.1
rootdir: .../pytest, inifile: tox.ini, testpaths: testing
plugins: hypothesis-5.1.0
collected 2637 items

...
testing/code/test_source.py ...........FF

=================================== FAILURES ===================================
_______________________________ test_isparseable _______________________________

    def test_isparseable() -> None:
>       assert Source("hello").isparseable()

.../pytest/testing/code/test_source.py:125: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest._code.source.Source object at 0x7f260e2130a0>, deindent = True

    def isparseable(self, deindent: bool = True) -> bool:
        """ return True if source is parseable, heuristically
            deindenting it by default.
        """
>       from parser import suite as syntax_checker
E       DeprecationWarning: The parser module is deprecated and will be removed in future versions of Python

.../pytest/src/_pytest/_code/source.py:143: DeprecationWarning
__________________________ TestAccesses.test_getrange __________________________

self = <test_source.TestAccesses object at 0x7f260d90da60>

    def test_getrange(self) -> None:
        x = self.source[0:2]
>       assert x.isparseable()

.../pytest/testing/code/test_source.py:146: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest._code.source.Source object at 0x7f260d4d7e20>, deindent = True

    def isparseable(self, deindent: bool = True) -> bool:
        """ return True if source is parseable, heuristically
            deindenting it by default.
        """
>       from parser import suite as syntax_checker
E       DeprecationWarning: The parser module is deprecated and will be removed in future versions of Python

.../pytest/src/_pytest/_code/source.py:143: DeprecationWarning
=========================== short test summary info ============================
...
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 2 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
=================== 2 failed, 606 passed, 7 skipped in 3.76s ===================
$ pip freeze
attrs==19.3.0
elementpath==1.3.3
hypothesis==5.1.0
more-itertools==8.0.2
packaging==19.2
pluggy==0.13.1
py==1.8.1
pyparsing==2.4.6
-e git+git@github.com:pytest-dev/pytest.git@26a2e1aba7d9e157b4537bbeb24ad372e8f8b919#egg=pytest
six==1.13.0
sortedcontainers==2.1.0
wcwidth==0.1.8
xmlschema==1.0.18

[hroncok]

This workarounds the issue:

diff --git a/tox.ini b/tox.ini
index d33a62224..388b41d5b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -168,6 +168,8 @@ filterwarnings =
     default:invalid escape sequence:DeprecationWarning
     # ignore use of unregistered marks, because we use many to test the implementation
     ignore::_pytest.warning_types.PytestUnknownMarkWarning
+    # https://github.com/pytest-dev/pytest/issues/6404
+    ignore:.*The parser module is deprecated.*:DeprecationWarning
 pytester_example_dir = testing/example_scripts
 markers =
     # dummy markers for testing

[nicoddemus]

Thanks @hugovk!

The replacement would be ast.parse, but I noticed isparseable isn't even used so I just removed it in #6405. 👍

[hroncok]

Thank You! I would probably just spend time porting it to ast.

[nicoddemus]

Indeed it was trivial (I did it before wondering if it was used), it was just a matter of replacing parser.suite by ast.parse basically.

[hroncok]

Would you accept a backport of that to the 4.6-maintenance branch, once merged to master, or should we carry the workaround until we switch to 5 (that should hopefully happen before Python 3.10)?

[nicoddemus]

Would you accept a backport of that to the 4.6-maintenance branc

Sure! 👍

[nicoddemus]

Backport: #6406 (went ahead and did it myself)