pytest doesn't display ResourceWarnings that unittest does

[mraspberry]

pytest doesn't seem to be displaying ResourceWarnings that unittest does display.
I'm using pytest 5.0.1 on python 3.6.8 on Ubuntu 18.04.2 LTS running in WSL on Windows 10.

Function being tested that triggers ResourceWarning in unittest but not pytest:

def trigger_resourcewarning(filename):
    return open(filename).read()

pytest test code:

import test_module

def test_module_trigger_resource_warning(tmp_path):
    contents = 'TEST\n'
    filename = tmp_path / 'test.txt'
    filename.write_text(contents)
    assert test_module.trigger_resourcewarning(filename) == contents

unittest test code:

import os
import tempfile
import unittest
import test_module

class TestModuleTriggerResourceWarning(unittest.TestCase):

    def test_module_trigger_resource_warning(self):
        contents = 'TEST\n'
        with tempfile.NamedTemporaryFile(delete=False) as fd:
            filename = fd.name
            fd.write(contents.encode())
        self.addCleanup(os.unlink, filename)
        self.assertEqual(test_module.trigger_resourcewarning(filename), contents)

pytest output:

platform linux -- Python 3.6.8, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/matt/projects/pytest-issue
collected 1 item

tests/test_resource_warning_proof.py .                                   [100%]

unittest output:

/home/matt/projects/pytest-issue/test_module/__init__.py:4: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/tmp1qw9u_dn' mode='r' encoding='UTF-8'>
  return open(filename).read()
.
----------------------------------------------------------------------
Ran 1 test in 0.006s

OK

[RonnyPfannschmidt]

Pathlib closes the fd on the method used in the domonstratiob

[RonnyPfannschmidt]

Oh wait, I did miss tread, please check if it happens with disabled assertion rewriting

[mraspberry]

Seems to have the same behavior

$ pytest --assert=plain
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/matt/projects/pytest-issue
collected 1 item

tests/test_resource_warning_proof.py .                                   [100%]

=========================== 1 passed in 0.05 seconds ===========================

[asottile]

interesting:

def test():
    open(__file__).read()


if __name__ == '__main__':
    exit(test())

$ python3 -Wonce t.py
t.py:2: ResourceWarning: unclosed file <_io.TextIOWrapper name='t.py' mode='r' encoding='UTF-8'>
  open(__file__).read()

$ pytest -qq t.py
.                                                                        [100%]
$ pytest -Wonce -qq t.py
.                                                                        [100%]
=============================== warnings summary ===============================
t.py::test
  /tmp/x/t.py:2: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/x/t.py' mode='r' encoding='UTF-8'>
    open(__file__).read()

-- Docs: https://docs.pytest.org/en/latest/warnings.html
$ pytest -Werror -qq t.py
.                            

[nicoddemus]

We register/unregister handlers at various stages, such as test setup/call/teardown, collection, etc. I think the last hook called where we still capture warnings is pytest_terminal_summary:

pytest/src/_pytest/warnings.py

Lines 131 to 137 in cb15e7c

	@pytest.hookimpl(hookwrapper=True)
	def pytest_terminal_summary(terminalreporter):
	config = terminalreporter.config
	with catch_warnings_for_item(
	config=config, ihook=config.hook, when="config", item=None
	):
	yield

What I want to say is that we are not capturing warnings during the entire session, but only at some points.

I wonder if we could start warning capture during pytest_configure and end during pytest_unconfigure... 🤔

[asottile]

I'm more surprised that when I elevate it to -Werror that it is nowhere to be seen 🤔

[mraspberry]

I'd like to take a stab at fixing this. How should I go about that?

[nicoddemus]

I think I would follow these steps:

1. Try to pinpont at which point exactly the warning is happening. I would start with -Werror, but as @asottile demonstrates that is not reliable as we expected.
2. After that, we should find the adequate pytest hook to implement and catch warnings around it.

---

I've investigated a bit the pytest_configure approach I mentioned in #5676 (comment), but it is not that simple, because we can't reuse catch_warnings, we would need to implement what catch_warnings does manually (and in fact it is what pytest itself used to do), because we need more fine grained control of what happens when a warning is issued, not just log it away for future use.

[asottile]

there's a hint in #8021 -- pytest should probably implement sys.unraisablehook for py3.8+ to help with this