[WIP] Debug flaky coverage

[blueyed]

No description provided.

[blueyed]

I am a bit surprised that the del (appears) to work, since with the following code it triggers it:

import logging

# f = open("/tmp/foo", "a")

log_file_handler = logging.FileHandler("temp.log", mode="w", encoding="UTF-8")

root_logger = logging.getLogger()
root_logger.addHandler(log_file_handler)
root_logger.removeHandler(log_file_handler)
root_logger.error("error")

del log_file_handler

(run with python -Werror::ResourceWarning t.py)

logging itself handles an internal list (using weakrefs), which normally
appears to hold until the logger is there (it has it in self.handlers).
But since we're removing it this reference gets lost.

Anyway, I think this gets triggered by some other test using "-W error" in another thread, while this test is running (xdist)
(test_as_errors, https://github.com/blueyed/pytest/blob/14ee268d925e375a1fc032b96262558e3fb0864b/testing/test_warnings.py#L92).

1. I think the test_as_errors test should be run in a subprocess, but maybe we should use some locking in our own logging code?! (e.g. via logging's existing locks?).
2. the code should be covered reliably (see test_collect_capturing: cover captured stderr #4989).

[blueyed]

Closing in favor of #4988 and #4989.