`pytest-xdist` breaks asyncio-based code expecting to be run in the main thread

[webknjaz]

When asyncio-based software needs to register signal handlers, calling add_signal_handler() only works when the current thread is main and raises a RuntimeError otherwise.

When I first tried integrating pytest-xdist into aiohttp's test suite a few years ago, we've faced a problem that about 15% of pytest invocations would hit this and had to disable the plugin:

• Optimize tests aio-libs/aiohttp#3419
• pytest-aiohttp + pytest-xdist cause RuntimeError: set_wakeup_fd only works in main thread aio-libs/aiohttp#3450

The reproducer is to clone aio-libs/aiohttp, add -n auto to addopts somewhere in setup.cfg, and re-run make test until you see the traceback. On aiohttp side, it's coming from the watcher.attach_loop() call invoked from the pytest plugin @ https://github.com/aio-libs/aiohttp/blob/742a8b6/aiohttp/pytest_plugin.py#L161.

Now that I've had some time to try to debug what's happening, I've stuck a debugger right before the line that raises RuntimeError and confirmed that the thread wasn't main:

>>> import threading
>>> threading.enumerate()
[<_MainThread(MainThread, started 140666216200000)>, <_DummyThread(Dummy-1, started daemon 140666196751936)>]
>>> threading.current_thread()
<_DummyThread(Dummy-1, started daemon                 
140666196751936)>

@asvetlov It should be possible to fix this on the asyncio side with ThreadedChildWatcher (https://bugs.python.org/issue35621 / python/cpython#14344) but it only appeared in the stdlib since Python 3.8: https://stackoverflow.com/a/58614689/595220.

It's hard to pinpoint where that thread is coming from but it appears to be execnet's fault:

pytest-xdist/src/xdist/workermanage.py

Line 41 in c5fadcd

self.group = execnet.Group()

+ https://github.com/pytest-dev/execnet/blob/116637ab183bc078647f281b52e662650e8178e7/execnet/multi.py#L27.

@nicoddemus do you have any insight into this? I'm rather lost at this point. Is there any way to avoid this race condition?

[RonnyPfannschmidt]

This is a issue that was Int to execnet with the addition of thread models for gevent /eventlet.

A workaround is to downgrade execnet.

Execnet currently is unmaintained.

[webknjaz]

I tried looking into it and thought that maybe using hasprimary=True would help (https://github.com/pytest-dev/execnet/blob/116637a/execnet/gateway_base.py#L165) but it just gets stuck on worker allocation...

@RonnyPfannschmidt is there a known execnet version I need to downgrade to?

[webknjaz]

Tried execnet<1.2 but that's too old for modern Pythons and it explodes trying to call deprecated isAlive().

Sounds like we'll need a backport of ThreadedChildWatcher after all.

@RonnyPfannschmidt do you think there's any reasonable way to have a fixture forcing xdist to bypass scheduling a specific test in workers and just execute it in the main process?

[RonnyPfannschmidt]

When using xdist, no test run in the main process

[graingert]

arn't tests in xdist workers also run in only the main thread?

[RonnyPfannschmidt]

So there is no reasonable way

[graingert]

I've stuck a debugger right before the line that raises RuntimeError and confirmed that the thread wasn't main:

@webknjaz which process was running the breakpoint()? the xdist leader or an xdist worker?

you should be able to track the stack of what created the <_DummyThread(Dummy-1, started daemon 140666196751936)> with tracemalloc

[RonnyPfannschmidt]

@graingert an addition to execnet broke that assumption

[graingert]

@RonnyPfannschmidt do you have a link to the commit that introduced it?

[RonnyPfannschmidt]

Not right now, I'm on mobile atm, and no longer a maintainer for execnet

[webknjaz]

@graingert can't do that right now but I think the thread is spawned by this call https://github.com/pytest-dev/execnet/blob/116637ab183bc078647f281b52e662650e8178e7/execnet/gateway_base.py#L315

[graingert]

so it looks like pytest-xdist will run tests on the main thread almost all the time, it's only if there are concurrent remote_exec calls that execnet will actually spawn a thread, eg we always want to hit: https://github.com/pytest-dev/execnet/blob/116637ab183bc078647f281b52e662650e8178e7/execnet/gateway_base.py#L299-L302

so I had a look around for places that remote_exec is called without an associated wait_close, and only found:

diff --git a/execnet/gateway.py b/execnet/gateway.py
index 8e2c1de..8a0da92 100644
--- a/execnet/gateway.py
+++ b/execnet/gateway.py
@@ -75,7 +75,10 @@ class Gateway(gateway_base.BaseGateway):
         """ return some sys/env information from remote. """
         if update or not hasattr(self, "_cache_rinfo"):
             ch = self.remote_exec(rinfo_source)
-            self._cache_rinfo = RInfo(ch.receive())
+            try:
+                self._cache_rinfo = RInfo(ch.receive())
+            finally:
+                ch.wait_close()
         return self._cache_rinfo
 
     def hasreceiver(self):
-- 
2.30.0

maybe this patch will help?

[webknjaz]

Maybe, to verify that we'd have to run a lot of parallel pytest invocations with the "is main thread" check. I'll try to do this when I have a minute.

[graingert]

it looks like fixing this test will fix the problem properly: https://github.com/pytest-dev/execnet/blob/116637ab183bc078647f281b52e662650e8178e7/testing/test_threadpool.py#L77-L103