Race condition in test_python_to_cpp_to_python_from_process test on 3.9?

[YannickJadoul]

Not sure what's going on, but this test failed again on 3.9, completely unrelated to the actual changes.
I don't know what that test is about, but I thought I'd log it.

Do we have a race condition? Does CPython 3.9 have a race condition?

__________________ test_python_to_cpp_to_python_from_process ___________________

    def test_python_to_cpp_to_python_from_process():
        """Makes sure there is no GIL deadlock when using processes.
    
        This test is for completion, but it was never an issue.
        """
>       assert _run_in_process(_python_to_cpp_to_python) == 0
E       assert -11 == 0
E        +  where -11 = _run_in_process(_python_to_cpp_to_python)

test_gil_scoped.py:81: AssertionError

[henryiii]

We also see a segfault here occasionally for Python 3.9:

-------------------------------------------------------------------------------
Restart the interpreter
-------------------------------------------------------------------------------
/Users/runner/work/pybind11/pybind11/tests/test_embed/test_interpreter.cpp:108
...............................................................................

/Users/runner/work/pybind11/pybind11/tests/test_embed/test_interpreter.cpp:108: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGSEGV - Segmentation violation signal

===============================================================================
test cases:  4 |  3 passed | 1 failed
assertions: 16 | 15 passed | 1 failed

make[3]: *** [tests/test_embed/CMakeFiles/cpptest] Error 139
make[2]: *** [tests/test_embed/CMakeFiles/cpptest.dir/all] Error 2
make[1]: *** [tests/test_embed/CMakeFiles/cpptest.dir/rule] Error 2

[EricCousineau-TRI]

Also in test_python_to_cpp_to_python_from_thread_multiple_parallel:
#2396 (comment)

[henryiii]

Is this mostly macOS? I could add an xfail(env.MACOS and env.pyv >= 3.9, reason="Occasional failure, TBD", strict=False) here, to help with unrelated PRs, but a little worried that might cause us to overlook it until Python 3.9 is released and it's too late to fix it if it's on Python's end.

[henryiii]

test_python_to_cpp_to_python_from_thread_multiple_sequential too. Seems to always be macOS.

[YannickJadoul]

Anything on this? Is this still happening? If so, we should probably report this to Python before the final release is out.

[YannickJadoul]

Another instantiation of this fluke?

 _______________________ test_multiple_inheritance_python _______________________

    @pytest.mark.skipif("env.PYPY and env.PY2")
    @pytest.mark.xfail("env.PYPY and not env.PY2")
    def test_multiple_inheritance_python():
    
        class MI1(m.Base1, m.Base2):
            def __init__(self, i, j):
                m.Base1.__init__(self, i)
                m.Base2.__init__(self, j)
    
        class B1(object):
            def v(self):
                return 1
    
        class MI2(B1, m.Base1, m.Base2):
            def __init__(self, i, j):
                B1.__init__(self)
                m.Base1.__init__(self, i)
                m.Base2.__init__(self, j)
    
        class MI3(MI2):
            def __init__(self, i, j):
                MI2.__init__(self, i, j)
    
        class MI4(MI3, m.Base2):
            def __init__(self, i, j):
                MI3.__init__(self, i, j)
                # This should be ignored (Base2 is already initialized via MI2):
                m.Base2.__init__(self, i + 100)
    
        class MI5(m.Base2, B1, m.Base1):
            def __init__(self, i, j):
                B1.__init__(self)
                m.Base1.__init__(self, i)
                m.Base2.__init__(self, j)
    
        class MI6(m.Base2, B1):
            def __init__(self, i):
                m.Base2.__init__(self, i)
                B1.__init__(self)
    
        class B2(B1):
            def v(self):
                return 2
    
        class B3(object):
            def v(self):
                return 3
    
        class B4(B3, B2):
            def v(self):
                return 4
    
        class MI7(B4, MI6):
            def __init__(self, i):
                B4.__init__(self)
                MI6.__init__(self, i)
    
        class MI8(MI6, B3):
            def __init__(self, i):
                MI6.__init__(self, i)
                B3.__init__(self)
    
        class MI8b(B3, MI6):
            def __init__(self, i):
                B3.__init__(self)
                MI6.__init__(self, i)
    
        mi1 = MI1(1, 2)
        assert mi1.foo() == 1
        assert mi1.bar() == 2
    
        mi2 = MI2(3, 4)
        assert mi2.v() == 1
        assert mi2.foo() == 3
        assert mi2.bar() == 4
    
>       mi3 = MI3(5, 6)

../../tests/test_multiple_inheritance.py:133: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../tests/test_multiple_inheritance.py:78: in __init__
    MI2.__init__(self, i, j)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_multiple_inheritance.test_multiple_inheritance_python.<locals>.MI3 object at 0x7f6a95e1f0e0>
i = 5, j = 6

    def __init__(self, i, j):
        B1.__init__(self)
>       m.Base1.__init__(self, i)
E       TypeError: __init__(self, ...) called with invalid `self` argument

../../tests/test_multiple_inheritance.py:73: TypeError

EDIT: This is not, though. This is related to #2564

[henryiii]

Running 3.9 macOS locally, the tests pass, but I get a segfault popup once in a while:

0   org.python.python             	0x000000010bf4f34a meth_dealloc + 120
1   org.python.python             	0x000000010bf2145a instancemethod_dealloc + 65
2   org.python.python             	0x000000010bf43feb PyDict_Clear + 409
3   org.python.python             	0x000000010bf616ec type_clear + 59
4   org.python.python             	0x000000010c00ef00 collect + 2558
5   org.python.python             	0x000000010c00e4f1 _PyGC_CollectNoFail + 67
6   org.python.python             	0x000000010bfe119c _PyImport_Cleanup + 1545
7   org.python.python             	0x000000010bff1680 Py_FinalizeEx + 160
8   org.python.python             	0x000000010bff2167 Py_Exit + 13
9   org.python.python             	0x000000010bff8117 handle_system_exit + 35
10  org.python.python             	0x000000010bff676f _PyErr_PrintEx + 40
11  org.python.python             	0x000000010bff64e8 PyRun_SimpleStringFlags + 116
12  org.python.python             	0x000000010c00d3c7 Py_RunMain + 425
13  org.python.python             	0x000000010c00dc86 pymain_main + 306
14  org.python.python             	0x000000010c00dcd4 Py_BytesMain + 42
15  libdyld.dylib                 	0x00007fff6910dcc9 start + 1

This happens when one of the XFails fails instead of XPasses.

[YannickJadoul]

Running 3.9 macOS locally, the tests pass, but I get a segfault popup once in a while:

This is another one, though, it seems. This is described in #2558 and debugged (and hopefully soon resolved) in #2576. Doesn't seem to be related to the issue described here, though.

[YannickJadoul]

More info in #2754