NumPy and SciPy are seeing segfaults with Pytest 5.1 and Python 3.5.6-dbg

[charris]

NumPy and SciPy are seeing segfaults with Pytest 5.1 and Python 3.5.6-dbg. For example, https://travis-ci.org/numpy/numpy/jobs/573017630, which also shows the environment. Plain old Python 3.5.6 passes.

[blueyed]

Can you git-bisect it?

[blueyed]

See also #5751 (comment) (by yours).

Good to have it separated out though probably.

[charris]

Can you git-bisect it?

I was hoping you wouldn't ask :) Yes, but I will need to install 3.5.6-dbg to do so, so it will be a bit of a slog.

[charris]

OK, got the debug version of 3.5.6 installed and can reproduce the error. I'll look to bisecting tomorrow.

[charris]

@blueyed Both pytest 5.0.1 and 5.1.0 crash on python3.6-dbg when the --durations flag is used. This is yet a different bug.

====================================================== slowest 10 test durations =======================================================
python3.6: ../Objects/unicodeobject.c:9486: _PyUnicode_InsertThousandsGrouping: Assertion `0 <= min_width' failed.
Fatal Python error: Aborted

Current thread 0x00007f5f733dc0c0 (most recent call first):
  File "/home/charris/.local/lib/python3.6/site-packages/_pytest/runner.py", line 59 in pytest_terminal_summary
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/manager.py", line 81 in <lambda>
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/manager.py", line 87 in _hookexec
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/hooks.py", line 289 in __call__
  File "/home/charris/.local/lib/python3.6/site-packages/_pytest/terminal.py", line 661 in pytest_sessionfinish
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/callers.py", line 203 in _multicall
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/manager.py", line 81 in <lambda>
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/manager.py", line 87 in _hookexec
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/hooks.py", line 289 in __call__
  File "/home/charris/.local/lib/python3.6/site-packages/_pytest/main.py", line 221 in wrap_session
  File "/home/charris/.local/lib/python3.6/site-packages/_pytest/main.py", line 228 in pytest_cmdline_main
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/manager.py", line 81 in <lambda>
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/manager.py", line 87 in _hookexec
  File "/home/charris/.local/lib/python3.6/site-packages/pluggy/hooks.py", line 289 in __call__
  File "/home/charris/.local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 77 in main
  File "/home/charris/Workspace/numpy.git/build/testenv/lib/python3.6/site-packages/numpy/_pytesttester.py", line 205 in __call__
  File "runtests.py", line 297 in main
  File "runtests.py", line 491 in <module>
Aborted (core dumped)

[pv]

Bisecting with pytest-leaks (whose test suite also triggers an assert with pytest 5.1.0 on debug build): without --durations flag

set -e

mkdir -p tmp
cd tmp

export PYTHON=python3.5-dbg

test -d pytest || git clone https://github.com/pytest-dev/pytest pytest
test -d pytest-leaks || git clone https://github.com/abalkin/pytest-leaks pytest-leaks

git -C pytest-leaks checkout 6e15091aa5bb7e586a4af12e0b04f83f4ff9970d
git -C pytest clean -f -d -x
git -C pytest-leaks clean -f -d -x

test -d env || $PYTHON -mvenv env

source env/bin/activate

pip install -e pytest-leaks
pip uninstall -y pytest
pip install -e pytest

cd pytest

cat <<__EOF__ > check.sh
if python -mpytest ../pytest-leaks/tests; then
    exit 0
else
    exit 1
fi
__EOF__

git bisect start
git bisect good 5.0.1
git bisect bad 5.1.0
git bisect run bash check.sh

says

39ba996133b560f520dbae5d6210aca9f66dadab is the first bad commit
commit 39ba996133b560f520dbae5d6210aca9f66dadab

    assertion.rewrite: use ast.NameConstant(None) directly
    
    `ast.parse("None")` was added/used in 3e6f1fa2d for differences between
    Python 2/3, but we do not support py2 anymore.

Indeed, gdb shows a traceback like

#0  0x00007ffff7c94e75 in raise () from /lib64/libc.so.6
#1  0x00007ffff7c7f895 in abort () from /lib64/libc.so.6
#2  0x00007ffff7c7f769 in __assert_fail_base.cold () from /lib64/libc.so.6
#3  0x00007ffff7c8d566 in __assert_fail () from /lib64/libc.so.6
#4  0x00000000005325b6 in assemble_lnotab (a=a@entry=0x7fffffff0600, i=i@entry=0xe26208) at Python/compile.c:4336
#5  0x00000000005328d5 in assemble_emit (a=a@entry=0x7fffffff0600, i=0xe26208) at Python/compile.c:4427
#6  0x000000000053570e in assemble (c=c@entry=0x7fffffff0760, addNone=addNone@entry=1) at Python/compile.c:4730
#7  0x0000000000537538 in compiler_function (c=0x7fffffff0760, s=0xd96b28, is_async=is_async@entry=0) at Python/compile.c:1705
#8  0x0000000000537734 in compiler_visit_stmt (c=c@entry=0x7fffffff0760, s=0xd96b28) at Python/compile.c:2613
#9  0x00000000005381e7 in compiler_body (c=c@entry=0x7fffffff0760, stmts=0xe23980) at Python/compile.c:1334
#10 0x00000000005388ef in compiler_mod (c=c@entry=0x7fffffff0760, mod=mod@entry=0xd96b60) at Python/compile.c:1354
#11 0x0000000000538a90 in PyAST_CompileObject (mod=mod@entry=0xd96b60, filename=filename@entry=0x7fffe9705ea0, flags=flags@entry=0x7fff
ffff0804, optimize=optimize@entry=-1, arena=arena@entry=0xe44d00) at Python/compile.c:331
#12 0x000000000051f11f in builtin_compile_impl (module=module@entry=0x7fffeac38cd8, source=<optimized out>, filename=0x7fffe9705ea0, mo
de=<optimized out>, flags=<optimized out>, dont_inherit=<optimized out>, optimize=-1) at Python/bltinmodule.c:744
#13 0x000000000051f2aa in builtin_compile (module=0x7fffeac38cd8, args=<optimized out>, kwargs=<optimized out>) at Python/clinic/bltinm
odule.c.h:171
#14 0x00000000004997eb in PyCFunction_Call (func=func@entry=0x7fffeac3f148, args=args@entry=0x7fffe9679490, kwds=kwds@entry=0x7fffe9695
a68) at Objects/methodobject.c:98
#15 0x0000000000523934 in do_call (func=func@entry=0x7fffeac3f148, pp_stack=pp_stack@entry=0x7fffffff09f8, na=na@entry=3, nk=nk@entry=1
) at Python/ceval.c:4986
#16 0x000000000052ff0c in call_function (pp_stack=pp_stack@entry=0x7fffffff09f8, oparg=oparg@entry=259) at Python/ceval.c:4785
#17 0x000000000052d551 in PyEval_EvalFrameEx (f=f@entry=0xd942f8, throwflag=throwflag@entry=0) at Python/ceval.c:3289

[charris]

Both pytest 5.0.1 and 5.1.0 crash on python3.6-dbg when the --durations flag is used.

That turns out to be a Python bug, fixed in 3.7 and 3.8-dev. I've asked for it to be backported to 3.6.

[blueyed]

@pv
This should be fixed with #5755 then.

[blueyed]

@charris
Is it the same for you like with @pv?
And does master work for you then?
(I've marked the other bug as offtopic, since it is filed separately)

[charris]

@blueyed I can confirm that 5.1.1 fixes my problem.

[nicoddemus]

Great, can we close this then?

[charris]

Great, can we close this then?

Fine with me, I was waiting for @pv to sign off.