Warning about assert None from XFAIL tests

[oscarbenjamin]

Running pytest master I see warnings from XFAIL tests about asserting None. The warnings suggests to change the assert to assert obj is None but the reason we are asserting None is because the test fails (hence XFAIL).

I don't think these warnings should be emitted from XFAIL tests.

Using pytest master:

import pytest

@pytest.mark.XFAIL
def test_f():
    assert g()

def g():
    # Shouldn't return None but doesn't work properly...
    return None

Running the tests gives:

$ pytest test_f.py 
====================================================== test session starts =======================================================
platform darwin -- Python 2.7.10, pytest-4.1.1, py-1.7.0, pluggy-0.8.0
rootdir: /Users/enojb/current/sympy/pytest, inifile: tox.ini
plugins: xdist-1.26.0, forked-0.2, doctestplus-0.3.0.dev0
collected 1 item                                                                                                                 

test_f.py F                                                                                                                [100%]

============================================================ FAILURES ============================================================
_____________________________________________________________ test_f _____________________________________________________________

    @pytest.mark.XFAIL
    def test_f():
>       assert g()
E       PytestWarning: asserting the value None, please use "assert is None"

test_f.py:5: PytestWarning
==================================================== short test summary info =====================================================
FAIL test_f.py::test_f
==================================================== 1 failed in 0.08 seconds ====================================================

[blueyed]

What would you assert if it worked? Also just assert g()?
Have you considered using assert g() is not None then?

[RonnyPfannschmidt]

Xfail should be lowercase we need strict marks as default

[oscarbenjamin]

I'm seeing a few warnings haven't worked through them yet but here's one: https://github.com/sympy/sympy/blob/sympy-1.3/sympy/assumptions/tests/test_matrices.py#L41

It's not clear to me that that test should be asserting is not None. It possibly could be changed to is True but I'm not sure.

[asottile]

I believe the mistake here is pytest.mark.XFAIL vs pytest.mark.xfail

trying in a little script this works as expected, even with assert None:

pytest.mark.xfail

$ cat t.py
import pytest

@pytest.mark.xfail
def test():
    assert None
$ pytest t.py 
============================= test session starts ==============================
platform linux -- Python 3.6.7, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
rootdir: /tmp/t, inifile:
collected 1 item                                                               

t.py x                                                                   [100%]

=============================== warnings summary ===============================
t.py::test
  /tmp/t/t.py:5: PytestWarning: asserting the value None, please use "assert is None"
    assert None

-- Docs: https://docs.pytest.org/en/latest/warnings.html
==================== 1 xfailed, 1 warnings in 0.07 seconds =====================
$ pytest -Werror t.py 
============================= test session starts ==============================
platform linux -- Python 3.6.7, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
rootdir: /tmp/t, inifile:
collected 1 item                                                               

t.py x                                                                   [100%]

========================== 1 xfailed in 0.03 seconds ===========================

pytest.mark.XFAIL

switching to the incorrect name:

$ cat t.py
import pytest

@pytest.mark.XFAIL
def test():
    assert None
$ pytest t.py
============================= test session starts ==============================
platform linux -- Python 3.6.7, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
rootdir: /tmp/t, inifile:
collected 1 item                                                               

t.py F                                                                   [100%]

=================================== FAILURES ===================================
_____________________________________ test _____________________________________

    @pytest.mark.XFAIL
    def test():
>       assert None
E       assert None

t.py:5: AssertionError
=============================== warnings summary ===============================
t.py::test
  /tmp/t/t.py:5: PytestWarning: asserting the value None, please use "assert is None"
    assert None

-- Docs: https://docs.pytest.org/en/latest/warnings.html
===================== 1 failed, 1 warnings in 0.05 seconds =====================
$ pytest -Werror t.py
============================= test session starts ==============================
platform linux -- Python 3.6.7, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
rootdir: /tmp/t, inifile:
collected 1 item                                                               

t.py F                                                                   [100%]

=================================== FAILURES ===================================
_____________________________________ test _____________________________________

    @pytest.mark.XFAIL
    def test():
>       assert None
E       _pytest.warning_types.PytestWarning: asserting the value None, please use "assert is None"

t.py:5: PytestWarning
=========================== 1 failed in 0.04 seconds ===========================

[asottile]

unrelated, I'd suggest using pytest instead of the deprecated py.test here

[oscarbenjamin]

Maybe I'm misunderstanding something here. For my example above test_f.py at the top of this issue I changed it from XFAIL to xfail and the problem does indeed go away but only if I'm running in the pytest root directory:

$ . venv/bin/activate
$ pip install -e pytest/
Obtaining file:///Users/enojb/current/sympy/pytest
  Installing build dependencies ... done
Requirement already satisfied: py>=1.5.0 in ./venv/lib/python3.7/site-packages (from pytest==4.1.2.dev19+g57bf9d67) (1.7.0)
Requirement already satisfied: six>=1.10.0 in ./venv/lib/python3.7/site-packages (from pytest==4.1.2.dev19+g57bf9d67) (1.11.0)
Requirement already satisfied: setuptools in ./venv/lib/python3.7/site-packages (from pytest==4.1.2.dev19+g57bf9d67) (39.0.1)
Requirement already satisfied: attrs>=17.4.0 in ./venv/lib/python3.7/site-packages (from pytest==4.1.2.dev19+g57bf9d67) (18.2.0)
Requirement already satisfied: more-itertools>=4.0.0 in ./venv/lib/python3.7/site-packages (from pytest==4.1.2.dev19+g57bf9d67) (4.3.0)
Requirement already satisfied: atomicwrites>=1.0 in ./venv/lib/python3.7/site-packages (from pytest==4.1.2.dev19+g57bf9d67) (1.2.1)
Requirement already satisfied: pluggy>=0.7 in ./venv/lib/python3.7/site-packages (from pytest==4.1.2.dev19+g57bf9d67) (0.8.0)
Installing collected packages: pytest
  Running setup.py develop for pytest
Successfully installed pytest
$ cd pytest
$ cat test_f.py 
import pytest

@pytest.mark.xfail
def test_f():
    assert g()

def g():
    # Shouldn't return None but doesn't work properly...
    return None
(venv) $ pytest test_f.py 
====================================================== test session starts =======================================================
platform darwin -- Python 3.7.1, pytest-4.1.2.dev19+g57bf9d67, py-1.7.0, pluggy-0.8.0
rootdir: /Users/enojb/current/sympy/pytest, inifile: tox.ini
plugins: xdist-1.26.0, forked-0.2, doctestplus-0.3.0.dev0
collected 1 item                                                                                                                 

test_f.py x                                                                                                                [100%]
==================================================== short test summary info =====================================================
XFAIL test_f.py::test_f

=================================================== 1 xfailed in 0.09 seconds ====================================================
$ cp test_f.py ..
$ cd ..
$ pytest test_f.py 
====================================================== test session starts =======================================================
platform darwin -- Python 3.7.1, pytest-4.1.2.dev19+g57bf9d67, py-1.7.0, pluggy-0.8.0
rootdir: /Users/enojb/current/sympy, inifile:
plugins: xdist-1.26.0, forked-0.2, doctestplus-0.3.0.dev0
collected 1 item                                                                                                                 

test_f.py x                                                                                                                [100%]

======================================================== warnings summary ========================================================
test_f.py::test_f
  /Users/enojb/current/sympy/test_f.py:5: PytestWarning: asserting the value None, please use "assert is None"
    assert g()

-- Docs: https://docs.pytest.org/en/latest/warnings.html
============================================= 1 xfailed, 1 warnings in 0.14 seconds ==============================================
$ cat test_f.py 
import pytest

@pytest.mark.xfail
def test_f():
    assert g()

def g():
    # Shouldn't return None but doesn't work properly...
    return None

[asottile]

that output looks correct to me, are you expecting xfail to silence the warning?

The reason it "appears" to silence the warning when run from pytest is we configure warnings-as-errors: here -- in reality it's upgrading the warning to an exception (which is gobbled by xfail)

[oscarbenjamin]

Okay I've reread everything above and I understand now.

The actual code I'm testing uses py.test.mark.xfail anyway. I can change that to pytest.mark.xfail but either way I see this problem.

After initially reading the comments above I was hoping that changing to xfail would remove these warnings because I don't think it makes sense to warn about this in a test that is already marked as xfail. I'll probably just fix this in Sympy though by changing the tests.

[asottile]

maybe the feature request is "xfail silences warnings" -- I think that could be considered but there's probably a lot of cases where you want the warnings even when you expect it to fail? I haven't put much thought to it.

[oscarbenjamin]

I think xfail silences warnings could be a feature request but that's not what I would propose.

I've read #3191 and #4146 and I don't understand the rationale for this change. A user who is confused and writes assert mock.assert... will immediately see a test failure because they are asserting None. I can see that it might be worthwhile to improve the test report for assert None when a test fails.

Adding this as a warning means that it will also show up for xfail tests (which is pointless).

[davidlowryduda]

I believe that this is resolved in sympy #15817 and that this issue can now be closed. I therefore suggest that this issue be closed.

[oscarbenjamin]

I would say that the issue is worked around in sympy. I still think this should be fixed in pytest.

[davidlowryduda]

Yes, now that you mention it, I agree. Thanks for your quick response.