calling pytest.Metafunc.parametrize multiple times

[calestyo]

Hey.

I think this is at least a documentation ambiguity. ;-)

pytest/src/_pytest/python.py

Lines 1242 to 1249 in cc0adf6

	Can be called multiple times, in which case each call parametrizes all
	previous parametrizations, e.g.
	::
	unparametrized: t
	parametrize ["x", "y"]: t[x], t[y]
	parametrize [1, 2]: t[x-1], t[x-2], t[y-1], t[y-2]

AFAIU however, this can be only called multiple times on different parameters.

Otherwise I get a:

ERROR test_foo.py - ValueError: duplicate 'cal'
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

I stumbled over this while trying what what I've asked for on discord... maybe the behaviour there is another bug, but maybe some pytest expert can look at it and confirm whether it's fishy... I'm still basically a complete pytest beginner... so I don't wanna file issues for everything that seems just strange to me ;-)

[RonnyPfannschmidt]

this is expected behaviour, but the user facing error could be better, thanks for pointing it out

[RonnyPfannschmidt]

to elaborate a little, if you already did parameterize for a parameter, its a error to add another parameter for it, similar to how one cannot use a keyword name multiple times

[calestyo]

this is expected behaviour, but the user facing error could be better, thanks for pointing it out

Uhm, well I though mostly that sentence of the documentation would need some updating. For that I could do a PR.

to elaborate a little, if you already did parameterize for a parameter, its a error to add another parameter for it, similar to how one cannot use a keyword name multiple times

Okay that is in principle reasonable, but what I don't get is:

AFAIU, one can overwrite fixtures with/without parametrisation, which I think is really nice, as it's e.g. quite likely that one may want to share common names like pathname or so but have them being actually different fixtures.

My understanding was, that this overwriting can be done even within a module (e.g. the one within a class overwrites the one at module level - and not just overwriting those from conftest.py).

Good,... but this doesn't seem to work as nice with pytest_generate_tests.

Consider:

conftest.py:

import pytest
import sys

def pytest_generate_tests(metafunc):
    f = open("/tmp/caldebug", "a")
    print("CONFTEST.PY: " + str(metafunc.fixturenames), file=f)
    f.close()
    if "cal" in metafunc.fixturenames:
        metafunc.parametrize("cal", [0, 1, 2])

test_foo.py:

import pytest
import sys

@pytest.fixture(scope="module")
def myfix():
    return ValueError("MYFIX")

def test_foo(myfix, cal1):
    print(id(myfix))
    print(cal1)
    assert 0

def test_bar(myfix):
    print(id(myfix))
    assert 0

def pytest_generate_tests(metafunc):
    f = open("/tmp/caldebug", "a")
    print("FOO: " + str(metafunc.fixturenames), file=f)
    f.close()
    if "cal1" in metafunc.fixturenames:
        metafunc.parametrize("cal1", ["a", "b", "c"])

test_baz.py:

import pytest
import sys

@pytest.fixture(scope="module")
def myfix():
    return ValueError("MYFIX2")

def test_baz(myfix, cal):
    print(id(myfix))
    print(cal)
    assert 0

Now running it:

$ pytest
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/calestyo/z/x
collected 7 items                                                                                                                                                                            

test_baz.py FFF                                                                                                                                                                        [ 42%]
test_foo.py FFFF                                                                                                                                                                       [100%]

========================================================================================== FAILURES ==========================================================================================
________________________________________________________________________________________ test_baz[0] _________________________________________________________________________________________

myfix = ValueError('MYFIX2'), cal = 0

    def test_baz(myfix, cal):
        print(id(myfix))
        print(cal)
>       assert 0
E       assert 0

test_baz.py:11: AssertionError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
140269563284544
0
________________________________________________________________________________________ test_baz[1] _________________________________________________________________________________________

myfix = ValueError('MYFIX2'), cal = 1

    def test_baz(myfix, cal):
        print(id(myfix))
        print(cal)
>       assert 0
E       assert 0

test_baz.py:11: AssertionError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
140269563284544
1
________________________________________________________________________________________ test_baz[2] _________________________________________________________________________________________

myfix = ValueError('MYFIX2'), cal = 2

    def test_baz(myfix, cal):
        print(id(myfix))
        print(cal)
>       assert 0
E       assert 0

test_baz.py:11: AssertionError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
140269563284544
2
________________________________________________________________________________________ test_foo[a] _________________________________________________________________________________________

myfix = ValueError('MYFIX'), cal1 = 'a'

    def test_foo(myfix, cal1):
        print(id(myfix))
        print(cal1)
>       assert 0
E       assert 0

test_foo.py:11: AssertionError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
140269563293280
a
________________________________________________________________________________________ test_foo[b] _________________________________________________________________________________________

myfix = ValueError('MYFIX'), cal1 = 'b'

    def test_foo(myfix, cal1):
        print(id(myfix))
        print(cal1)
>       assert 0
E       assert 0

test_foo.py:11: AssertionError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
140269563293280
b
________________________________________________________________________________________ test_foo[c] _________________________________________________________________________________________

myfix = ValueError('MYFIX'), cal1 = 'c'

    def test_foo(myfix, cal1):
        print(id(myfix))
        print(cal1)
>       assert 0
E       assert 0

test_foo.py:11: AssertionError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
140269563293280
c
__________________________________________________________________________________________ test_bar __________________________________________________________________________________________

myfix = ValueError('MYFIX')

    def test_bar(myfix):
        print(id(myfix))
>       assert 0
E       assert 0

test_foo.py:15: AssertionError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
140269563293280
================================================================================== short test summary info ===================================================================================
FAILED test_baz.py::test_baz[0] - assert 0
FAILED test_baz.py::test_baz[1] - assert 0
FAILED test_baz.py::test_baz[2] - assert 0
FAILED test_foo.py::test_foo[a] - assert 0
FAILED test_foo.py::test_foo[b] - assert 0
FAILED test_foo.py::test_foo[c] - assert 0
FAILED test_foo.py::test_bar - assert 0
===================================================================================== 7 failed in 0.04s ======================================================================================

works, and one sees:

• test_baz.py which doesn't "overwrite" pytest_generate_tests gets only the one from conftest.py called.
• but test_foo.py gets both called

Now I can imagine that the latter is done to get parametrisation from the "higher" level (i.e. conftest.py) which is not overwritten in test_foo.py.

But the prolem is, if in test_foo.py I s/cal1/cal/g, i.e. let it use the same name as the param that's already parametrised in the higher level conftest.py, one get's the duplication error.

Wouldn't it be better if, in that case, the parametrisations from test_foo.py clear&override those from conftest.py - but only those that actually are re-defined?

Also, it would be nice if I could, analogously to overriding fixture functions, have multiple pytest_generate_tests per module. I think about being able to put it n class definitions (maybe even at multiple levels, with nested classes).

The benefits would be IMO better readability (keeping the tests in their class close with the code the parametrises them) and also allowing more fine grained overwriting

What do you think? :-)

[calestyo]

Oh and the output from the debug file above would be:

CONFTEST.PY: ['myfix', 'cal']
FOO: ['myfix', 'cal1']
CONFTEST.PY: ['myfix', 'cal1']
FOO: ['myfix']
CONFTEST.PY: ['myfix']

[RonnyPfannschmidt]

This one plays into hooks

For generate tests all applying implementations are called

[calestyo]

Uhm? What does that mean? That the idea of allowing overwriting via pytest_generate_tests couldn't be done?

[RonnyPfannschmidt]

As of now its impossible, and enabling it is full of pitfalls

[calestyo]

Would it be worth to add a feature request for it in order to track ideas?

[RonnyPfannschmidt]

For now id avoid a tracking issue as it's likely it would Just rot

If we get someone actively working on it that may change

[calestyo]

How easy do you think would such implementation be? I mean it kinda already works for fixture functions? In that setmulti() function that notices the duplicate... couldn't that "simply" overwrite the previous parametrisations if they were made by a higher level (or is that information not available)?

[RonnyPfannschmidt]

It's not safely available as the Tracking for single parameter vs multi parameter is not done

For safe replace it's not safely possible in case of a subset

Eg Parametrize with foo and bar at once and then trying to only replace bar

As far as I'm concerned we shouldn't allow that type of replacement as it's very error prone to do in retrospect

Even more so if we introduce covariant parametrization

[nicoddemus]

Fixed by #11291

[calestyo]

But it is supported, that one calls pytest_generate_tests (like once from a test module and once from conftest.py), or isn't it?

Documentation on that is unfortunately pretty sparse. ;-)

Would the other idea I had above be feasible, i.e. that one could define pytest_generate_tests also in classes, and thus allow the definitions of the tests (if made in classes) be more close to the parametrisation?