metafunc parametrization with session scope causes multiple fixture teardowns

[bluetech]

Originally posted by @Idanjc in #8089

Hi,

I'm not sure if this is something I'm doing wrong, a bug or known behaviour in recent versions of pytest, so trying to post here first.
Apologies in advanced if it's not the right place for it.

When I'm using metafunc.parametrize on session scoped fixtures it causes a teardown after each test function, instead of after the entire session.
This causes a heavy setup I have that should only run once per configuration (could have several configs in a single run) to run after every test function.

It doesn't happen in pytest 5.3.5, but does happen in pytest 5.4+ versions (reproduced on 5.4.3 & 6.1.2).

Reproduction code example:

# conftest.py

def pytest_addoption(parser):
    parser.addoption('--config', action='store', type=str,
                     help='load configuration for tests from some file (yaml/json etc)')

@pytest.fixture(scope='session')
def my_fixture(setup):
    # would yield a specific test object , e.g. db
    yield 'my_fixture'

@pytest.fixture(scope='session')
def setup(config):
    # do some setup to use in tests
    # actually yield an orchestration holding all the needed objects in tests after setup (server, db etc)
    yield 

@pytest.fixture(scope='session', autouse=True)
def config(request):
    conf = request.param
    # build final config
    yield conf

def pytest_generate_tests(metafunc):
    if metafunc.config.getoption('--config'):
        # load configurations from some file
        confs = [{'key1': 'val1'}, {'key2': 'val2'}]
        confs_ids = ['conf1', 'conf2']
        metafunc.parametrize('config', confs, ids=confs_ids, indirect=True)

# test_param.py

def test_foo(my_fixture):
    print('\n test_foo \n')


def test_bar(my_fixture):
    print('\n test_bar \n')

Running with: pytest -s --setup-show --config 'path/to/file'

Output from pytest version 5.3.5:

test_param.py 
SETUP    S config[{'key1': 'val1'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_foo[conf1] (fixtures used: config, my_fixture, request, setup)
 test_foo 

.
        test_param.py::test_bar[conf1] (fixtures used: config, my_fixture, request, setup)
 test_bar 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key1': 'val1'}]
SETUP    S config[{'key2': 'val2'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_foo[conf2] (fixtures used: config, my_fixture, request, setup)
 test_foo 

.
        test_param.py::test_bar[conf2] (fixtures used: config, my_fixture, request, setup)
 test_bar 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key2': 'val2'}]

Output from pytest versions 5.4.3 & 6.1.2:

test_param.py 
SETUP    S config[{'key1': 'val1'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_foo[conf1] (fixtures used: config, my_fixture, request, setup)
 test_foo 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key1': 'val1'}]
SETUP    S config[{'key1': 'val1'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_bar[conf1] (fixtures used: config, my_fixture, request, setup)
 test_bar 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key1': 'val1'}]
SETUP    S config[{'key2': 'val2'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_foo[conf2] (fixtures used: config, my_fixture, request, setup)
 test_foo 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key2': 'val2'}]
SETUP    S config[{'key2': 'val2'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_bar[conf2] (fixtures used: config, my_fixture, request, setup)
 test_bar 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key2': 'val2'}]

Thanks, appreciate the help,
Idanjc

[bluetech]

Originally posted by @Idanjc in #8089 (comment)

Hi,

Following up, I now noticed this seems to happen only when passing mutable objects to metafunc paramertize.

If we take my example above and change in pytest_generate_tests confs to:
confs = [('key1', 'val1'), ('key2', 'val2')]

Then it runs properly (same output as 5.3.5 version), but passing mutable object would cause the issue I mentioned.
Originally I passed a dict (real configuration is also a dict), but changing the above working example from tuple to list would also cause the issue, due to list being mutable:
confs = [['key1', 'val1'], ['key2', 'val2']]

Do you think it's a bug in newer pytest versions (since it worked before), or is it the expected behaviour for mutable object in paramertization and I should change how we load the configuration?

Thanks,
Idanjc

[RonnyPfannschmidt]

i believe the issue is that parameterize will not trigger a exception when the scope of the parameterize differs from the scope of the fixture, instead all the things get confused

[bluetech]

Agree with @RonnyPfannschmidt that the parametrization here is quite unusual. Remember that pytest_generate_tests is called for each test function, but here it is used to parametrize the config fixture which is session-scoped, with the expectation that equal param values would result in the same parametrization instance. I'm not sure that's perfectly legit but by some miracle it does mostly work...

In any case, the actual problem here and the mutable/immutable difference happens because pytest uses is to decide whether it needs to re-execute the fixture or not, as an optimization:

pytest/src/_pytest/fixtures.py

Lines 1056 to 1068 in 810b878

	# note: comparison with `==` can fail (or be expensive) for e.g.
	# numpy arrays (#6497).
	cache_key = self.cached_result[1]
	if my_cache_key is cache_key:
	if self.cached_result[2] is not None:
	_, val, tb = self.cached_result[2]
	raise val.with_traceback(tb)
	else:
	result = self.cached_result[0]
	return result
	# We have a previous but differently parametrized fixture instance
	# so we need to tear it down before creating a new one.
	self.finish(request)

[BTW, equal tuples are not guaranteed to have equal identities, I guess it works just due to some CPython implementation detail, but can easily break as well.]

So if you want to keep it working you'd need to guarantee that the object IDs remain the same between pytest_generate_tests invocations. You can do this by caching the parsing of the config file for example or whatever way else you'd like.

[Idanjc]

I see, so it worked before purely by mistake under a false assumption.
I've tried caching in the small reproduction example and it worked very well, always running a single setup/teardown for session fixtures.
I'll run this fix on the actual configuration code and check it works on it as well, I have my hopes up it will.

Thanks a lot :)

[RonnyPfannschmidt]

Note on tuple id equality, afair static tuples within the same code get the same identity due to compiler optimisation and constant storage

[dancavallaro]

Agree with @RonnyPfannschmidt that the parametrization here is quite unusual. Remember that pytest_generate_tests is called for each test function, but here it is used to parametrize the config fixture which is session-scoped, with the expectation that equal param values would result in the same parametrization instance. I'm not sure that's perfectly legit but by some miracle it does mostly work...

@bluetech Would you mind clarifying a bit more why you think this parametrization is unusual? Or I suppose what I really mean to ask is, is there a better way to achieve this in Pytest?

I have a similar use case where I would like to dynamically parameterize (at runtime, based on configuration and CLI arguments) a session-scoped fixture. To oversimplify, my fixture encapsulates a bunch of very expensive initialization, so I want to guarantee that I only create each fixture once and use it for all relevant tests. I have this mostly working now using pytest_generate_tests and metafunc.parametrize, but this cache key identity comparison is the last issue I need to work around.

I do agree that using pytest_generate_tests to parametrize a fixture feels a bit awkward (and it was certainly very difficult to even identify this approach in the first place), but is there another way to dynamically parametrize a fixture at runtime?

Edit: Sorry, I think I misinterpreted you here. I don't think you're saying that using pytest_generate_tests to parametrize a fixture is weird, just that using it to parametrize a session-scoped fixture, and expecting the caching to work, is weird. I guess I agree with that, and I'd certainly prefer a more clear/direct way of achieving this, but I haven't found one.