Issue with execution order of tests

[awaelchli]

🐛 Bug

We currently have side-effects in our tests. The order in which we execute the test suite matters.

To Reproduce

See PR here, where I used pytest-random-order to trigger a different run order.

Expected behavior

Any random order of test execution passes the same way.

Environment

Currently observed in our conda-ci, but likely an issue in all environments.

Additional context

Also has been a blocker for #11377 for quite some time

cc @tchaton @rohitgr7 @akihironitta @carmocca @Borda @kaushikb11 @ninginthecloud

[carmocca]

@akihironitta can you take a look at this?

[awaelchli]

We haven't seen these issues anymore.