How to disable plugin based on environment?

[jaraco]

In python/typed_ast#144, I confirmed that typed_ast, and by extension pytest-black and pytest-mypy, are not supported on PyPy (and cause crashes on installation). I'd like to fashion a technique that reflects this limitation and bypasses the installation and enabling of these plugins when testing on PyPy.

I've found I can selectively include or exclude the plugin modules by adding environment markers to the requirement:

	pytest-black >= 0.3.7; platform_python_implementation != "PyPy"
	pytest-cov
	pytest-mypy; platform_python_implementation != "PyPy"

However, these plugins require a two-step installation process:

1. Install the dependency.
2. Enable the plugin with --black or --mypy during pytest invocation.

Leaving the --<plugin> in pytest.ini (example) while deselecting the plugin for installation results in an InvocationError:

keyring master $ tox -r -e pypy3
pypy3 recreate: /Users/jaraco/code/main/keyring/.tox/pypy3
pypy3 develop-inst: /Users/jaraco/code/main/keyring
pypy3 installed: attrs==20.1.0,cffi==1.14.0,coverage==5.2.1,docutils==0.16,flake8==3.8.3,greenlet==0.4.13,importlib-metadata==1.7.0,iniconfig==1.0.1,-e git+gh://jaraco/keyring@a6980e4b0a0c25143ab027e437a1b5b81f9c0bf5#egg=keyring,mccabe==0.6.1,more-itertools==8.4.0,packaging==20.4,pluggy==0.13.1,py==1.9.0,pycodestyle==2.6.0,pyflakes==2.2.0,pyparsing==2.4.7,pytest==6.0.1,pytest-checkdocs==2.1.1,pytest-cov==2.10.1,pytest-flake8==1.0.6,readline==6.2.4.1,six==1.15.0,toml==0.10.1,zipp==3.1.0
pypy3 run-test-pre: PYTHONHASHSEED='4203199892'
pypy3 run-test: commands[0] | pytest
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --black --mypy
  inifile: /Users/jaraco/code/main/keyring/pytest.ini
  rootdir: /Users/jaraco/code/main/keyring

ERROR: InvocationError for command /Users/jaraco/code/main/keyring/.tox/pypy3/bin/pytest (exited with code 4)
_____________________________________________________________ summary ______________________________________________________________
ERROR:   pypy3: commands failed

Best I can tell, there's no easy way to programmatically disable the plugin (or only enable it if present).

In pytest-dev/pytest-cov#418, I encountered a similar problem where it was suitable to install coverage but its presence needed to be disabled at runtime. The workaround for that approach was specifically tuned to implementation details of that plugin.

What I'd really like is a uniform way for plugins to declare there presence, for them to be enabled by default, and layers of configuration (global, system, user, environment, invocation) where the enabling of that feature can be toggled on or off based on certain factors (maybe environment markers or maybe arbitrary Python logic).

But thinking short-term, are there any options available to selectively disable the enabling of the black and mypy plugins, such that they're enabled by default, but disabled when the environment is built on PyPy?

[bluetech]

If I understand correctly, you want to load the plugins, so that pytest accepts the flags, but you don't want them to actually run. This is not a normal scenario so there isn't any particular support for it. I think the best solution would be to just have the plugins behave appropriately (i.e. not crash) on PyPy.

However if this is really not possible, some workarounds I can think of are:

• unregister the plugins at runtime, after cmd line parsing but before they do anything. Not sure how well that works.

• Add a local plugin which parses and ignores the --black and --mypy flags, just so things work when the real plugins are not loaded.

[jaraco]

you want to load the plugins, so that pytest accepts the flags

Actually, that doesn't help in this particular situation, because the plugins can't even be installed.

What I imagine is a framework that offers one or more of these features:

• installed (present) plugins are enabled by default, so that only one action is required to add a plugin (declare/install the plugin) or disable it (uninstall the plugin).
• plugins can be enabled or disabled through environment variables, config files, and command-line overrides.
• unrecognized command line options will raise warnings but proceed, or can be configured to exist (e.g. pytest.allow_missing_params=--black,--mypy).

Actually, just the first feature would be sufficient, but I suspect that without the second feature, many users would find the enabled-by-default functionality annoying or even problemmatic.

Features one and two could be achieved by decoupling a plugin's config from the pytest API. If each plugin was responsible to load its own config (or had its own section in config), the absence of a plugin would be unaffected by the presence of config for that plugin. Pytest could provide a generic interface, like --enable-plugin=black or --disable-plugin=mypy that would allow overriding of any plugin by name at the command-line.

The fact that almost all configuration is buried in a single setting "addopts" makes it non-obvious what the behavior is when the value is set in the environment and the config and the command-line and puts the burden on plugins to support settings that override settings at a broader scope.

The first workaround won't work in this case because the plugins aren't installed. The second option doesn't exactly work because it needs to work across every project that might be affected (any project that may want to use pytest-black or pytest-mypy on pypy). I'll try the second option and create a generic plugin that does nothing but implement support for select command-line options.

[jaraco]

I've released jaraco.test 3.0, which has a pytest plugin that allows defining meaningless options in pyproject.toml. For example, the following toml will work to expect --black and --mypy parameters when those plugins aren't present:

[tool.jaraco.pytest.opts.--black]
action = "store_true"

[tool.jaraco.pytest.opts.--mypy]
action = "store_true"

[nicoddemus]

Hi @jaraco,

Actually, just the first feature would be sufficient, but I suspect that without the second feature, many users would find the enabled-by-default functionality annoying or even problemmatic.

This is actually the plugin's implementation choice, that they require an extra flag to be active. They could also optionally be enabled via config, which does not produce an error for unknown configs.

About disabling the plugins, that's possible through all the different ways you listed:

• environment variables: PYTEST_ADDOPTS, with -p no:<plugin>.
• config files: using addopts: -p no:<plugin>
• command-line overrides: again -p to the rescue.

Features one and two could be achieved by decoupling a plugin's config from the pytest API. If each plugin was responsible to load its own config (or had its own section in config), the absence of a plugin would be unaffected by the presence of config for that plugin. Pytest could provide a generic interface, like --enable-plugin=black or --disable-plugin=mypy that would allow overriding of any plugin by name at the command-line.

Not sure, there's nothing that prevents a plugin to load its configuration from some other place, it's their choice. Of course, almost all plugins use the pytest config API as that's what plugins are supposed to do. Given that this is not a normal situation, I don't think it makes much sense for pytest to implement an alternative interface.

The fact that almost all configuration is buried in a single setting "addopts" makes it non-obvious what the behavior is when the value is set in the environment and the config and the command-line and puts the burden on plugins to support settings that override settings at a broader scope.

Not sure I follow, the precedence of each is well defined: command-line first, config second, env vars last.

I'll try the second option and create a generic plugin that does nothing but implement support for select command-line options.

Yeah I think this is the way to go. While I can see we adding a switch to pytest to transform unknown options into warnings instead of errors, I think this is not common enough to add a new switch to the already large list of configuration options we have.

I've released jaraco.test 3.0, which has a pytest plugin that allows defining meaningless options in pyproject.toml.

Nice! This allows a escape hatch for others in the same situation as you (that I believe are a bit rare however).

[graingert]

I've released jaraco.test 3.0, which has a pytest plugin that allows defining meaningless options in pyproject.toml.

I made pytest-forwards-compatible for addopts options, but it's currently only useful for .ini/.cfg projects

[jaraco]

The technique in jaraco.test doesn't work (jaraco/jaraco.test#1), so I'm back to the drawing board.

I'll check out pytest-forwards-compatible.

[jaraco]

Not sure, there's nothing that prevents a plugin to load its configuration from some other place, it's their choice. Of course, almost all plugins use the pytest config API as that's what plugins are supposed to do.

Right, so plugins are advised to follow an approach that runs afoul of this issue. Since pytest advises plugins to solicit their config from command-line parameters, it becomes difficult to selectively provide that config. The "addopts" technique also dosen't allow overriding a value at a closer scope that was defined at a broader scope. That is, you can't define "PYTEST_ADDOPTS=--black" at the environment level then "pytest_addopts=--no-black" at the config level then "--black" at the command-line. More importantly, the absence of the plugin causes the whole test suite to break due to any of these settings.

I think my next approach is going to be to provide a translator. To solicit settings for various plugins in separate sections of a config file and to only enable those settings when the plugin is available.

I apologize for being unclear about the challenges I'm facing. I've long found the "addopts" approach to be clumsy but it's only now that its clumsiness is really affecting my ability to use pytest. I'm conflating several separate concerns:

• need for configuration to be decoupled from plugin presence
• desire for plugin presence to imply plugin functionality
• ability to disable a plugin based on environment
• ability to configure plugins (with overrides) at various scopes

[nicoddemus]

Since pytest advises plugins to solicit their config from command-line parameters, it becomes difficult to selectively provide that config.

Would be possible for pytest-black and pytest-mypy to add configuration options (in pytest.ini)? Then you can drop --black and --mypy from the command-line, and add them to your pytest.ini.

EDIT: hmm but I see now that probably doesn't apply to pytest-black and pytest-mypy, as they are more like linters that you want to run in separate pytest invocations, correct?

I apologize for being unclear about the challenges I'm facing. I've long found the "addopts" approach to be clumsy but it's only now that its clumsiness is really affecting my ability to use pytest

No worries, I agree addopts is a bit clumsy sometimes.

need for configuration to be decoupled from plugin presence

You mean command-line arguments here, correct? To be clear, ini options today don't emit an error when an unknown option is defined in pytest.ini (which arguably is a problem). You can for example add a key named black=1 in pytest.ini and pytest will only show a warning (unless --strict-config is given, in which case this becomes an error).

desire for plugin presence to imply plugin functionality

This is really up to each plugin to implement as they see fit, and not something pytest can really enforce, I think. For example, installing pytest-sugar will enable it right away, without the need for a configuration flag.

ability to disable a plugin based on environment
ability to configure plugins (with overrides) at various scopes

That's interesting, but we probably would need a formal proposal to be able to discuss all the details. We should also continue to try to figure out if this can be implemented as a separate plugin, instead of putting directly into the core.

[jaraco]

they are more like linters that you want to run in separate pytest invocations, correct?

I prefer to think of linters as just another class of test that are run as a matter of course when testing the code, enabled by default and disabled selectively if needed (such as through -k "not black" or --plugin="no:black").

You mean command-line arguments here, correct?

I mean configuration generally, but since the recommendation is for plugin authors to solicit configuration from the command line and that's what plugin authors are doing, that's where it's problematic.

[jaraco]

In jaraco.test 3.1.1, I've created a plugin that appears to be working and I suspect isn't subject to the race conditions because it uses the pytest_load_initial_conftests hook. The implementation ultimately was pretty straightforward.

This approach has the added advantage that supplying -p no:black now works instead of causing the invocation to fail with "no option --black". I plan to utilize this for --cov and --flake8 also. I'm pretty happy with this behavior. I'll test it out for some time, but do feel like a similar design would be good for pytest to adopt as a first-class feature and recommended usage.