-q no longer impacts --collect-only (when used with `--log-cli-level`)

[adamtheturtle]

I have an application which parses pytest test collection output.

Until pytest 4.3.1 I could get easily parseable output with --collect-only -q.

~> pytest --collect-only -q tests/test_dcos_e2e/backends/docker/test_distributions.py::TestCoreOS::test_oss
tests/test_dcos_e2e/backends/docker/test_distributions.py::TestCoreOS::test_oss

no tests ran in 0.02 seconds

From @RonnyPfannschmidt here we see:

thats supported with -q for just test ids and -qq for just test filenames

However, with pytest>=4.4.0 this does not work - we see the full output even with -q:

~> pytest --collect-only -qq tests/test_dcos_e2e/backends/docker/test_distributions.py::TestCoreOS::test_oss
============================================================================================= test session starts ==============================================================================================
platform darwin -- Python 3.7.3, pytest-4.4.0, py-1.8.0, pluggy-0.12.0 -- /Users/adam/.virtualenvs/e2e/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/adam/Documents/repositories/work/mesosphere/dcos-e2e, inifile: setup.cfg
plugins: timeout-1.3.3, cov-2.7.1, requests-mock-1.6.0
collected 1 item                                                                                                                                                                                               
<Package /Users/adam/Documents/repositories/work/mesosphere/dcos-e2e/tests/test_dcos_e2e/backends/docker>
  <Module test_distributions.py>
    Tests for the ``linux_distribution`` option on the Docker backend.
    <Class TestCoreOS>
      Tests for the CoreOS distribution option.
        <Function test_oss>
          DC/OS OSS can start up on CoreOS.

========================================================================================= no tests ran in 0.03 seconds =========================================================================================

[asottile]

I'm not able to reproduce, could you boil this down to a minimal reproduction?

$ pytest t.py::Test  --collect-only -qq
t.py: 1


$ pytest t.py::Test  --collect-only -q
t.py::Test::test

no tests ran in 0.00 seconds
$ pytest --version
This is pytest version 4.6.2, imported from /tmp/venv/lib/python3.6/site-packages/pytest.py

[adamtheturtle]

Sorry about that @asottile - I found that I had a tool:pytest section in my setup.cfg. I reproduced this with a new project and a minimal file - using the --log-cli-level option, set to any value:

--log-cli-level=ERROR

~/D/pytest-experiment (pytest-experiment) ~> ls
test_experiment.py
~/D/pytest-experiment (pytest-experiment) ~> cat test_experiment.py
class TestExample:
    def test_foo():
        pass
~/D/pytest-experiment (pytest-experiment) ~>
pytest --collect-only -q test_experiment.py  --log-cli-level=ERROR
========================================= test session starts ==========================================
platform darwin -- Python 3.7.3, pytest-4.6.2, py-1.8.0, pluggy-0.12.0 -- /Users/adam/.virtualenvs/pytes
t-experiment/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/adam/Desktop/pytest-experiment
plugins: timeout-1.3.3, cov-2.7.1, requests-mock-1.6.0
collected 1 item
<Module test_experiment.py>
  <Class TestExample>
      <Function test_foo>

===================================== no tests ran in 0.02 seconds =====================================
~/D/pytest-experiment (pytest-experiment) ~>

[adamtheturtle]

I have updated the title as appropriate.

[RonnyPfannschmidt]

pytest/src/_pytest/logging.py

Lines 413 to 414 in 5976f36

	if self._log_cli_enabled() and config.getoption("verbose") < 1:
	config.option.verbose = 1

[blueyed]

Changed in 23146e7#diff-a36360edbbdcadcea4fbcbde217c2333R392.

[nicoddemus]

Thanks @adamtheturtle,

cc @Thisch

[blueyed]

@nicoddemus
You've added this initially in 8f6a592 - but previously -q would turn verbose into -1, and it was skipped.
I am not sure we should auto-enable verbose in the beginning - what was the reason for it?

[nicoddemus]

@blueyed thanks for digging this up; this was introduced in #3124 as part of the discussion in #3013.

I don't remember the specifics though. 😬

[nicoddemus]

Oh the reason for that is written in the PR description:

Live logging automatically sets the output to verbose so the logs don't get interleaved with the test progress output;

This is what we thought was a good tradeoff at the time.

[twmr]

Since we only need this special verbose=1 code to fix the test progress output, we could also move it to a hook that is run after the collection hook. I'm currently working on a fix.

[adamtheturtle]

Folks, I just want to say the above is really great to see - only an hour ago I narrowed this down and already there are people on it and an active discussion.

Having found the issue, I have a workaround for me - add -c '<empty file path>' to my command.