Less coverage reporting

[blueyed]

Via #4829 (comment).

[blueyed]

In the long run we should only have around 10 uploads / jobs reporting coverage to cover everything.
This also improves CI run times in general, of course.

[blueyed]

Will push the following next (but only after it has reported with the current state).

@nicoddemus
Would be good if we could control it with a single env var for Azure also.
I would then also like to enable it there explicitly.

diff --git a/.travis.yml b/.travis.yml
index 17087bf9..9616a081 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -12,15 +12,15 @@ install:
   - python -m pip install --upgrade --pre tox
 env:
   matrix:
-    - TOXENV=py27
+    - TOXENV=py27 PYTEST_COVERAGE=1
     # Specialized factors for py27.
-    - TOXENV=py27-nobyte
-    - TOXENV=py27-xdist
-    - TOXENV=py27-pluggymaster-xdist PYTEST_NO_COVERAGE=1
+    - TOXENV=py27-nobyte PYTEST_COVERAGE=1
+    - TOXENV=py27-xdist PYTEST_COVERAGE=1
+    - TOXENV=py27-pluggymaster-xdist
     # Specialized factors for py37.
     - TOXENV=py37-pexpect,py37-trial,py37-numpy
-    - TOXENV=py37-pluggymaster-xdist PYTEST_NO_COVERAGE=1
-    - TOXENV=py37-freeze PYTEST_NO_COVERAGE=1
+    - TOXENV=py37-pluggymaster-xdist
+    - TOXENV=py37-freeze
 
 matrix:
   allow_failures:
@@ -30,9 +30,9 @@ matrix:
 jobs:
   include:
     # Coverage tracking is slow with pypy, skip it.
-    - env: TOXENV=pypy-xdist PYTEST_NO_COVERAGE=1
+    - env: TOXENV=pypy-xdist
       python: 'pypy2.7-6.0'
-    - env: TOXENV=pypy3-xdist PYTEST_NO_COVERAGE=1
+    - env: TOXENV=pypy3-xdist
       python: 'pypy3.5-6.0'
 
     - env: TOXENV=py34-xdist
@@ -60,7 +60,7 @@ jobs:
         - test $(python -c 'import sys; print("%d%d" % sys.version_info[0:2])') = 37
 
     # Jobs only run via Travis cron jobs (currently daily).
-    - env: TOXENV=py38-xdist PYTEST_NO_COVERAGE=1
+    - env: TOXENV=py38-xdist
       python: '3.8-dev'
       if: type = cron
 
@@ -72,7 +72,6 @@ jobs:
 
     - stage: deploy
       python: '3.6'
-      env: PYTEST_NO_COVERAGE=1
       install: pip install -U setuptools setuptools_scm
       script: skip
       deploy:
@@ -88,7 +87,7 @@ jobs:
 
 before_script:
   - |
-    if [[ "$PYTEST_NO_COVERAGE" != 1 ]]; then
+    if [[ "$PYTEST_COVERAGE" = 1 ]]; then
       export COVERAGE_FILE="$PWD/.coverage"
       export COVERAGE_PROCESS_START="$PWD/.coveragerc"
       export _PYTEST_TOX_COVERAGE_RUN="coverage run -m"
@@ -99,7 +98,7 @@ script: tox --recreate
 
 after_success:
   - |
-    if [[ "$PYTEST_NO_COVERAGE" != 1 ]]; then
+    if [[ "$PYTEST_COVERAGE" = 1 ]]; then
       set -e
       # Add last TOXENV to $PATH.
       PATH="$PWD/.tox/${TOXENV##*,}/bin:$PATH"

[blueyed]

https://pytest-dev.visualstudio.com/cabd24c8-a88c-4306-b96f-5a60a0c12f48/_build/results?buildId=148 is a HTTP 500 currently - everything is broken.. :D

[blueyed]

Currently there are 36 uploads/builds: https://codecov.io/gh/pytest-dev/pytest/commit/ee62674322206299e0caaa36c172d9bb63feeaaf/build (some are bigger (in the sense of covering more code) than others of course).

[codecov]

Codecov Report

Merging #4839 into master will decrease coverage by 1.12%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4839      +/-   ##
==========================================
- Coverage   95.68%   94.55%   -1.13%     
==========================================
  Files         113      113              
  Lines       25163    25163              
  Branches     2498     2498              
==========================================
- Hits        24076    23794     -282     
- Misses        771     1046     +275     
- Partials      316      323       +7

Flag	Coverage Δ
#linux	91.91% <ø> (-3.56%)	⬇️
#osx	?
#windows	93.7% <ø> (-0.03%)	⬇️

Impacted Files	Coverage Δ
testing/test_pdb.py	48.14% <0%> (-50.93%)	⬇️
src/_pytest/debugging.py	60.37% <0%> (-17.62%)	⬇️
src/_pytest/unittest.py	87.3% <0%> (-6.88%)	⬇️
src/_pytest/doctest.py	94.2% <0%> (-2.18%)	⬇️
src/_pytest/cacheprovider.py	95.75% <0%> (-1.42%)	⬇️
src/_pytest/pytester.py	86.34% <0%> (-0.72%)	⬇️
testing/test_terminal.py	99.24% <0%> (-0.57%)	⬇️
src/_pytest/capture.py	93.66% <0%> (-0.46%)	⬇️
testing/test_assertrewrite.py	83.25% <0%> (-0.31%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee62674...e3eb26f. Read the comment docs.

[blueyed]

With this PR currently at 26: https://codecov.io/gh/pytest-dev/pytest/commit/86070f0b7dfa0308c818c9a23051f6a9d33dc19c/build
https://codecov.io/gh/pytest-dev/pytest/pull/4839 is timing out.
Also codecov/project: https://codecov.io/gh/pytest-dev/pytest/compare/ee62674322206299e0caaa36c172d9bb63feeaaf...86070f0b7dfa0308c818c9a23051f6a9d33dc19c - says 95.66% (-0.02%) compared to ee62674.

[nicoddemus]

Would be good if we could control it with a single env var for Azure also.

Yeah not sure how to accomplish that at the moment; environment variables don't persist between tasks, and I'm not sure how to define some of them conditionally based on another variable (PYTEST_COVERAGE in this case). Anyway we can find a better solution later.

500 currently - everything is broken

Is back up now. 😁

[blueyed]

@nicoddemus

Merging #4839 into master will decrease coverage by 1.12%.

That indicates that we're missing something here.
Why have you merged it already?
It is still timing out on codecov for me.

[nicoddemus]

Oh sorry, I thought that was one step in this endeavor and you would follow in another PR.

Care to open a new PR to continue then?

[blueyed]

Yes, it was meant to be only first step(s), not going to be merged until it either fixes something (timeouts), or at least does not regress something..
Will not continue before tomorrow, so feel free to improve it on Azure's side already.