Make it easier to run tests for the first time, with nicer error messages

[bertilhatt]

What's the problem this feature will solve?

As a new contributor, I cloned the repository and tried to run a test on master. I was hit with hundreds of error: 15k lines of error messages, when executing either pytest -n 4, or tox -e py37.

I suspect that the bulk of these is due to missing dependencies but it’s not clear what I need to do from there:

• I followed the first paragraph of the guideline;
• the messages are over-whelming and unclear;
• there are no requirements.txt or similar to follow.

Describe the solution you'd like

If the problem is a lack of dependencies, a clearer (and shorter) error message and interrupting the test.

That was, when a new contributor tries to set up a local version, they can have green tests before they contribute, and test locally.

Alternative Solutions

Another solution would be to detect and install requirements as the test go.

Additional context

New contributor, part of an introduction event

[benoit-pierre]

Using tox is the right way to run the tests (not calling pytest directly), can you give a snapshot of those errors? I don't see why this would not work, except maybe if your version of tox is older than 3.4 (see #5914).

[bertilhatt]

The output of tox is in the description too: https://gist.github.com/bertilhatt/3ddc67914a937c7cda22259af240457f

[bertilhatt]

:pip bertil.hatt$ tox --version
3.5.2 imported from /usr/local/lib/python3.7/site-packages/tox/__init__.py

[benoit-pierre]

Oups, sorry, missed that! As for the errors: it looks like your changes are the issue! Did you run the testsuite first without any local changes?

[benoit-pierre]

Also, use the pytest -x argument to exit instantly on first error or failed test, e.g.: tox -e py -- -x.

[bertilhatt]

All those are tested on master, without any changes from the origin repository.

[benoit-pierre]

I'm ignoring the output from running pytest directly (use tox), the output from running tox shows a number of errors related to the use of SETUPTOOLS_SHIM, are you sure?

[bertilhatt]

The tests that I’m seeing failing are not mentioning SETUPTOOLS_SHIM:

:pip bertil.hatt$ tox -e py37 -- -x
py37 develop-inst-noop: /Users/bertil.hatt/Repositories/pip
py37 installed: apipkg==1.5,atomicwrites==1.2.1,attrs==18.2.0,certifi==2018.10.15,chardet==3.0.4,coverage==4.5.1,execnet==1.5.0,filelock==3.0.9,idna==2.7,more-itertools==4.3.0,nose==1.3.7,pip==18.1,pipenv==2018.10.13,pluggy==0.8.0,py==1.7.0,pytest==3.9.1,pytest-cov==2.6.0,pytest-forked==0.2,pytest-xdist==1.23.2,requests==2.20.0,requirements==0.1,scripttest==1.3,setuptools==40.4.3,six==1.11.0,toml==0.10.0,tox==3.5.2,urllib3==1.24,virtualenv==16.0.0,virtualenv-clone==0.4.0,-e git+git@github.com:pypa/wheel.git@3d3708c659f7b9988f958539e7748a1a01ee6e77#egg=wheel
py37 run-test-pre: PYTHONHASHSEED='2557811667'
py37 run-test-pre: commands[0] | python -c 'import shutil, sys; shutil.rmtree(sys.argv[1], ignore_errors=True)' /Users/bertil.hatt/Repositories/pip/tests/data/common_wheels
py37 run-test-pre: commands[1] | python /Users/bertil.hatt/Repositories/pip/tools/tox_pip.py wheel -w /Users/bertil.hatt/Repositories/pip/tests/data/common_wheels -r /Users/bertil.hatt/Repositories/pip/tools/tests-common_wheels-requirements.txt
Collecting setuptools (from -r /Users/bertil.hatt/Repositories/pip/tools/tests-common_wheels-requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/82/a1/ba6fb41367b375f5cb653d1317d8ca263c636cff6566e2da1b0da716069d/setuptools-40.5.0-py2.py3-none-any.whl
  Saved ./tests/data/common_wheels/setuptools-40.5.0-py2.py3-none-any.whl
Collecting wheel (from -r /Users/bertil.hatt/Repositories/pip/tools/tests-common_wheels-requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/5a/9b/6aebe9e2636d35d1a93772fa644c828303e1d5d124e8a88f156f42ac4b87/wheel-0.32.2-py2.py3-none-any.whl
  Saved ./tests/data/common_wheels/wheel-0.32.2-py2.py3-none-any.whl
Skipping setuptools, due to already being wheel.
Skipping wheel, due to already being wheel.
py37 runtests: commands[0] | pytest --timeout 300 -x
============================================ test session starts ============================================
platform darwin -- Python 3.7.0, pytest-3.8.2, py-1.7.0, pluggy-0.8.0
rootdir: /Users/bertil.hatt/Repositories/pip, inifile: setup.cfg
plugins: xdist-1.23.2, timeout-1.3.2, rerunfailures-4.2, forked-0.2, cov-2.6.0
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 944 items                                                                                         

tests/functional/test_check.py .F

================================================= FAILURES ==================================================
____________________________________ test_basic_check_missing_dependency ____________________________________

script = <tests.lib.PipTestEnvironment object at 0x112cfa518>

    def test_basic_check_missing_dependency(script):
        # Setup a small project
        pkga_path = create_test_package_with_setup(
            script,
            name='pkga', version='1.0', install_requires=['missing==0.1'],
        )
        # Let's install pkga without its dependency
>       res = script.pip('install', '--no-index', pkga_path, '--no-deps')

tests/functional/test_check.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/lib/__init__.py:379: in pip
    return self.run(exe, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.lib.PipTestEnvironment object at 0x112cfa518>
args = ('python', '-m', 'pip', 'install', '--no-index', Path('/private/var/folders/7z/vcpr99y5437f9vj2hp3_7pxmzzdkv1/T/pytest-of-bertil.hatt/pytest-10/test_basic_check_missing_depen0/workspace/scratch/pkga'), ...)
kw = {}
cwd = Path('/private/var/folders/7z/vcpr99y5437f9vj2hp3_7pxmzzdkv1/T/pytest-of-bertil.hatt/pytest-10/test_basic_check_missing_depen0/workspace/scratch')
run_from = None

    def run(self, *args, **kw):
        if self.verbose:
            print('>> running %s %s' % (args, kw))
        cwd = kw.pop('cwd', None)
        run_from = kw.pop('run_from', None)
        assert not cwd or not run_from, "Don't use run_from; it's going away"
        cwd = cwd or run_from or self.cwd
        if sys.platform == 'win32':
            # Partial fix for ScriptTest.run using `shell=True` on Windows.
            args = [str(a).replace('^', '^^').replace('&', '^&') for a in args]
        return TestPipResult(
>           super(PipTestEnvironment, self).run(cwd=cwd, *args, **kw),
            verbose=self.verbose,
        )
E       AssertionError: Script returned code: 1

tests/lib/__init__.py:363: AssertionError
------------------------------------------- Captured stdout call --------------------------------------------
Script result: python -m pip install --no-index /private/var/folders/7z/vcpr99y5437f9vj2hp3_7pxmzzdkv1/T/pytest-of-bertil.hatt/pytest-10/test_basic_check_missing_depen0/workspace/scratch/pkga --no-deps
  return code: 1
-- stderr: --------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/7z/vcpr99y5437f9vj2hp3_7pxmzzdkv1/T/pytest-of-bertil.hatt/pytest-10/test_basic_check_missing_depen0/workspace/tmp/pip-req-build-fnz4u8gn/

-- stdout: --------------------
Processing ./pkga
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    NameError: name 'sys' is not defined
    
    ----------------------------------------

========================================== short test summary info ==========================================
FAIL tests/functional/test_check.py::test_basic_check_missing_dependency
============================================= warnings summary ==============================================
/Users/bertil.hatt/Repositories/pip/.tox/py37/lib/python3.7/site-packages/yaml/constructor.py:126: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  if not isinstance(key, collections.Hashable):

/private/var/folders/7z/vcpr99y5437f9vj2hp3_7pxmzzdkv1/T/pytest-of-bertil.hatt/pytest-10/pip0/pip/src/pip/_vendor/requests/status_codes.py:18: DeprecationWarning: invalid escape sequence \o
  """

-- Docs: https://docs.pytest.org/en/latest/warnings.html
============================== 1 failed, 1 passed, 2 warnings in 16.14 seconds ==============================
ERROR: InvocationError for command '/Users/bertil.hatt/Repositories/pip/.tox/py37/bin/pytest --timeout 300 -x' (exited with code 1)
__________________________________________________ summary __________________________________________________
ERROR:   py37: commands failed

[cjerdonek]

The test failures have this--

NameError: name 'sys' is not defined

And in the PR, you add the use of sys without a corresponding import:

+ "import setuptools, tokenize; sys.argv[0] = {0!r}; __file__={0!r};"

[bertilhatt]

Yes, that was committed in #5936.