tox -e fix fails on first run, compiling editdistance

[pfmoore]

This is on Windows 10, with Python 3.9 and MSVC 2019 installed.

❯ tox -e fix
fix create: C:\Work\Projects\build\.tox\fix
fix installdeps: pre-commit>=2
fix installed: appdirs==1.4.4,cfgv==3.2.0,distlib==0.3.1,filelock==3.0.12,identify==2.1.4,nodeenv==1.5.0,pre-commit==2.11.1,PyYAML==5.4.1,six==1.15.0,toml==0.10.2,virtualenv==20.4.3
fix run-test-pre: PYTHONHASHSEED='352'
fix run-test: commands[0] | pre-commit run --all-files --show-diff-on-failure
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/psf/black.
[INFO] Initializing environment for https://github.com/PyCQA/isort.
[INFO] Initializing environment for https://github.com/asottile/setup-cfg-fmt.
[INFO] Initializing environment for https://github.com/PyCQA/flake8.
[INFO] Initializing environment for https://github.com/PyCQA/flake8:flake8-bugbear == 20.1.4.
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/PyCQA/isort.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/asottile/setup-cfg-fmt.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('C:\\Users\\UK03306\\.cache\\pre-commit\\repouqm_cn5e\\py_env-python3\\Scripts\\python.EXE', '-mpip', 'install', '.')
return code: 1
expected return code: 0
stdout:
    Processing c:\users\uk03306\.cache\pre-commit\repouqm_cn5e
    Collecting identify[license]>=1.3.0
      Using cached identify-2.1.4-py2.py3-none-any.whl (98 kB)
    Collecting editdistance
      Using cached editdistance-0.5.3.tar.gz (27 kB)
    Building wheels for collected packages: setup-cfg-fmt, editdistance
      Building wheel for setup-cfg-fmt (setup.py): started
      Building wheel for setup-cfg-fmt (setup.py): finished with status 'done'
      Created wheel for setup-cfg-fmt: filename=setup_cfg_fmt-1.16.0-py2.py3-none-any.whl size=8576 sha256=dd149ced17e9c6d665fab8db149c24e5a75be8e5d12c11de624f969a20dddbe7
      Stored in directory: C:\Users\UK03306\AppData\Local\Temp\pip-ephem-wheel-cache-f909jhrz\wheels\c2\ed\76\999c71ce57bbfa3f2a24411f177f0a910e3650e757ccca34f5
      Building wheel for editdistance (setup.py): started
      Building wheel for editdistance (setup.py): finished with status 'error'
      Running setup.py clean for editdistance
    Successfully built setup-cfg-fmt
    Failed to build editdistance
    Installing collected packages: identify, editdistance, setup-cfg-fmt
        Running setup.py install for editdistance: started
        Running setup.py install for editdistance: finished with status 'error'

stderr:
      ERROR: Command errored out with exit status 1:
       command: 'C:\Users\UK03306\.cache\pre-commit\repouqm_cn5e\py_env-python3\Scripts\python.EXE' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\UK03306\\AppData\\Local\\Temp\\pip-install-bc6d_nv1\\editdistance_e7944e4d830f46f5a3074f9db47a0fa2\\setup.py'"'"'; __file__='"'"'C:\\Users\\UK03306\\AppData\\Local\\Temp\\pip-install-bc6d_nv1\\editdistance_e7944e4d830f46f5a3074f9db47a0fa2\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\UK03306\AppData\Local\Temp\pip-wheel-y28ulqg5'
           cwd: C:\Users\UK03306\AppData\Local\Temp\pip-install-bc6d_nv1\editdistance_e7944e4d830f46f5a3074f9db47a0fa2\
      Complete output (12 lines):
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-3.9
      creating build\lib.win-amd64-3.9\editdistance
      copying editdistance\__init__.py -> build\lib.win-amd64-3.9\editdistance
      copying editdistance\_editdistance.h -> build\lib.win-amd64-3.9\editdistance
      copying editdistance\def.h -> build\lib.win-amd64-3.9\editdistance
      running build_ext
      building 'editdistance.bycython' extension
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
      ----------------------------------------
      ERROR: Failed building wheel for editdistance
        ERROR: Command errored out with exit status 1:
         command: 'C:\Users\UK03306\.cache\pre-commit\repouqm_cn5e\py_env-python3\Scripts\python.EXE' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\UK03306\\AppData\\Local\\Temp\\pip-install-bc6d_nv1\\editdistance_e7944e4d830f46f5a3074f9db47a0fa2\\setup.py'"'"'; __file__='"'"'C:\\Users\\UK03306\\AppData\\Local\\Temp\\pip-install-bc6d_nv1\\editdistance_e7944e4d830f46f5a3074f9db47a0fa2\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\UK03306\AppData\Local\Temp\pip-record-4sq_33gz\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\UK03306\.cache\pre-commit\repouqm_cn5e\py_env-python3\include\site\python3.9\editdistance'
             cwd: C:\Users\UK03306\AppData\Local\Temp\pip-install-bc6d_nv1\editdistance_e7944e4d830f46f5a3074f9db47a0fa2\
        Complete output (12 lines):
        running install
        running build
        running build_py
        creating build
        creating build\lib.win-amd64-3.9
        creating build\lib.win-amd64-3.9\editdistance
        copying editdistance\__init__.py -> build\lib.win-amd64-3.9\editdistance
        copying editdistance\_editdistance.h -> build\lib.win-amd64-3.9\editdistance
        copying editdistance\def.h -> build\lib.win-amd64-3.9\editdistance
        running build_ext
        building 'editdistance.bycython' extension
        error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
        ----------------------------------------
    ERROR: Command errored out with exit status 1: 'C:\Users\UK03306\.cache\pre-commit\repouqm_cn5e\py_env-python3\Scripts\python.EXE' -u -c 'import sys, 
setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\UK03306\\AppData\\Local\\Temp\\pip-install-bc6d_nv1\\editdistance_e7944e4d830f46f5a3074f9db47a0fa2\\setup.py'"'"'; __file__='"'"'C:\\Users\\UK03306\\AppData\\Local\\Temp\\pip-install-bc6d_nv1\\editdistance_e7944e4d830f46f5a3074f9db47a0fa2\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\UK03306\AppData\Local\Temp\pip-record-4sq_33gz\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\UK03306\.cache\pre-commit\repouqm_cn5e\py_env-python3\include\site\python3.9\editdistance' Check the logs for full command output.     

Check the log at C:\Users\UK03306\.cache\pre-commit\pre-commit.log
ERROR: InvocationError for command 'C:\Work\Projects\build\.tox\fix\Scripts\pre-commit.EXE' run --all-files --show-diff-on-failure (exited with code 3)
________________________________________________________________________ summary ________________________________________________________________________
ERROR:   fix: commands failed

The error "Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/" is incorrect, as I do have MSVC installed.

The log file shows nothing extra.

What is weird is that running pre-commit manually works. And after that, tox -e fix works, presumably because either the pre-commit env is already built, or pip has cached a built wheel for editdistance.

Maybe tox is dropping some environment variable that means MSVC isn't recognised? But if that were the case, I'd expect this to be a wider issue.

It looks like the editdistance project hasn't released binaries for Python 3.9 on Windows, which doesn't help here, as it means people need a C compiler to run the pre-checks 🙁

[jugmac00]

Maybe tox is dropping some environment variable that means MSVC isn't recognised?

Except a few vital one's, you have to whitelist all env vars you or a command within tox wants to use.

Scroll down to passenv
https://tox.readthedocs.io/en/latest/config.html

[pfmoore]

Yes that's what I was thinking. I have no idea what to whitelist, though, so that's something that this project would need to sort out (or arguably, tox shouldn't remove environment variables needed to build C extensions).

Having said that, if this was a general issue with tox and C compiler detection, I'm sure it would me much more common - so I suspect there's something else going on here.

[gaborbernat]

Yeah, the question here is what environment variable provisions the visual studio compiler on Windows? @asottile or @zooba ideas?

[asottile]

I have no idea, but I've been strongly considering forking editdistance and providing abi3 wheels for all platforms just because of this painpoint

[asottile]

bit the bullet, here's editdistance-s -- I'll be replacing the usage in pre-commit/identify#184

[gaborbernat]

Closing as fixed upstream.

[layday]

I was confused that I couldn't react to asottile's comment but he appears to have blocked me. Thanks for the fix anyway.