Wheel 0.34 broke installing in docker-alpine

[andronov04]

When i build docker image I get an error below

Use:

• python:3.7.6-alpine3.10
• Docker version 19.03.5, build 633a0ea

      Collecting setuptools>=34.4
        Using cached https://files.pythonhosted.org/packages/42/3e/2464120172859e5d103e5500315fb5555b1e908c0dacc73d80d35a9480ca/setuptools-45.1.0.zip
      Collecting wheel
        Using cached https://files.pythonhosted.org/packages/15/39/728ccb81172c73f14084948b2d4561d5e51649dcf4b21d1f207b7363b1d7/wheel-0.34.0.tar.gz
      ERROR: Exception:
      Traceback (most recent call last):
        File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 188, in main
          status = self.run(options, args)
        File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 345, in run
          resolver.resolve(requirement_set)
        File "/usr/local/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve
          self._resolve_one(requirement_set, req)
        File "/usr/local/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one
          abstract_dist = self._get_abstract_dist_for(req_to_install)
        File "/usr/local/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 307, in _get_abstract_dist_for
          self.require_hashes
        File "/usr/local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 213, in prepare_linked_requirement
          with self.req_tracker.track(req):
        File "/usr/local/lib/python3.6/contextlib.py", line 81, in __enter__
          return next(self.gen)
        File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_tracker.py", line 94, in track
          self.add(req)
        File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_tracker.py", line 63, in add
          % (link, fp.read()))
      LookupError: https://files.pythonhosted.org/packages/15/39/728ccb81172c73f14084948b2d4561d5e51649dcf4b21d1f207b7363b1d7/wheel-0.34.0.tar.gz#sha256=a69231973dccde8558f80f2ef058335a1d0bc1b55b2a5a2dfb5097e151e3c498 (from https://pypi.org/simple/wheel/) (requires-python:!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7) is already being built: wheel from https://files.pythonhosted.org/packages/15/39/728ccb81172c73f14084948b2d4561d5e51649dcf4b21d1f207b7363b1d7/wheel-0.34.0.tar.gz#sha256=a69231973dccde8558f80f2ef058335a1d0bc1b55b2a5a2dfb5097e151e3c498

[LeonJansen]

Same error here!

[agronholm]

This error comes from the internals of pip. Do you mind digging deeper to figure out what the LookupError is about? I can repro this outside of Docker with pip install --user wheel==0.34.0 --no-binary=wheel.

[tomasmor42]

I can try to help. Could you elaborate on what exactly do you mean by digging deeper?

[jamadden]

Here's another example, taken from pip install --no-binary :all: ./wheel-0.34.0.tar.gz. It looks like a dependency resolution loop. The setup_requires that wheel has on setuptools_scm[toml] is leading to trying to install wheel, because setuptools_scm has an setup_requires on wheel. (This is all in pyroject.toml files.)

$ pip install --no-binary :all: ./wheel-0.34.0.tar.gz
Processing ./wheel-0.34.0.tar.gz
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/bin/python3.8 /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-build-env-eewzron7/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --pre -- 'setuptools >= 42' 'setuptools_scm[toml]>=3.4'
       cwd: None
  Complete output (51 lines):
  Collecting setuptools>=42
    Downloading setuptools-45.1.0.zip (859 kB)
  Collecting setuptools_scm[toml]>=3.4
    Downloading setuptools_scm-3.4.3.tar.gz (45 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/bin/python3.8 /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-build-env-v8z5aszd/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --pre -- 'setuptools>=34.4' wheel
         cwd: None
    Complete output (39 lines):
    Collecting setuptools>=34.4
      Using cached setuptools-45.1.0.zip (859 kB)
    Collecting wheel
      Downloading wheel-0.34.0.tar.gz (60 kB)
      Installing build dependencies: started
      Installing build dependencies: finished with status 'error'
      ERROR: Command errored out with exit status 2:
       command: /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/bin/python3.8 /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-build-env-6fxxxyn6/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --pre -- 'setuptools >= 42' 'setuptools_scm[toml]>=3.4'
           cwd: None
      Complete output (27 lines):
      Collecting setuptools>=42
        Using cached setuptools-45.1.0.zip (859 kB)
      Collecting setuptools_scm[toml]>=3.4
        Using cached setuptools_scm-3.4.3.tar.gz (45 kB)
      ERROR: Exception:
      Traceback (most recent call last):
        File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
          status = self.run(options, args)
        File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 331, in run
          resolver.resolve(requirement_set)
        File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
          discovered_reqs.extend(self._resolve_one(requirement_set, req))
        File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
          abstract_dist = self._get_abstract_dist_for(req_to_install)
        File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
          abstract_dist = self.preparer.prepare_linked_requirement(req)
        File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 515, in prepare_linked_requirement
          abstract_dist = _get_prepared_distribution(
        File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 94, in _get_prepared_distribution
          with req_tracker.track(req):
        File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
          return next(self.gen)
        File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/req/req_tracker.py", line 148, in track
          self.add(req)
        File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/req/req_tracker.py", line 115, in add
          raise LookupError(message)
      LookupError: https://files.pythonhosted.org/packages/fe/bd/bc2fe0b14ce234bb5e2af5f3b574c5a8ef1b7845bfa41e7cf69a78627ec8/setuptools_scm-3.4.3.tar.gz#sha256=26b8a108783cd88f4b15ff1f0f347d6b476db25d0c226159b835d713f9487320 (from https://pypi.org/simple/setuptools-scm/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*) is already being built: setuptools_scm[toml]>=3.4 from https://files.pythonhosted.org/packages/fe/bd/bc2fe0b14ce234bb5e2af5f3b574c5a8ef1b7845bfa41e7cf69a78627ec8/setuptools_scm-3.4.3.tar.gz#sha256=26b8a108783cd88f4b15ff1f0f347d6b476db25d0c226159b835d713f9487320
      ----------------------------------------
    ERROR: Command errored out with exit status 2: /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/bin/python3.8 /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-build-env-6fxxxyn6/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --pre -- 'setuptools >= 42' 'setuptools_scm[toml]>=3.4' Check the logs for full command output.
    ----------------------------------------
  ERROR: Command errored out with exit status 1: /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/bin/python3.8 /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-build-env-v8z5aszd/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --pre -- 'setuptools>=34.4' wheel Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/bin/python3.8 /Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-build-env-eewzron7/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --pre -- 'setuptools >= 42' 'setuptools_scm[toml]>=3.4' Check the logs for full command output.

EDIT: Sorry, not install_requires, setup_requires

wheel's pyproject.toml:

[build-system]
requires = ["setuptools >= 42", "setuptools_scm[toml]>=3.4"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]
write_to = "src/wheel/_version.py"
version_scheme = "post-release"

setuptools_scm's pyproject.toml:

[build-system]
requires = ["setuptools>=34.4", "wheel"]
build-backend = "setuptools.build_meta"

[agronholm]

I myself am trying to debug pip to understand what is going wrong. It's as if wheel is somehow requiring itself during the build, but I don't quite see how. If you could delve into pip code to help me with this, either in vivo or in vitro, that could speed up the process.

[agronholm]

The install_requires that wheel has on setuptools_scm[toml] is leading to trying to install wheel, because setuptools_scm has an install_requires on wheel

Damn, that would explain things.

[agronholm]

It's not in install_requires for setuptools_scm, though, so why would pip try to install its build requirements?

[jamadden]

It's not in install_requires for setuptools_scm, though, so why would pip try to install its build requirements?

Sorry, I edited my post to clarify that. It's in the build requirements in pyproject.toml (essentially setup_requires) for setuptools_scm.

[agronholm]

Yes, but those are only installed if the project is being built from source, which in this case should not be.

[agronholm]

To clarify, I am getting a similar error even when specifying only --no-binary=wheel. Thus only wheel itself should be installed from the sdist in that case.

[dyeager-recurve]

I get the same error with /usr/local/bin/python -m pip download --dest /tmp/wheel-error/ wheel==0.34.0 --exists-action i --no-binary :all:

Collecting wheel==0.34.0
  Using cached wheel-0.34.0.tar.gz (60 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-x427jk6u/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools >= 42' 'setuptools_scm[toml]>=3.4'
       cwd: None
  Complete output (39 lines):
  Collecting setuptools>=42
    Using cached setuptools-45.1.0.zip (859 kB)
  Collecting setuptools_scm[toml]>=3.4
    Using cached setuptools_scm-3.4.3.tar.gz (45 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 2:
     command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f7wcywhz/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=34.4' wheel
         cwd: None
    Complete output (27 lines):
    Collecting setuptools>=34.4
      Using cached setuptools-45.1.0.zip (859 kB)
    Collecting wheel
      Using cached wheel-0.34.0.tar.gz (60 kB)
    ERROR: Exception:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
        status = self.run(options, args)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 331, in run
        resolver.resolve(requirement_set)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
        discovered_reqs.extend(self._resolve_one(requirement_set, req))
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
        abstract_dist = self._get_abstract_dist_for(req_to_install)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
        abstract_dist = self.preparer.prepare_linked_requirement(req)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 516, in prepare_linked_requirement
        req, self.req_tracker, self.finder, self.build_isolation,
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 94, in _get_prepared_distribution
        with req_tracker.track(req):
      File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
        return next(self.gen)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 148, in track
        self.add(req)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 115, in add
        raise LookupError(message)
    LookupError: https://files.pythonhosted.org/packages/15/39/728ccb81172c73f14084948b2d4561d5e51649dcf4b21d1f207b7363b1d7/wheel-0.34.0.tar.gz#sha256=a69231973dccde8558f80f2ef058335a1d0bc1b55b2a5a2dfb5097e151e3c498 (from https://pypi.org/simple/wheel/) (requires-python:!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7) is already being built: wheel==0.34.0 from https://files.pythonhosted.org/packages/15/39/728ccb81172c73f14084948b2d4561d5e51649dcf4b21d1f207b7363b1d7/wheel-0.34.0.tar.gz#sha256=a69231973dccde8558f80f2ef058335a1d0bc1b55b2a5a2dfb5097e151e3c498
    ----------------------------------------
  ERROR: Command errored out with exit status 2: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f7wcywhz/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=34.4' wheel Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-x427jk6u/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools >= 42' 'setuptools_scm[toml]>=3.4' Check the logs for full command output.

The context for discovery here is that our Apache Beam/Dataflow jobs first run a pip download of what is needed and started failing this morning. Hopefully this helps.

[dyeager-recurve]

For more context - the issue above was traced when trying to do a pip download -r requirements.txt and suddenly python-dateutil install was breaking, so I suspect this may be impacting several libraries

[agronholm]

This is mystifying:

pip install --user wheel==0.34.0 --no-binary=wheel
Collecting wheel==0.34.0
  Using cached wheel-0.34.0.tar.gz (60 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... error

Doesn't look like setuptools_scm's build dependency is getting in the way at all. But where are the "backend dependencies" coming from?

[jamadden]

To clarify, I am getting a similar error even when specifying only --no-binary=wheel. Thus only wheel itself should be installed from the sdist in that case.

Oh. In that case, I think it's because pip wants to install a project that has a pyproject.toml by…making a wheel of it. Or, more directly, it's because the setuptools build backend implementation of get_requires_for_build_wheel includes wheel in the return value. Here's a portion of the output from the above command, instrumented to print that value:

  Successfully installed setuptools-45.1.0 setuptools-scm-3.4.3 toml-0.10.0
  Cleaning up...
  Removed build tracker: '/private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-req-tracker-n3eh82a_'
  Installing build dependencies ... done
  Running command //bin/python3.8 ///pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel //T/tmpig291pld
***
Requirements: ['wheel', 'setuptools_scm>=3.4']
****
  Getting requirements to build wheel ... done
  Running command //bin/python3.8 //site-packages/pip install --ignore-installed --no-user --prefix //T/pip-build-env-5b5fcnl5/normal --no-warn-script-location -v --no-binary wheel --only-binary :none: -i https://pypi.org/simple --pre -- wheel
  Non-user install by explicit request
  Created temporary directory: /T/pip-ephem-wheel-cache-yg_vs_tf
  Created build tracker: /T/pip-req-tracker-n3eh82a_
  Entered build tracker: //T/pip-req-tracker-n3eh82a_
  Created temporary directory: //T/pip-install-itmu_l3v
  1 location(s) to search for versions of wheel:
  * https://pypi.org/simple/wheel/
  Fetching project page and analyzing links: https://pypi.org/simple/wheel/
   Collecting wheel
      Created temporary directory: /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-unpack-887ms133
      Getting credentials from keyring for files.pythonhosted.org
      Looking up "https://files.pythonhosted.org/packages/15/39/728ccb81172c73f14084948b2d4561d5e51649dcf4b21d1f207b7363b1d7/wheel-0.34.0.tar.gz" in the cache
      Current age based on date: 2435
      Ignoring unknown cache-control directive: immutable
      Freshness lifetime from max-age: 365000000
      The response is "fresh", returning cached response
      365000000 > 2435
      Using cached wheel-0.34.0.tar.gz (60 kB)
    Cleaning up...
      Removing source in /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-install-0h5ui0t6/wheel
    Removed build tracker: '/private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-req-tracker-n3eh82a_'
    ERROR: Exception:
    Traceback (most recent call last):
      File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
        status = self.run(options, args)
      File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 331, in run
        resolver.resolve(requirement_set)
      File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
        discovered_reqs.extend(self._resolve_one(requirement_set, req))
      File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
        abstract_dist = self._get_abstract_dist_for(req_to_install)
      File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
        abstract_dist = self.preparer.prepare_linked_requirement(req)
      File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 515, in prepare_linked_requirement
        abstract_dist = _get_prepared_distribution(
      File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 94, in _get_prepared_distribution
        with req_tracker.track(req):
      File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
        return next(self.gen)
      File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/req/req_tracker.py", line 148, in track
        self.add(req)
      File "/Users/jmadden/Projects/VirtualEnvs/tmp-b5481b392b4dda0c/lib/python3.8/site-packages/pip/_internal/req/req_tracker.py", line 115, in add
        raise LookupError(message)
    LookupError: https://files.pythonhosted.org/packages/15/39/728ccb81172c73f14084948b2d4561d5e51649dcf4b21d1f207b7363b1d7/wheel-0.34.0.tar.gz#sha256=a69231973dccde8558f80f2ef058335a1d0bc1b55b2a5a2dfb5097e151e3c498 (from https://pypi.org/simple/wheel/) (requires-python:!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7) is already being built: wheel from https://files.pythonhosted.org/packages/15/39/728ccb81172c73f14084948b2d4561d5e51649dcf4b21d1f207b7363b1d7/wheel-0.34.0.tar.gz#sha256=a69231973dccde8558f80f2ef058335a1d0bc1b55b2a5a2dfb5097e151e3c498