cache poisoning via automated wheels

[mmerickel]

A lot of projects distributed as an sdist have custom runtime code in setup.py that depends on the exact environment being run. It appears as if when the wheel is built and put into the wheel cache this is not done "per runtime" (at least by python version). This is especially evident when running tests via tox which uses multiple python versions and pip.

For example, installing mako on py33 and then on py32 will install markupsafe into py32 (which isn't supported). Mako's setup.py properly avoids markupsafe on 3.2 but since the wheel was originally built on 3.3 the wheel now depends on markupsafe.

The issue here is that mako is not distributed as a wheel and thus does not have a setup.cfg or anything broadcasting the proper metadata and yet pip assumes it can just build the wheel once and use it everywhere. This should be considered a cache poisoning bug in the wheel cache.

[dstufft]

/cc @rbtcollins

Off hand, I don't think it's unreasonable to reduce our cache hit rate a bit in order to work in more situations. The crushing weight of legacy is a pain. This is only really a factor for pure python wheels, since wheels that have compiled code will be version specific anyways, which are going to be fairly quick to build anyways.

[rbtcollins]

One case doesn't make a trend.

Mako's sdist tarball has a setup.cfg in it containing:

[wheel]
universal = 1

I think we have to trust that folk configuring universal wheels intend to do so.

if this is a general systemic problem then sure, we could pass an option to disable the building of universal wheels for the wheels we put in the cache. But I'm not convinced it is so far.

[rbtcollins]

Oh, I should mention that while I don't think its super common, its actually pathological and something we can't defend against: https://bitbucket.org/runeh/anyjson/src/0026a68c035696bdc8db8628e364139eba9a8ba8/setup.py?at=default#setup.py-57

I think we should push the 'get it right by default' logic, whatever that is, down to wheel, since pips goal is to consume binaries where possible, and folk building bad wheels and uploading them to pypi is as much an issue as wheels in the cache. (I've done that, for instance).

[rbtcollins]

So perhaps looking at it this way.

Some % of packages apply logic to determine dependencies at build time and export that as unconditional install time requirements. Sometimes that logic is coupled to Python version, sometimes not.

Most of those packages cannot be built into wheels today:

• Some % of them will claim, as Mako does, that they can build universal wheels. This is bogus :).
• Some % will not make that claim and will only consult Python major version, thus being compatible with wheels default behaviour.
• The rest won't make that claim but will consult e.g. the Python minor version, or the set of importable packages, or pkg_resources or $anything. These are completely incompatible with being built into wheels.

There's no oracle I can think of to determine which category a given package falls into (wheel metadata is valid, wheel metadata is invalid but explicitly chosen that way by the package, wheel metadata is invalid due to accident).

Our logic to inject setuptools to get the bdist_wheel command is pragmatically fine: a distutils package doesn't have machine expressed dependencies anyway.

So this builds down to the following questions IMO:

A: How do we handle packages that build wheels dependent on more than what their wheel by default expresses, but which can be expressed by wheel without changing egg-info metadata.

B: How do we handle packages that build wheels dependent on more than Python major.minor version [and don't express it via egg-info metadata].

@dstufft on IRC suggested passing a more restrictive wheel tag (e.g. cp26 on CPython 2.6).

That would fix A) as I understand it, but not B).

I'm personally more worried by B than A, since A can be easily fixed by the package author, but B cannot. For instance, anyjson requires ecosystem changes - an alternatives mechanism for dependencies.

[dstufft]

Right, but we can't really solve B currently, we can't fix things for anyjson right now until we make more ecosystem changes. I don't think it's going to be possible for us to really solve this for every single scenario, but I think that using a more restrictive wheel tag will at least reduce the cases that we have problems in. I should note that another thing people do is different packages for different interpreters and different packages for different operating systems.

For different interpreters, the more restrictive tag will also fix these cases since the more restrictive tag includes the interpreter baked into it as well. We won't solve the case where people use different dependencies on different OSs, but I don't think it's reasonable to assume that the cache can be shared between OSs.

[rbtcollins]

Another option would be to not solve attempt to solve this systematically and instead solve it case by case.

The --no-binary option exists to let folk opt out of wheels on a per package basis - its the escape clause.

Folk should use --no-binary anyjson today, but they don't know that they should. I'm using anyjson as a specific example since its not fixable any other way [today]. Publishing a known-problematic list of packages would let this be socialised. As I suggested on IRC we could even publish that on a REST API for programmatic consumption.

I would like to provide some signal to package authors like Mako - where things can be done better - that they should move to conditional dependencies rather than computed dependencies.

[mmerickel]

Would this be an issue if mako did not have the wheel directive in setup.cfg ? Presumably then the cache would contain platform tags instead of being cached as universal? If that's the case then this is arguably a misconfiguration in mako to support universal wheels without the proper metadata section in setup.cfg.

[dstufft]

It would still be a problem, it would just have a py2 and a py3 wheel instead of a py2.py3 wheel.

[rbtcollins]

BTW, I've filed https://bitbucket.org/zzzeek/mako/issues/249/mako-wheels-have-computed-install_requires

[rbtcollins]

It is a bug in Mako. But its a bug we expect lots of packages to have because legacy. bdists had the same issues for years but few folk used them. pip's wheel cache is massively increasing the surface areas of packagers' packages interacting with wheels.

[mmerickel]

There should be a way (such as [wheel] universal=1) that a package broadcasts support for being built with a wheel... if it doesn't then maybe still try to build it but with the most specific platform tags possible, or don't cache it at all.

[mmerickel]

Is there any progress on this? This is a serious regression as far as any use cases regarding tox so I'm surprised it isn't being fixed sooner. I basically can't use tox without disabling the download cache anymore.

[rbtcollins]

There's a proposed fix for mako. The restrictions @dstufft proposed here haven't been coded up yet - and won't fix mako anyhow (because py2/py3 is not specific enough)

[mgedmin]

pip's wheel caching pushed me to submit fixes for at least two packages to use environment markers instead of dynamically computing install_requires in setup.py.

If I had to submit a patch for a 3rd package, I would have to research the marker syntax from scratch again, because it's underdocumented.

I guess my point is: the best way to fix this is to improve documentation.