Support wheel cache when using --require-hashes

[alex]

Currently if you use --require-hashes, pip will ignore the wheel cache because it doesn't know that the wheel was built from an sdist with that hash.

pip should record the hash of the sdist used to build wheels in the wheel cache, and then check that and use the wheel if it matches the pinned hash.

[pradyunsg]

Hey there! Thanks for filing this issue.

What is the functional benefit of doing this?

[alex]

Speeeeeed.

This is a huge performance drain on warehouse's CI, because it cannot reuse compiled wheels.

[ewdurbin]

Dove in to this a bit more today and at a cursory level it appears this could be done in a robust manner without sacrificing the value of the hash to begin with.

Since the final directory of the cached wheel is generated based on the InstallRequirement it looks like we may be able to reliably determine if a given wheel in the wheel cache was built from a given source dist hash?

I haven't dug much deeper, but this would significantly improve the CI performance for warehouse.

[uranusjr]

Yes it seems so. I think the most problematic part in implementation would be to correctrly refactor existing code to expose the wheel cache to the resolver. Not difficult, but need someone committed to sweat it out.

[sbidoul]

I'm experimenting with this in #11042.

@alex @ewdurbin It would be great if you could check if that PR fits your use cases.

[sbidoul]

There is one potential complexity: which hash algorithm to use when recording the cache entry. If we implement this naively, it will create backward compatibility issues for users who have requirements.txt with different hash algorithms. So we may need to re-download if the hash algorithm don't match... Nothing is simple with pip.

[ewdurbin]

This has been getting in my way so much lately that I've decided to put up a personal bounty for an implementation that closes it: https://twitter.com/EWDurbin/status/1638834445900734471